Page 1 of 4
INFS 2049 – UO Experimental Design
Assessment 2 – Project
Due: Friday in Week 10 at 5 PM
Instructions:
• This assessment is worth 50% of your final grade. It is due no later than 5 pm on Friday in
Week 10.
• You will need to submit your assignment via learnonline. The file you submit needs to be in a
pdf format and prepared using the template provided.
• Your submission will be marked out of 50.
• The statistical analysis portion of the project is worth the majority of marks and should be
well-detailed. For all analysis include the necessary assumption checking visualisations where
applicable and provide thorough interpretations for the results.
• All statistical tests should be conducted at the 0.05 significance level.
Assessment Task Overview:
Photos by Julian Hochgesang, Austin Distel and CardMapr.nl on Unsplash
Digital advertising is one of the most effective ways for businesses to expand their reach, find new
customers, and diversify their revenue streams. Businesses can pay to have their ads appear on
popular online channels, including search engines, social media platforms, websites, and more.
Digital advertising is very cost-effective since a business pays only when someone takes the desired
action, clicks a link to the company’s website. An online platform will typically run an auction every
time an ad space is available. Each auction decides which ads will be shown in that space. There are
several ways businesses can bid for their ads, depending on what matters most to them. Most of the
time, they focus on clicks, impressions, conversions or views.
Imagine you are a Data Analyst and you have a client who has done some A/B testing of the bidding
method they cu
ently use, against an alternative method that has just become available to them.
They would like to understand if the new bidding method has potential to
ing more conversions,
ut otherwise do not have any direction for the analysis. You will now analyse their data and
prepare a report for them with your findings.
This project is your opportunity to
ing together the knowledge you have acquired in this course,
apply it to a real-world scenario and further develop your communication skills.
Page 2 of 4
Assessment Task Details:
Perform the statistical analyses indicated in this document and then write a report that explains your
findings and recommendations. Follow instructions provided in this document. There are two data
files for this project: control.csv and test.csv. Please refer to ‘Appendix A’ for variable descriptions.
Outline of the experiment:
TradeTulip.com is an online retail company that has been running digital advertising campaigns on
BrandHive online platform using maximum bidding. BrandHive has recently introduced a new
idding option based on target cost. TradeTulip.com has decided to test this new feature to
understand if target cost bidding would
ing them more conversions than maximum bidding.
Conversions are important as they allow a business to gauge the performance of their advertising
campaigns. A conversion occurs when a visitor to a website completes a desired action. For
TradeTulip.com, a desired action is a customer making a purchase.
TradeTulip.com has set up an A/B test to compare maximum bidding to target cost bidding. The A/B
test has run for one month and TradeTulip.com now expects you to analyse the results of this A/B
test and report on your findings.
Analysis to be performed in R:
For each of the following questions, identify an appropriate analysis method (e.g. t-test, ANOVA). Do
all the necessary assumption checking and perform any follow-up tests if you think they are needed
or appropriate. Note: Some data cleaning and wrangling may be required.
Results from your R analyses will form the basis of your report. Do not submit all your output.
Instead, focus on reporting results and documenting your R code as you will need to submit it for
assessment.
Note: The data may not comply with some of the assumptions for the analysis method you identify
for each scenario; you need to check this and consider the severity of any deviations. If you decide
there is a problem, you can employ one of the following:
• A data transformation (e.g. Box-Cox transformation);
• A non-parametric analysis method;
• Sampling to create treatment groups with equal sample sizes.
1. Begin by calculating three new measures for each date in the data set:
Conversion rate = percentage of visitors to the website that make a desired action
Cost per click (CPC) = Total cost of ads / Total number of clicks
Cost per action (CPA) = total cost of ads / Total number of desired actions
2. Now perform some exploratory data analysis. Use summaries of the data as well as
visualisations to gain insights into characteristics of the dataset and potential relationships
etween factors and response variables.
For the report: Select some key descriptive statistics and three to four visualisations to be
included in the report.
3. Is there a statistically significant difference between maximum bidding and target cost bidding
ased on:
• Conversion rate?
• Cost per click?
Page 3 of 4
• Cost per action?
• Cost of the campaign?
• Reach?
Note: Data from this experiment is paired, and you need to take this into account in your
analysis.
4. Is there any other question that you think is worth asking and could be answered with the data
from this A/B experiment? Pick one question and perform the co
esponding analysis.
Report:
Once you have completed your analyses and understood your results, write a report describing your
findings. Do not simply answer the questions on pages 2-3 in this document; they are provided as a
guide for your analysis. Focus on interpretation and practical significance of your results. Select the
most relevant output and think about how to present it effectively for your client.
A template for the project report is provided in learnonline.
Presentation and structure:
The structure should be in a logical format that flows well. Please use the structure outlined in the
eport template – you can add to it with sub-headings if you wish.
Since this is a report for a client, it should be presented in a professional format making it easy to
ead. An efficient layout is also important but do not spend too much time on making it look good
and not enough time on the content.
Using bullet points are OK occasionally but you will need sentences for each point (i.e. just a bullet
point list with no explanation is not suitable).
Word limit:
There is no word limit as such, but it is expected that your report will be approximately 5 to 6 pages
including relevant graphs and tables, with your R code in an appendix.
Assessment Criteria:
The project will be marked on how well you cover the following:
Area Weighting
Data analysis, incl. R code 50%
Introduction and methods 10%
Discussion and conclusions 25%
Use of formal business or academic language,
including co
ect grammar and spelling
10%
Layout and professional presentation 5%
Page 4 of 4
Appendix A – Data Dictionary
Variables in the data files are as follows:
Variable Description Variable name
Control = maximum bidding method
This bidding strategy sets the maximum amount that can
e spent on a single bid.
Test = target cost bidding method
This bidding strategy aims to keep the average cost per
action at or below the target cost at the set end date.
Campaign
Calendar date Date
Day of the week Day
The amount spent on ads in USD Spend
The number of times the ad was displayed, whether it was
clicked on or not. In practice, an impression occurs any
time a user opens an app or website, and an ad is visible
Impressions
The total number of unique viewers who saw the ad. Reach
The number of viewers who clicked the link to the
company website displayed in the ad
WebsiteClicks
The number of searches performed within the company
website
Searches
The number of times product details were viewed ContentViews
The number of products that were added to the cart AddCart
The number of products that were purchased Purchase
INFS 2049 – UO Experimental Design
Project Report
Submitted by
[Enter your full name here]
[Enter your student ID number here]
[Title of your report]
Introduction
The introduction should be able to be understood by a layperson and should include the motivation for the experiment as well as an outline of the contents of your report.
There is no word limit. As a guideline, one paragraph will be sufficient.
[Delete instruction text before submitting]
[Type your introduction here]
Methods
Describe the experimental design, participants and variables that you have analysed. Also provide a list of analysis methods that you have used.
Note: Do not include the full data dictionary; instead summarise the types of variables that were collected (e.g. psychological wellbeing).
There is no word limit. As a guideline, one to two paragraphs for this section will be sufficient.
[Delete instruction text before submitting]
[Type your description of methods here]
Results & Discussion
First, summarise the main results of your analyses. You may use subsections, tables etc. as you see fit. Present and discuss results in a clear and simple way:
Present findings of statistical analyses in a logical sequence. Descriptive statistics and visualisations are usually presented first, followed by the results of further analyses.
Include visualisations with your results.
State each result and the co
esponding analysis procedure, and report P-values to three decimal places. However, do not include code or dumps of R output. Results should either be incorporated into sentences or else formatted into neatly presented tables.
Next, interpret your findings by discussing their practical significance. Use plain language; there should be no technical details or statistical terminology. Are any of the results surprising in any way? Ensure you address the following:
Did participants end up eating more fruit and vegetables?
Which intervention proved to be effective and in what way?
Were there any significant improvements to wellbeing of participants and in what sense?
Finally, in another paragraph discuss shortcomings, if any, of the experimental design and analyses that were performed.
There is no word limit. As a guideline, three pages will be sufficient for this section, including any tables and graphs.
[Delete instruction text before submitting]
[Type your results and discussion here]
Recommendations & Conclusions
Type your recommendations and conclusions here
What do you conclude overall about the effects of increased fruit and vegetable consumption on psychological well-being of young adults, and the effectiveness of proposed app intervention?
Do you have any recommendations for the client?
There is no word limit. As a guideline, one paragraph will be sufficient. Do not introduce any new information in this section, and do not simply repeat statements made elsewhere in your report!
[Delete instruction text before submitting]
[Type your recommendations and conclusions here]
Appendix
Your full R code goes here, including data prep and all relevant condition checking.
Make sure that the code is well commented. In particular, clearly indicate which questions are being addressed with each section of the code, and what is to be achieved. Comment
iefly on the outcome.
Do not include the full output from your code.
[Delete instruction text before submitting]
1