Generating Business Value from A/B Testing (ENG)

Hilda Kosorus

Date: November 29, 2017

Location: University of Applied Social Sciences. Kalvarijų str. 137E, LT-08221 Vilnius, Lithuania

About the workshop

Abstract
A/B testing can be a highly valuable tool when making data-driven business decisions. It is easy to use and efficient in terms of resources. However, it can also be very easily misused and, when not conducted properly, resources invested in experiments might go wasted. The goal of this training is to give the attendees a definitive and practical guide to A/B testing, starting with the very early stages of planning, designing, conducting and interpreting results. Accompanied by examples, we will learn about the basic theory of hypothesis testing, how to execute, learn from and iterate on experiments, and what pitfalls to look out for in order to avoid failure. At the end of the training, the attendees will be able to successfully conduct or guide others through an A/B test experiment.

Agenda
1. Introduction to the theoretical aspects of hypothesis testing (45min + 45min)
a. Terminology: definitions, role and examples
b. The frequentist approach of hypothesis testing: what it is and how it works
c. T-test (one-tailed vs. two-tailed)
d. Chi-squared test for independence
e. Demystifying the p-value: common misconceptions
f. Power analysis: determining how long to run a test
g. Disadvantages of frequentist approaches
h. Introduction to Bayesian A/B testing
2. The definitive A/B testing guide (45min + 45min)
a. The 4 + 1 steps of A/B testing
b. Planning and designing experiments correctly
c. The A/B testing checklist
d. Identifying areas of opportunity
e. Defining a hypothesis
f. Designing experiments
g. Iterating and learning
h. Evaluating and interpreting results
i. Pitfalls to avoid
3. Group work (45min + presentation)
a. Planning and designing your own A/B test from scratch using the previous learnings and guide
b. Presenting results and learning from each other

Objective
The goal of this workshop is to help attendees understand the statistical background of experimentation, provide them with a definitive guide to correctly plan, design and evaluate A/B tests. Accompanied with a plethora of examples, the attendees will leave at the end fully equipped to conduct and generate business value from experiments.

Target audience
This workshop is intended to data scientist, data analysts or business users who are planning to integrate A/B testing in their daily work and want to use the power of experimentation to create value for the business.

Course prerequisites
A personal computer with internet connection.

About the trainer
Hilda Kosorus is Data Scientist at Runtastic. Coming from the academic and research world after finishing her PhD in Computer Science, she took on the challenge of laying the Data Science foundation at Runtastic. She focuses on bridging the gap between analytics and other departments, creating value with every experiment and data product and helping Runtastic become more data-driven.