Georgi Georgiev
Apr 23, 2021

--

This solution does not make sense at the level of an individual test for a number of reasons. A big one is that continuous measurement does not work well with the requirement for getting representative samples. Another big one is that none of the business metrics relevant to each test are considered. It doesn't make sense to use business-level metrics at all. For a list of metrics one should consider including in a decision-making model surrounding A/B tests, I'd suggest this as a starting read: https://blog.analytics-toolkit.com/2017/risk-vs-reward-ab-tests-ab-testing-risk-management/

--

--

Georgi Georgiev
Georgi Georgiev

Written by Georgi Georgiev

Applied statistician and optimizer by calling. Author of “Statistical Methods in Online A/B Testing”. Founder of Analytics-Toolkit.com and GIGAcalculator.com.

No responses yet