by John Mount Ph.D.
Data Scientist at Win-Vector LLC
Our most recent article was a dynamic programming solution to the A/B test problem. Explicitly solving such dynamic programs is a long and tedious process, so you are well served by finding and introducing clever invariants to track (something better than just raw win-rates). This clever idea, called “sequential analysis”, was introduced by Abraham Wald (whom we have written about before). If you have ever heard of a test plan such as “first process to get more than 30 wins ahead of the other is the one we choose” you have seen methods derived from Wald’s sequential analysis technique.
Wald’s famous airplane armor problem
In this “statistics as it should be” article we will discuss Wald’s sequential analysis.
A particularly compelling (and unusual) type of test plan is the graphical sequential inspection procedure designed by Abraham Wald. Wald treats A/B testing as a “stopping problem.” Wald asks the business partner for four parameters: u0,u1 (the desired bounds on relative error allowed in the estimate); and alpha,beta (the power and significance goals). With these parameters Wald designs an inspection plan that is a single chart (shown below).
Figure 14, Section 6.4.2, page 111, Abraham Wald, Sequential Analysis, Dover 2004 (reprinting a 1947 edition).
The procedure
The chart is used as follows: we send traffic to processes 1 and 2 in matched pairs. We get back pair measurements (c1,c2) where c1 = 1 if process 1 paid off (0 otherwise) and c2 = 1 if process 2 paid off (zero otherwise). Only look at pairs (0,1) or (1,0) (we discard all (0,0) and (1,1) pairs, they are not allowed to update the chart). Start at the origin of the inspection chart. When you see a (0,1) or (1,0) pair move on the inspection …read more
Source:: r-bloggers.com