Getting A Higher A/B Test Win Rate Using Patterns: 2017 Q2 Performance Report

Continuing to measure the performance of our patterns as started in Q1 of this year, here is the follow up Q2 report. As a reminder we're trying to do better than 50/50 (win/lose) and we have managed to score a 60/40 win rate in this quarter based on 15 a/b tests.

Question 1: Can Conversion Patterns Improve Our Win Rate?

We're continuing to answer the key question: can conversion patterns improve our win rate and help us do better than 50/50? We have results from another 15 tests that we ran in Q2 of 2017. All 15 tests were based on conversion patterns that were predicted to win. That is, the patterns on which these tests were based on informed us that they had a positive median effect and a positive reproducibility score. Out of these 15 tests, 9 tests had a positive effect at the time of stopping. That's a 60% win rate. This is slightly lower than last quarter, but we're still doing better than chance.

Question 2: Does A Higher Reproducibility Lead To A Higher Win Rate?

The second question is whether patterns with a higher reproducibiltiy can be even more effective at predicting positive test outcomes? To answer this, we combined all of the Q1 and Q2 tests together to a total of 38 a/b tests and sorted by the degree of reproducibility (certainty). Then we compared both top and bottom halves of the predicted results. The top half (with the higher reproducibility scores) contained 12 winning tests, whereas the bottom half contained 11. So now we have +1 extra winning test from patterns with a higher repeatability (certainty) score. If this keeps up, it will mean that our approach to analyzing multiple a/b tests may pay off in the long run - we'll see.

Continued Optimism For The Future

This second assessment of conversion patterns continues to be very promising to me. In Q2 we continue to beat chance in our experimentation using past test data. Many patterns are also showing additional repeatability as we get collect data from more tests. Looking forward to doing this in Q3.

Get The Patterns That We Use To Win More Often