To be fair to Search Engine Land, its editors declaimed the opinions of guest writer Matt Van Wagner of Find Me Faster. Van Wagner’s article, The Pitfalls Of A/B Ad Split Testing, Part 2, might as well have been titled “The Pitfalls of Pointless Analysis,” given how it goes on about matters insignificant to the business of improving results through SEM.
Though A/B testing is crucial to success in this pursuit, it’s not a simple thing to get right. Beginners seeking pearls of wisdom in Van Wagner’s lengthy piece would be better off asking their engine sales rep (who are probably not accomplished testers, either, but then again some of them were trained at Razorfish).
Van Wagner offers a free lobster dinner (he’s from New Hampshire, where crustaceans are currency) for help with the common A/B test conundrum of a winning ad that performs poorly on its own.
He correctly identifies the problem — lack of a true A/B split among rotated ads — but gets woefully lost on the way to a solution, considering complications from custom ad serving, search histories, repeat queries and something he calls “over optimization” without identifying the classic culprit of a back-test failure.
He should have asked: Were any of the keywords in this problematic test on broad match?
By far, the most common faulty assumption in search A/B testing is that both ads are eligible to show on the same query set. This is only true on exact match. Beyond exact, the eligible query set expands (i.e. broad match gets broader) for the ad with the higher CTR. During a test attempt, the ad earning the higher CTR for the shared query set will seem to suffer a CTR reduction as the engine finds the maximum yield (for itself) via query-set expansion. Sustained A/B tests on broad match routinely “fail” to achieve a significant result as the algorithm automatically challenges the winner, driving its CTR down. The test isn’t really a failure, because increasing yield is what the algorithm is designed to do.
This scenario causes confusion about the value of A/B testing in search. But there is no controversy: Understanding how paid search works enables experts to test to our hearts’ content. And the learning that pours in from a correctly executed A/B testing program makes our clients enough wampum to buy their own lobster.