To be fair to Search Engine Land, its editors declaimed the opinions of guest writer Matt Van Wagner of Find Me Faster. Van Wagner’s article, The Pitfalls Of A/B Ad Split Testing, Part 2, might as well have been titled “The Pitfalls of Pointless Analysis,” given how it goes on about matters insignificant to the business of improving results through SEM.

Though A/B testing is crucial to success in this pursuit, it’s not a simple thing to get right. Beginners seeking pearls of wisdom in Van Wagner’s lengthy piece would be better off asking their engine sales rep (who are probably not accomplished testers, either, but then again some of them were trained at Razorfish).

Van Wagner offers a free lobster dinner (he’s from New Hampshire, where crustaceans are currency) for help with the common A/B test conundrum of a winning ad that performs poorly on its own.

He correctly identifies the problem — lack of a true A/B split among rotated ads — but gets woefully lost on the way to a solution, considering complications from custom ad serving, search histories, repeat queries and something he calls “over optimization” without identifying the classic culprit of a back-test failure.

He should have asked: Were any of the keywords in this problematic test on broad match?

By far, the most common faulty assumption in search A/B testing is that both ads are eligible to show on the same query set. This is only true on exact match. Beyond exact, the eligible query set expands (i.e. broad match gets broader) for the ad with the higher CTR. During a test attempt, the ad earning the higher CTR for the shared query set will seem to suffer a CTR reduction as the engine finds the maximum yield (for itself) via query-set expansion. Sustained A/B tests on broad match routinely “fail” to achieve a significant result as the algorithm automatically challenges the winner, driving its CTR down. The test isn’t really a failure, because increasing yield is what the algorithm is designed to do.

This scenario causes confusion about the value of A/B testing in search. But there is no controversy: Understanding how paid search works enables experts to test to our hearts’ content. And the learning that pours in from a correctly executed A/B testing program makes our clients enough wampum to buy their own lobster.

We’re so pleased to have been able to participate in a Q&A session with an experienced search engineer, we’re going to forgive the narcissism required to pack yet another all-day event almost entirely with content irrelevant to everyone not infatuated with Google. The agency-relations team saved this one by drafting Daniel Dulitz, one of its few engineers who works on both paid and natural search, to represent the company for a few precious minutes. There’s nothing like hearing how something works from someone who helped build it. It’s all the more satisfying given how much Razorfish has contributed, by driving the growth of search marketing within so many large organizations, to Google’s reinvention of media and advertising. Thanks, Google, for finally looping us back in.

Though we can’t disclose what we learned, we’re pleased to share that one sneaking suspicion we’ve had about Google was ruled out: Last year, when Google banned A/B testing of landing pages across multiple domains, it was NOT because Google forgot that the $14-billion-per-year SEM industry pretty much grew up on A/B testing.