In the applied sciences, A/B testing is known as hypothesis testing. It is used particularly in medicine where it is at the heart of testing the efficacy of new pharmaceuticals, for example. The fact that A/B testing is an axiomatic feature of TMI’s campaign management is not exceptional. It is simply obligatory for agencies to A/B test variations in elements of our clients’ digital marketing campaigns – particularly variations in text ad copy and creative content in general.
All the search engine advertising and trafficking platforms make it very user-friendly for campaign managers to set up A/B tests. It is similarly easy to report on the results of these tests. However, TMI feels beholden to the underlying methodology of hypothesis testing as a scientific process that advances knowledge. A/B test data must be presented in terms of not only the variations in, for example, click-through-rates or conversion rates, that result when different ad copy is tested in the ad auctions.
Far more crucially, the A/B test results must be examined to determine whether the variations are random or statistically significant fluctuations. That is to say, A/B testing only means something for the client if agencies can determine that the variations tested formed the underlying causation of the different outcomes in the engine metrics of interest to clients. Determination of statistical significance is how PPC becomes scientific and how knowledge replaces speculative inference for campaign managers at TMI.