SEO A/B Testing: How to Run Split Tests on Page Groups

How an SEO split test works

In a classic A/B test, the same page is shown to different users in two versions. That does not work in SEO: a search engine has a single "user", its crawler, and it sees one version of a page. Showing the crawler and people different content is cloaking.

So an SEO test splits pages, not visitors. You take many pages of the same type (product pages, articles, city pages), divide them into two groups, change something only in the variant group and compare how its organic traffic moved relative to the control group.

When an SEO test is possible at all

• Many similar pages on one template. Dozens at the very least, hundreds is better. On five pages, noise will drown any effect.
• Enough organic traffic. If the pages barely get impressions, you will not see a difference.
• The change can be applied to a subset. The template must be able to serve a different version depending on the group.

How to pick control and variant groups

1. Select a homogeneous pool. One page type, one template, one language version.
2. Look at traffic history. You need data from before the test to understand how the groups behaved.
3. Split in a stratified way, not purely at random. Sort pages by traffic and assign them to groups alternately, so both have strong and weak pages.
4. Check that groups move together. Before the test, the groups' traffic curves should move in sync. If they do not, reshuffle.
5. Exclude anomalies. Pages with one-off spikes, seasonal products or recent major changes are better removed from the pool.

What to test

One rule: one change per test. If you change titles and internal links at the same time, you will not know what worked.

How long to run the test

There is no fixed duration; it depends on several factors:

• recrawl speed: until the crawler has seen the change on most variant pages, the test has not really started;
• traffic volume: less traffic means more time to collect data;
• effect size: small changes need longer to stand out from the noise;
• weekly cycles: the test should cover whole weeks.

Check in Search Console or Yandex Webmaster that the changed pages have actually been recrawled, and only then start counting.

How to read results against seasonality

Comparing "before" and "after" for the variant group alone is misleading: traffic may have grown because of the season, an algorithm update or an ad campaign. That is what the control group is for.

1. Build a forecast. Use the pre-test period to establish how the variant group normally relates to the control group.
2. Compare actuals to the forecast. If after the change the variant group clearly beats the forecast while the control behaves as before, the effect is probably real.
3. Look at the confidence interval, not a single number. Methods such as causal impact analysis of time series or difference-in-differences are used for this.
4. Account for external events. A search engine update during the test may affect the groups differently; note it in the report.
5. Separate metrics. Clicks, impressions and average position can move differently: more impressions without more clicks means something other than a higher CTR.

Common mistakes

• Groups that are too small and conclusions drawn from the first few days.
• Several changes in one test.
• Different versions for the crawler and for people.
• Ignoring that the change has not been indexed yet.
• Rolling out site-wide without respecting a neutral or negative result: "no growth" is a result too.