SEO Experiments Without Lying to Yourself

SEO teams often call any before-and-after chart an experiment. That language creates more confidence than the design deserves. Search demand moves, competitors publish, result pages change, crawlers revisit URLs on uneven schedules, and analytics pipelines break.

You do not need perfect laboratory conditions to learn. You do need a question narrow enough to test, a comparison that can fail, and a result statement that matches the evidence.

Start with the decision

Write the decision the test will inform. “Improve rankings” is not a decision. “Should we add concise answer blocks to the remaining support articles?” is. The second question identifies a repeatable change and a population that could receive it.

Define what you will do for a positive, negative, and inconclusive result before looking at the data.

Choose comparable pages

Group pages by template, purpose, demand pattern, current performance, and business value. Do not compare a seasonal buying guide with an evergreen glossary page because both happen to be articles.

If possible, assign similar pages to changed and unchanged groups. When random assignment is impractical, document how the groups differ. That limitation belongs in the conclusion.

Freeze the change

Specify the exact title, content block, internal-link rule, schema change, or technical adjustment being tested. Save the affected URLs and deploy identity. Avoid changing templates, navigation, and copy at the same time.

Check that the change actually rendered, remained indexable, and was available long enough to be crawled. A completed task or deployment status is not evidence that the treatment reached every page.

Pick primary and guardrail metrics

Choose one primary measure tied to the hypothesis, such as non-brand impressions for the target query group. Add guardrails such as conversions, indexed coverage, page speed, or support completion. This prevents a narrow win from hiding a worse customer outcome.

Keep ranking averages in context. A small mean change can be driven by query mix, low-volume terms, or pages entering and leaving the report.

Set the observation window

Use enough pre-change history to see normal variation and a post-change window long enough for discovery. Account for weekday patterns and seasonality. Do not keep extending the window until a preferred result appears.

Record other events during the period: releases, outages, campaigns, major news, and measurement changes. They are candidate explanations, not footnotes to hide.

Read the distribution, not just the total

Inspect results by page and query cluster. If three large pages improved while most pages declined, the average may not support a broad rollout. Look for the conditions under which the treatment appears useful.

Compare changed pages with their own baseline and with the unchanged group. Neither comparison is perfect, but agreement between them is more informative than one chart alone.

Make “inconclusive” a valid result

A noisy or mixed result does not mean the test failed operationally. It means the evidence does not support a confident rollout. Keep the change only if it is independently useful and low risk, or restore the baseline.

Do not turn “we did not detect harm” into “the change improved SEO.” Those are different claims.

Write a result note another team can challenge

Include the hypothesis, page set, exact change, dates, primary metric, guardrails, exclusions, confounders, result, and next decision. Link to the raw report and deployment evidence.

The standard is not certainty. It is whether a skeptical colleague can reproduce the comparison and understand why the team chose its next move.