A/B testing starts with a better question

Changing a button color is easy. Learning whether the change improves a business outcome takes a more deliberate process. A useful A/B test begins with a question about customer behavior: what is preventing the visitor from taking the next step, and why should a specific change help? That question determines the design, measurement and decision you can make afterward.

Write a hypothesis you can challenge

Describe the audience, observed friction, proposed change and expected outcome. Visitors to a service page may hesitate because the form does not explain what happens next. A version that explains the response process could improve completed, qualified enquiries. This is a hypothesis to investigate, not a promise that reassurance will increase conversions.

Choose the outcome before the experiment

Select one primary metric that reflects the decision. A form experiment might use completed enquiries per eligible visitor, with spam rate and lead quality as guardrails. Button clicks alone can mislead if fewer people finish. Define how events and repeat visits are recorded. Check instrumentation on both versions before the experiment.

Make assignment consistent

Visitors should receive a consistent experience, and the intended allocation should be checked against the actual traffic split. Decide whether assignment uses a visitor, account or another unit that matches the question. A logged-in customer using several devices may need a different approach from an anonymous landing-page visitor. Document the choices.

Plan the analysis and stopping rule

Estimate the traffic and effect size needed to make the experiment informative. Choose the analysis method and stopping criteria in advance. Repeatedly checking a conventional significance test and stopping at the first attractive result can produce unreliable conclusions. A sequential method needs its own appropriate rules. If traffic cannot support a useful test, consider interviews or usability research instead.

Keep a decision record

Record the hypothesis, variants, audience, allocation, dates, metric definitions and incidents. Review guardrails before rolling out a change. An inconclusive result can identify a weak hypothesis or measurement problem; it should not be rewritten as a success. Trustworthy experimentation combines engineering and analysis choices throughout the process.

A worked measurement example before a real experiment

Consider a fictional service enquiry test. Version A keeps the existing form. Version B explains the next step and asks for the same information. The primary outcome is a successfully accepted enquiry per eligible assigned visitor. A submit-button click is only a diagnostic event. This example checks counting rules; it does not estimate lift or supply a sample-size recommendation.

Write the event contract before collecting results. Keep an assignment ID, variant, eligibility flag, first exposure time and an accepted-enquiry event ID. Avoid putting names, email addresses or the enquiry message into analytics. Establish how your actual consent setup affects measurement, and keep analytics separate from the operational record needed to answer the enquiry.

• Visitor one sees A, submits once and retries after a slow response. Both responses refer to the same accepted enquiry. Count one eligible visitor and one conversion.
• Visitor two sees A and leaves. Count one eligible visitor and no conversion.
• Visitor three sees B, clicks Submit, receives a validation error and leaves. Count one eligible visitor and no conversion.
• An internal tester sees B. Exclude that person according to the eligibility rule established before launch, rather than deleting inconvenient results later.

The synthetic fixture therefore contains two eligible A visitors with one accepted enquiry and one eligible B visitor with none. Those tiny counts are deliberately unsuitable for a winner decision. Their job is to reveal a faulty event definition: counting every retry or treating a validation error as success would give the wrong result before statistics enter the discussion.

Run similar checks against the real implementation in a safe test environment: repeat a request, revisit with the same assignment, reject invalid input, and confirm the guardrail events. Check the assignment ratio and missing events independently. Correct instrumentation before launching the experiment; do not repair a broken test by quietly changing the conversion definition halfway through it.

Use the GA4 experiment measurement guide to separate assignment from reporting. For a seasonal campaign, the offer-to-order checklist helps identify the operational promises that must stay consistent across both variants.

Updated 19 October 2026: added a practical example and implementation checks. Examples are illustrative unless a specific studio measurement is identified.

Continue reading

• Building a contact form that keeps the conversation moving
• Improve product discovery before redesigning your store