Experiments and Version Comparison

Use experiments to compare measured outcomes for variants your site actually serves. Pintelly records eligible exposures and outcomes; creating an experiment does not change your pages or automatically assign visitors to variants.

Before you start

1. Choose a specific question, a control and variant keys that match your site's variation logic.
2. Define a primary metric: a tracked element click, page view or custom event. Add an optional guardrail to watch a second outcome.
3. Make sure the site serves each variant and attaches the correct experiment tag, or pb_variant / pb_tag URL parameter, to the visitor's actual experience.
4. Check that the metric and variant exposure are collected with granted consent. Test each variant before comparing results.

Use draft while preparing, running for the active experiment, paused to pause the experiment workflow and completed when you finish. Changing the experiment's state in Pintelly does not remove or replace the variant content served by your site.

How to read outcomes

• Exposures indicate recorded exposure to a variant, not everyone who could have visited it.
• Metric hits and conversion rate depend on the chosen event and observed population. Verify that the event represents the action you intended.
• Lift compares outcomes with the control; a large lift from a small sample is not automatically a reliable winner.
• Confidence intervals and significance, where available, describe uncertainty in the observed comparison.
• Sample-ratio mismatch (SRM) flags a difference from the expected variant distribution. Investigate missing tags, uneven delivery or collection differences before declaring a winner.

Consent refusal, tracking settings, plan limits and available history affect coverage. Keep periods and filters comparable; review device segments for differences hidden by an overall rate.

Version comparison

Use version comparison to inspect changes around product versions and outcomes. Dates and observed differences can help you choose where to investigate; other traffic, seasonal or product changes can occur at the same time. A before/after comparison alone does not establish that the version caused the result.

If results are missing or inconsistent

Check the experiment state, variant keys/tags, actual site content, primary event definition, consent and date filters. An event with no matching exposure or a wrong tag can make the comparison misleading. Do not fix a reporting gap by assigning a variant label that the visitor did not actually see.

See Analytics for custom events and Consent for collection conditions.