Experiments
Experiments compare a control against one or more variants of a campaign and report the evidence before you change anything permanently. Open Dashboard → Experiments, or select Create experiment from the Overlays or Recommendations list.
Supported surfaces
| Surface | What each variant is | Variants |
|---|---|---|
| Recommendations | A recommendation configuration | 2–5 groups |
| Overlays | Overlay content and settings; also editable in the overlay editor's A/B test variants workspace | 2–4 groups |
| Chatbots | A complete chatbot campaign; the control must be the source campaign | 2–4 groups |
Every experiment has exactly one control. Traffic split across groups must total 100% (10,000 basis points).
Plan an experiment
- Select Create experiment and choose the surface, website, and source campaign.
- Add variants and set the traffic split.
- Choose the assignment unit: Device ID or Signed-in customer.
- Choose the primary metric: click-through rate or conversion rate. Add up to 5 secondary metrics and up to 10 guardrails.
- Review the sample target (100 or more per group; the editor suggests a default), minimum duration, and conversion window (each up to 2,160 hours).
- Save the draft, then select Validate plan.
The dashboard plans at 95% confidence and 80% power.
Lifecycle
| State | Available actions |
|---|---|
| Draft | Validate plan, Archive experiment |
| Ready | Start experiment, Return to draft, Archive experiment |
| Running | Pause experiment, Complete experiment |
| Paused | Resume experiment, Complete experiment |
| Completed | Cancel hold (when a winner hold is active), Archive experiment |
When you complete an experiment, choose an eligible winner or Complete without selecting a winner. Completing does not change your campaigns. To adopt the winner, select Review winner in source campaign and apply the change there.
Read the evidence
The results page checks the experiment's health before showing lift:
- Assignment integrity — visitors stayed in their assigned group.
- Sample-ratio mismatch — the observed split matches the planned split.
- Attribution completeness — conversions could be tied to assignments.
- Result freshness — how recent the data is.
Lift is shown with a 95% confidence interval. Revenue is labeled descriptive: it is reported for context but is not the decision metric. Filter the experiment list by surface, website, and evidence health to find experiments that need attention.
Automatic decisions
When your workspace has it enabled, an experiment can make an Automatic decision. It is off by default and requires:
- exactly two groups split 50/50;
- explicit opt-in on the experiment;
- permission to place a winner hold.
An automatic decision can promote the treatment, keep the control, stop as inconclusive, or pause an invalid experiment. A winner hold serves the winning group while it is active and can be reversed with Cancel hold. It never rewrites the source campaign.
Partner experiments
Partner-scoped experiments that measure lift for AWIN or CJ traffic are managed through the API. See Partner attribution.