Generate testable A/B test hypotheses with clear success metrics, statistical rigor, and operator-ready experiment briefs.
You are an A/B testing strategist helping growth operators design rigorous experiments.
I'm planning an A/B test with these details:
**Product/Feature:** {{product_or_feature}}
**Current State:** {{current_state_or_metric}}
**Observation or Problem:** {{observation_or_problem}}
**Proposed Change:** {{proposed_change}}
**Target Audience:** {{target_audience}}
**Current Baseline Metric:** {{baseline_metric}} = {{baseline_value}}
**Expected Lift:** {{expected_lift_percent}}%
**Test Duration:** {{test_duration_days}} days
**Sample Size Available:** {{sample_size}}
Write a crisp A/B test hypothesis that includes:
1. **Hypothesis Statement** (one sentence, if-then format)
2. **Rationale** (why you expect this change to work; cite user research, competitor behavior, or prior test learnings if applicable)
3. **Primary Success Metric** (name, definition, target lift with confidence level)
4. **Secondary Metrics** (2–3 metrics to monitor for unintended effects)
5. **Statistical Plan** (minimum sample size needed, recommended test duration, significance threshold)
6. **Confounds & Controls** (what could skew results; how to isolate the change)
7. **Go/No-Go Criteria** (what result triggers rollout vs. iteration)
Be specific and operator-ready. Avoid generic language. Flag any assumptions that need validation before launch.1. Gather experiment context: what you're testing, why, and what success looks like. 2. Fill in all placeholders with specific, measurable values—avoid vague language. 3. Run the prompt to generate a structured hypothesis and statistical plan. 4. Share the output with your team and data analyst to validate sample size and duration. 5. Use the Go/No-Go criteria to decide rollout before the test ends. 6. Document results and learnings in your experiment backlog for future reference.
**Product/Feature:** Checkout flow (payment step) **Current State:** Checkout abandonment at payment step is 35% **Observation or Problem:** User interviews revealed the payment form feels too long and intimidating; users abandon after seeing 12 required fields **Proposed Change:** Reduce payment form from 12 to 6 fields by auto-filling address from shipping data and removing optional fields **Target Audience:** All checkout visitors, desktop and mobile **Current Baseline Metric:** Payment step completion rate = 65% **Expected Lift:** 8% **Test Duration:** 21 days **Sample Size Available:** ~15,000 checkout visitors per day
**Hypothesis Statement:** If we reduce the payment form from 12 to 6 required fields by auto-filling address and removing optional fields, then payment step completion rate will increase from 65% to 73% (8% relative lift) because shorter forms reduce cognitive load and perceived friction. **Rationale:** User interviews (conducted Q3 2024) identified form length as the top friction point. Competitor analysis shows Stripe Checkout and similar tools use 4–6 fields. Prior internal test (Q2 2024) showed removing one optional field increased completion by 1.2%; this change removes 6 fields total. **Primary Success Metric:** Payment Step Completion Rate: (users who submit payment form) / (users who reach payment step) × 100. Target: 73% (65% baseline + 8% lift). Confidence level: 95% (p < 0.05). **Secondary Metrics:** - Payment error rate (flag if auto-fill causes validation failures) - Cart abandonment rate (ensure users don't drop off earlier in flow) - Time to payment submission (expect 15–20% reduction) **Statistical Plan:** Minimum sample size: ~7,500 per variant (control + test) to detect 8% lift at 95% confidence, assuming 65% baseline and 20% variance. At 15,000 visitors/day, test duration: 21 days (sufficient for 2–3 weekly cycles). Significance threshold: p < 0.05. Use two-tailed test. **Confounds & Controls:** - Device type: stratify analysis by desktop/mobile; auto-fill behavior differs. - Payment method: ensure test includes all methods (card, PayPal, Apple Pay). - Traffic source: new vs. returning users may behave differently; segment analysis. - Timing: avoid major sales events or marketing campaigns that could inflate baseline. **Go/No-Go Criteria:** - **Rollout:** Payment completion ≥ 72% AND payment error rate ≤ 2% AND no significant drop in cart completion. - **Iterate:** Completion 66–71% (marginal; test longer or refine form further). - **Kill:** Completion < 66% OR payment error rate > 3% (auto-fill causing issues).