Statistical analysis report with hypothesis tests and effect sizes

# Checkout redesign A/B test — analysis plan and interpretation ## What the design supports Two independent groups, roughly 6,200 per arm, with a binary primary outcome (completed purchase) and a continuous secondary outcome (order value, right-skewed as order values almost always are). Randomisation at the session level with a 21-day run covering three full weekly cycles is sound. One caveat that limits every conclusion below: sessions, not users, were randomised, so a returning visitor could land in both arms. That biases the estimate toward zero, which means a positive result is conservative and a null result is harder to interpret. ## Descriptive layer to report first For each arm: conversions, sessions, conversion rate with a Wilson score interval (not a normal approximation — it behaves better at low rates), and for order value the median with an interquartile range alongside the mean, since the mean alone will be pulled by the tail. ## Test selection **Primary outcome.** Two-proportion z-test, or equivalently a chi-square test of independence on the 2×2 table. Both are appropriate here: the outcome is binary, groups are independent, and expected cell counts are far above the threshold where Fisher's exact test would be needed. Report the absolute difference in percentage points with a confidence interval — that is the number the decision actually turns on. Relative lift is fine as a headline but should never appear without the absolute figure. **Secondary outcome.** Welch's t-test on log-transformed order value, or a Mann-Whitney U test on the raw values. Welch's is preferred over Student's because equal variances should not be assumed across arms. Report Cohen's d on the transformed scale, then translate it back into currency, because d alone will not tell a stakeholder whether the change pays for itself. ## Assumptions to check before trusting either result Sample ratio mismatch first: if the two arms differ from the intended split by more than chance allows, stop and fix the assignment before interpreting anything else. Then check that conversion rates are stable across the three weeks in each arm — a result driven entirely by one promotional weekend is not a result about the redesign. ## How to read the outcome With arms of this size the test is well powered for a moderate lift but underpowered for a small one, so a non-significant result should be reported as "no effect large enough to detect at this sample size", not as "no effect". State the minimum detectable effect explicitly in the write-up. Most importantly: compare the lower bound of the confidence interval against the engineering cost of shipping the redesign. A statistically significant lift whose interval spans values below your break-even point is not yet a decision to ship — it is a decision to run longer. **Inputs still needed:** conversion counts per arm and the order value summary statistics. Supply those and this plan becomes a completed analysis.