AI Driven AB Testing Without False-Positive Damage

AI Driven AB Testing Without False-Positive Damage

Most AI testing wins are statistical artefacts. The platform declares a winner on day three, the brand ships the variant, and 90 days later the cohort that bought the winning variant repurchases at a lower rate than the control cohort would have. Nobody notices, because nobody re-audits, and the ledger of false-positive winners just keeps growing inside the test history.

The pitch the bandit vendors run is seductive. Faster wins, less wasted traffic, automatic reallocation. The pitch is also the problem. The faster you declare a winner, the less time the purchase cycle has to play out, and the more your "winner" is actually a sample of early-session shoppers who behave nothing like the cohort that determines next quarter's revenue.

The Five-To-Tenfold False-Positive Tax

The math on early stopping is not a hot take. It is settled experimental statistics, and it has been published as such for a decade. Kohavi book summary walks through Ron Kohavi's published guidance from the trustworthy online experiments literature. Continuously monitoring p-values during a test and stopping early as soon as significance is reached produces a five-to-tenfold inflation in false-positive rate compared to a fixed-horizon test. That is not a rounding error. That is the math an AI bandit runs by default, dressed up as Bayesian inference and hidden behind a vendor dashboard that congratulates the operator on shipping a winner that is statistically indistinguishable from random.

CXL bandit guide is the cleanest practitioner write-up of when bandits are appropriate and when they are not. The short version: bandits work for short-cycle decisions where the conversion event happens inside the same session and there is no downstream behavioural cost to picking the wrong arm. They fail for long-cycle decisions where the variant changes repeat-purchase rate, refund rate, or AOV in ways that only show up over weeks. Physical product brands run almost exclusively in the second category. Their tests are not "which colour button gets clicked." Their tests are "which landing page produces customers who buy again," and that question cannot be answered in three days regardless of how sophisticated the model is.

VWO MAB guide is the vendor's own explanation of adaptive traffic allocation, which is useful because it names the mechanism precisely. The bandit allocates progressively more traffic to the leading arm based on conversions observed so far. The mechanism is doing exactly what it says on the tin. The mechanism is also why the test cannot answer the question physical product operators actually need answered, because by the time the purchase cycle has played out, 95 percent of the traffic has already been routed to a variant chosen on the basis of three-day data.

The damage compounds in two places. The first is shipped variants that look like winners on day three and rot the cohort by day ninety. The second is variants that would have been long-term winners but lost the early traffic battle and were declared losers before their cohort effect emerged. Both errors land on the same brand's P&L, and both are invisible inside the test history because the test history records the day-three declaration, not the 90-day audit.

Contextual bandit glossary gives the textbook definition of contextual bandits. The contextual variant adds personalisation: different visitors see different arms based on context features. The personalisation makes the false-positive problem worse, not better, because it splits the already-thin sample into smaller cells, each of which now has even less time to accumulate enough purchase-cycle data to support a real declaration.

The platform vendors know this. VWO guardrail docs is VWO's own admission that guardrail metrics are required to prevent bandits from damaging primary KPIs while chasing the optimisation target. Guardrails are not a feature. Guardrails are an admission that the bare bandit is unsafe.

The Bandit Discipline Protocol

The replacement is The Bandit Discipline Protocol. The principle is single-sentence simple: AI testing only produces trustworthy wins when the experiment guarantees a minimum-duration window that covers the purchase cycle and reserves a fixed control cell for repeat-buyer measurement, with a 90-day cohort re-audit gate before any change is locked in.

The Protocol has three constraints. First, a minimum-duration floor that is set by the brand's actual purchase cycle, not by the platform's default. Second, a fixed control cell that the bandit cannot allocate away from, sized large enough to support repeat-buyer measurement at 90 days. Third, a 90-day cohort re-audit gate, where every declared winner is re-tested against its control cohort on contribution margin per repeat buyer before the variant is locked in.

I have run the Protocol on enough Shopify Plus brands now that the failure mode is predictable. The merchandising lead wants to ship the day-three winner. The CRO lead wants to start the next test immediately. The Protocol holds both in check by refusing to lock the variant before the 90-day cohort has been audited. The discipline is mechanical, not judgemental. The platform can declare whatever it wants. The Protocol decides what gets shipped.

The Protocol does not replace the bandit. It runs on top of any bandit tool, including the major personalisation engines, VWO, and the Shopify-native experimentation layer. VWO test duration is the vendor's own admission that even Bayesian tests need a one-to-two-week minimum duration to be reliable, and the Protocol takes that admission and extends it to the actual purchase cycle, which on most physical product brands is closer to four to six weeks.

MAB vs AB ecommerce is the academic backbone. The KDD paper compares multi-armed bandits and classical A/B testing on confidence interval and power across ecommerce conditions, and the conclusion lines up with what the Protocol enforces: the bandit's traffic-allocation speed is real, but that speed is purchased by giving up confidence in the long-tail behavioural metric. If the operator does not need the long-tail metric, the bandit is fine. If the operator needs to know whether the variant changed repeat-purchase rate, the bandit alone cannot answer the question and the Protocol's audit gate has to. The trade is rarely surfaced inside the platform UI, because the platform is selling speed and the operator buying speed has been led to believe the speed is free.

Phase 1: Configure the Guardrails (Days 1-30)

The first 30 days are about configuring the bandit so it cannot ship a false-positive winner without the Protocol catching it.

Week 1: measure your actual purchase cycle. Pull 12 months of order data and calculate the median time between a customer's first session and their first purchase, then the median time between their first purchase and their second. The minimum-duration floor on every test going forward is the longer of those two intervals. Most physical product brands land between 21 and 45 days. The default bandit duration of seven days is a fantasy.

Week 2: set the holdout cell. Reserve 10 to 15 percent of traffic as a fixed control that the bandit cannot allocate away from. The holdout's only job is to provide a clean baseline cohort for the 90-day audit. Smaller holdouts will not support the audit at the brand's volume; larger holdouts waste optimisation potential. Ten percent at $5M revenue is roughly the floor.

Week 3: define the secondary metrics. Primary metric is whatever the test is optimising. Secondary metrics, monitored as guardrails, must include AOV, refund rate, and 30-day repeat-purchase rate. The bandit can declare a winner on the primary; the Protocol will block the winner from shipping if any guardrail moves more than 1.5 standard deviations against the brand's baseline.

Week 4: lock the duration. Every test now ships with a minimum-duration parameter that matches your purchase cycle, not the platform's default. Shopify Rollouts limits is the practitioner critique of Shopify's own bandit testing and is required reading if your brand is on Plus, because the native tool does not give you the duration controls the Protocol needs and you will need to wrap it in your own scheduling layer.

The deliverable at end of Phase 1 is a written test charter. Every new experiment opens with the charter filled in: primary metric, minimum duration, holdout size, secondary metric guardrails, audit date 90 days post-launch. No charter, no test. No test, no shipped variant.

Phase 2: The 90-Day Cohort Re-Audit (Month 2-6)

Phase 2 is where the Protocol earns its keep, because Phase 2 is where the false-positive ledger gets cleaned out.

Month 2: pull every winner shipped in the prior quarter. For each one, identify the cohort of customers who first transacted on the winning variant and the matched cohort on the holdout cell. Calculate contribution margin per repeat buyer at 60 days and 90 days. The variants that survive the audit get locked in. The variants that do not survive get rolled back, and the rollback is a documented event the team reviews.

Month 3: build the re-audit dashboard. The dashboard has one row per shipped variant, with columns for declared lift (day three), audit lift at 60 days, audit lift at 90 days, and survival status. The dashboard is not optional. Without the dashboard, the team forgets which variants are still under audit and the rollbacks get skipped under launch pressure.

Month 4: instrument the rollback workflow. Most teams have never rolled back a shipped variant. The first rollback is operationally painful and politically painful. The Protocol forces the rollback to be mechanical: if the 90-day cohort delta is negative, the variant rolls back, regardless of who shipped it or how confident the platform was. The mechanical rule is the only thing that survives political pressure.

Month 5 and Month 6: tune the false-positive rate. After 90 to 180 days of running the Protocol you will have a survival rate per test type. Tests on PDPs typically survive at higher rates than tests on collection pages. Tests on email subject lines survive at lower rates than tests on email body content. The survival rate by test type tells you where the bandit is producing real wins and where the bandit is producing artefacts, and the team can prioritise testing budget against the survivors.

The team running the Protocol is small. One CRO lead owns the test charter. One analyst owns the audit dashboard. The merchandising and email leads are accountable for the survival rate inside their own surface. Three named roles. One charter. One dashboard. One mechanical rollback rule. That is the entire build.

The North Star: Cohort Margin Delta, Not Session Conversion

The most damaging thing about the standard bandit deployment is the metric the dashboard reports. Single-session conversion lift is the wrong number to optimise on for any physical product brand whose economics depend on repeat-purchase rate, and the wrong number to optimise on quietly poisons every test the team runs.

The Bandit Discipline Protocol replaces single-session conversion with a single replacement metric: cohort margin delta at 90 days. Defined cleanly, it is the difference in contribution margin per repeat buyer between the cohort that first transacted on the winning variant and the cohort on the holdout cell, measured 90 days after the test was declared. Positive delta, the variant is real. Negative or zero delta, the variant is a false-positive and the platform's three-day declaration was wrong.

The brands that run the Protocol stop celebrating day-three declared winners and start measuring the only thing that actually matters. The bandit still allocates traffic. The Protocol still gets the speed benefit. The brand stops paying the false-positive tax, because the audit gate refuses to let false positives ship. The metric the team reports on Monday morning is no longer "we shipped X winners last quarter," it is "we shipped X winners and Y survived the audit," and the second number is the only one the CFO should care about.

You do not need a smarter bandit. You need a minimum-duration floor, a holdout cell, a 90-day audit, and a mechanical rollback rule. The platform's optimiser was always doing fine. The discipline around the optimiser was the missing piece, and the Protocol installs it.

The brands I have watched run the Bandit Discipline Protocol for two full quarters share a common pattern: their declared-winner volume drops by roughly half, their shipped-and-survived winner volume is stable or up, and their false-positive ledger goes from invisible to documented. The CRO lead complains about the slowdown for the first month and then stops complaining once the audit dashboard makes the rollbacks public.