Experiments
Experiments (A/B tests)
track(); NanoFlags computes the difference, its confidence interval and whether it is significant. The rollout you already use is the assignment, so a feature flag becomes an experiment without a second system.On this page
How an experiment works
- Variants come from the rule. Units outside the rollout's percentage get the control value (the rule's "otherwise"), units inside it the treatment value. A 50% rollout is a 50/50 test.
- Assignment is sticky by unit. The rule buckets each unit by its context attribute (
keyby default, or the rule'sbucketBy, such asaccountId) with a deterministic hash, so the same unit always gets the same variant on every server, browser and SDK. - Exposure = the first evaluation. The first time a unit evaluates the flag through the experiment's rule, the SDK logs an exposure. Only exposed units are analysed, so users who never reached the feature do not dilute the result. Exposures are never sampled and are counted once per unit.
- Traffic. An optional share (0–100%) of the rule's units is enrolled, using a second, independent hash. Units outside it get the same values but are not counted.
- It runs through a publish. Starting an experiment adds its metadata to the environment's draft; SDKs start logging once you publish that draft, with your usual review and approvals. Stopping takes effect for the analysis immediately.
Set one up
- In the flag's draft, add a rule that serves your new value to a percentage (for example 50%) of the users you want to test on.
- Create the metrics you will decide on in Experiments → Metrics catalog (see below), and send their events from your code with track().
- Experiments → New experiment: choose the flag and the rule, write the hypothesis, pick one primary metric and any secondary or guardrail metrics, and the analysis (fixed horizon or sequential).
- Start the experiment, then publish the environment. Exposures appear within minutes; results are recomputed every 10 minutes.
- When you have an answer, stop it and record the decision. Then change the rule (for example to 100% treatment) and publish.
const onePage = client.boolean("checkout-one-page", false, { key: user.id }); // logs the exposure
// …
client.track("checkout_completed", { key: user.id }); // conversion metric
client.track("order_total", order.total, { key: user.id }); // value metricMetrics
A metric names an event and how it is counted per exposed unit. Units without the event count as 0.
| Kind | Per unit | Compared with |
|---|---|---|
| Conversion | 1 if the unit sent the event at least once, else 0 | Two-proportion z-test |
| Count | Number of events | Welch's t-test |
| Value | Sum of the events' values (optionally capped per unit) | Welch's t-test |
An event counts for a unit when it arrives after the unit's first exposure, or up to 10 minutes before it (exposure and event often come from different processes). Events that arrive after the experiment stopped are not counted. Each metric has a desired direction, so "better" and "worse" mean what you expect for error rates too. Ratio metrics (for example revenue per session) are planned.
Reading the results
- Per variant: exposed units and the metric's value (conversion rate or mean per unit).
- Difference from the control with its confidence interval (95% at the default alpha of 0.05), and the relative lift with a delta-method interval.
- p-value, two-sided. A result is Better or Worse only when it is significant at the experiment's alpha; with more than one treatment, alpha is Bonferroni-corrected.
- Guidance: the sample size per variant needed to detect your minimum detectable effect with 80% power, progress toward the planned sample, and warnings for small samples (fewer than 100 units, or fewer than 10 conversions per variant).
Peeking and sequential testing
A fixed-horizon test (the default) is valid when you read it once, at the sample size you planned. Checking every day and stopping the first time p drops below 0.05 makes a false positive several times more likely than 5%. Plan the sample size, wait for it, then decide.
If you need to stop early, choose sequential analysis (Solo plan and above). It uses a mixture sequential probability ratio test (mSPRT): its p-value and confidence interval are valid whenever you look, so you may stop as soon as the result is significant. The price is wider intervals than a fixed-horizon test at its planned size.
CUPED
CUPED (Scale plan) removes the part of each unit's result that its own behaviour before the experiment already predicts, using the same metric over a pre-period (1–30 days, default 14) before the unit's first exposure. The estimate stays unbiased; its variance shrinks by roughly the squared correlation between the two periods, which often means the same answer with far fewer units. It needs metrics with Collect history turned on (NanoFlags then keeps each unit's daily value for 31 days). The results page can show adjusted and unadjusted numbers side by side.
Sample ratio mismatch
NanoFlags checks the observed split of units against the configured one with a chi-square test and flags a mismatch at p < 0.001. A mismatch means something assigns or logs units unevenly (bots, redirects that lose one variant, logging the exposure in one code path only), and the results should not be trusted until it is explained. Units later seen with the other variant (the rule changed, or an unstable unit attribute) stay in their first variant and are reported.
Privacy
- SDKs send a SHA-256 hash of the unit identity, scoped to the project, environment and attribute — never the identity, the context, other attributes or flag values. Raw identities are refused by the edge.
- NanoFlags stores only a keyed HMAC of that hash, so stored ids cannot be matched to people even by someone who can compute the hash for every known user.
- A track event carries an event name, an optional number and the hashed unit. Nothing else is accepted or stored. Per-unit data is deleted when the plan's retention after the experiment ends has passed; aggregated results are kept.
Plans and limits
| Explore | Solo | Team | Scale | |
|---|---|---|---|---|
| Experiments | Included (fixed horizon) | Included | Included | Included |
| Running at the same time (organization) | 1 | 2 | 5 | 50 |
| Experiment events per month (organization: exposures + track events) | 1,000,000 | 10,000,000 | 100,000,000 | 500,000,000 |
| Technical guard per environment and day | 1,000,000 | 2,000,000 | 2,000,000 | 25,000,000 |
| Per-unit data kept after an experiment stops | 90 days | 90 days | 180 days | 400 days |
| Sequential testing | — | Included | Included | Included |
| CUPED and metric history | — | — | — | Included |
Experiment events are monthly credits for the whole organization (UTC calendar month): every accepted exposure and track event counts once, in any project or environment. The experiments pages show "X of Y events this month". Enterprise credits are set by contract, and extra blocks of 100 million events a month are available on request for Team and Scale.
When the credits run out, collection continues up to 150% for the rest of that month, once per calendar quarter, at no charge. After that, exposures and events are not counted until the next month starts (the ingest answers "capped": true); flags keep evaluating normally and nothing is billed automatically. The per-environment daily guard only protects against runaway traffic. Existing experiments and results stay readable after a downgrade, and a running experiment can always be stopped.