Skip to content
NanoFlags
Start free

Experiments

Experiments (A/B tests)

An experiment compares the two values of one percentage rollout rule of a flag. Your SDK logs who saw which value; your code reports what they did with track(); NanoFlags computes the difference, its confidence interval and whether it is significant. The rollout you already use is the assignment, so a feature flag becomes an experiment without a second system.

How an experiment works

  • Variants come from the rule. Units outside the rollout's percentage get the control value (the rule's "otherwise"), units inside it the treatment value. A 50% rollout is a 50/50 test.
  • Assignment is sticky by unit. The rule buckets each unit by its context attribute (key by default, or the rule's bucketBy, such as accountId) with a deterministic hash, so the same unit always gets the same variant on every server, browser and SDK.
  • Exposure = the first evaluation. The first time a unit evaluates the flag through the experiment's rule, the SDK logs an exposure. Only exposed units are analysed, so users who never reached the feature do not dilute the result. Exposures are never sampled and are counted once per unit.
  • Traffic. An optional share (0–100%) of the rule's units is enrolled, using a second, independent hash. Units outside it get the same values but are not counted.
  • It runs through a publish. Starting an experiment adds its metadata to the environment's draft; SDKs start logging once you publish that draft, with your usual review and approvals. Stopping takes effect for the analysis immediately.

Set one up

  1. In the flag's draft, add a rule that serves your new value to a percentage (for example 50%) of the users you want to test on.
  2. Create the metrics you will decide on in Experiments → Metrics catalog (see below), and send their events from your code with track().
  3. Experiments → New experiment: choose the flag and the rule, write the hypothesis, pick one primary metric and any secondary or guardrail metrics, and the analysis (fixed horizon or sequential).
  4. Start the experiment, then publish the environment. Exposures appear within minutes; results are recomputed every 10 minutes.
  5. When you have an answer, stop it and record the decision. Then change the rule (for example to 100% treatment) and publish.
Server and browser: nothing changes in how you read the flag
const onePage = client.boolean("checkout-one-page", false, { key: user.id }); // logs the exposure
// …
client.track("checkout_completed", { key: user.id });               // conversion metric
client.track("order_total", order.total, { key: user.id });         // value metric
Do not change the rule during the experiment
Changing the percentage or the values while it runs changes who is in which variant over time and can bias the result. The experiment page warns when the draft rule no longer matches the rule the experiment started with.

Metrics

A metric names an event and how it is counted per exposed unit. Units without the event count as 0.

KindPer unitCompared with
Conversion1 if the unit sent the event at least once, else 0Two-proportion z-test
CountNumber of eventsWelch's t-test
ValueSum of the events' values (optionally capped per unit)Welch's t-test

An event counts for a unit when it arrives after the unit's first exposure, or up to 10 minutes before it (exposure and event often come from different processes). Events that arrive after the experiment stopped are not counted. Each metric has a desired direction, so "better" and "worse" mean what you expect for error rates too. Ratio metrics (for example revenue per session) are planned.

Reading the results

  • Per variant: exposed units and the metric's value (conversion rate or mean per unit).
  • Difference from the control with its confidence interval (95% at the default alpha of 0.05), and the relative lift with a delta-method interval.
  • p-value, two-sided. A result is Better or Worse only when it is significant at the experiment's alpha; with more than one treatment, alpha is Bonferroni-corrected.
  • Guidance: the sample size per variant needed to detect your minimum detectable effect with 80% power, progress toward the planned sample, and warnings for small samples (fewer than 100 units, or fewer than 10 conversions per variant).

Peeking and sequential testing

A fixed-horizon test (the default) is valid when you read it once, at the sample size you planned. Checking every day and stopping the first time p drops below 0.05 makes a false positive several times more likely than 5%. Plan the sample size, wait for it, then decide.

If you need to stop early, choose sequential analysis (Solo plan and above). It uses a mixture sequential probability ratio test (mSPRT): its p-value and confidence interval are valid whenever you look, so you may stop as soon as the result is significant. The price is wider intervals than a fixed-horizon test at its planned size.

CUPED

CUPED (Scale plan) removes the part of each unit's result that its own behaviour before the experiment already predicts, using the same metric over a pre-period (1–30 days, default 14) before the unit's first exposure. The estimate stays unbiased; its variance shrinks by roughly the squared correlation between the two periods, which often means the same answer with far fewer units. It needs metrics with Collect history turned on (NanoFlags then keeps each unit's daily value for 31 days). The results page can show adjusted and unadjusted numbers side by side.

Sample ratio mismatch

NanoFlags checks the observed split of units against the configured one with a chi-square test and flags a mismatch at p < 0.001. A mismatch means something assigns or logs units unevenly (bots, redirects that lose one variant, logging the exposure in one code path only), and the results should not be trusted until it is explained. Units later seen with the other variant (the rule changed, or an unstable unit attribute) stay in their first variant and are reported.

Privacy

  • SDKs send a SHA-256 hash of the unit identity, scoped to the project, environment and attribute — never the identity, the context, other attributes or flag values. Raw identities are refused by the edge.
  • NanoFlags stores only a keyed HMAC of that hash, so stored ids cannot be matched to people even by someone who can compute the hash for every known user.
  • A track event carries an event name, an optional number and the hashed unit. Nothing else is accepted or stored. Per-unit data is deleted when the plan's retention after the experiment ends has passed; aggregated results are kept.

Plans and limits

ExploreSoloTeamScale
ExperimentsIncluded (fixed horizon)IncludedIncludedIncluded
Running at the same time (organization)12550
Experiment events per month (organization: exposures + track events)1,000,00010,000,000100,000,000500,000,000
Technical guard per environment and day1,000,0002,000,0002,000,00025,000,000
Per-unit data kept after an experiment stops90 days90 days180 days400 days
Sequential testing—IncludedIncludedIncluded
CUPED and metric history———Included

Experiment events are monthly credits for the whole organization (UTC calendar month): every accepted exposure and track event counts once, in any project or environment. The experiments pages show "X of Y events this month". Enterprise credits are set by contract, and extra blocks of 100 million events a month are available on request for Team and Scale.

When the credits run out, collection continues up to 150% for the rest of that month, once per calendar quarter, at no charge. After that, exposures and events are not counted until the next month starts (the ingest answers "capped": true); flags keep evaluating normally and nothing is billed automatically. The per-environment daily guard only protects against runaway traffic. Existing experiments and results stay readable after a downgrade, and a running experiment can always be stopped.