Skip to content
NanoFlags
Start free

Experiments

Guarded rollouts

A guarded rollout raises one percentage rollout rule of a flag step by step — for example 1% → 5% → 25% → 50% → 100%, each step held for a set time — while guards watch your error events and Datadog monitors. When a guard is breached, NanoFlags rolls the flag back on its own and tells the people responsible.

How it works

  • One rule, one environment. The rollout drives the percentage of a rollout rule of the flag. Units inside the percentage get the rule's value (the rollout); the rule's other units get its "otherwise" value and are the comparison group.
  • One review for the whole plan. Steps, hold times and guards are fixed when you create the rollout, and are approved once, through a normal change.
  • Every step is a signed revision. Later steps are published automatically as new revisions, signed and distributed like every publish, and recorded in the audit log as an automatic step of the approved rollout.
  • Only the percentage moves. A step publishes the environment's published configuration with only that rule's percentage changed. Pending draft work is never published by it.

Create one

  1. Give the flag a percentage rollout rule in the environment's draft (it can start at 0%).
  2. On the flag page choose Guarded rollout, pick the rule, the steps and their hold times, and add guards.
  3. Creating the plan sets the rule to the first step in the draft and opens a change for the flag. Its description spells out the plan.

Review and the first step

The change goes through the normal flow: request approval, an independent approver (not the person who created the plan, and not anyone who edited the draft since) approves it, and someone with the publish permission publishes it. That publish is step 1 and starts the rollout. Step 1 is never published without an independent approval, even in an environment that does not require approval: the rollout publishes later steps without asking again, so the plan itself must be reviewed.

The revision published right before step 1 is the safe revision: a rollback returns the flag to its configuration in it.

Guards

GuardReadsBreach
Error rateUnits first exposed during the current step, per side of the rule, and how many sent at least one bad event (an event name you choose, such as checkout_failed, sent with track())The bad-event rate of the rollout is significantly higher than the comparison group's by more than the allowed increase
Datadog monitorThe monitor's overall state, read with a Datadog connection from Settings → Integrations (API key and application key)The monitor is in Alert (or Alert or Warn, as you choose)
  • Valid under repeated looks. Guards are checked every minute. The error-rate guard uses a sequential test (mSPRT), so looking every minute does not raise the chance of a false alarm within a step. The sample restarts at each step, so earlier steps cannot hide a problem at a higher percentage.
  • Minimum sample. Until both sides have the minimum number of units, the guard reports "waiting for samples" and the step waits.
  • Exposures. The error-rate guard uses the exposure pipeline of experiments: NanoFlags starts a guard experiment on the rule with the rollout (or reuses the one running on it) and stops it when the rollout ends. Events count toward your experiment event credits.
  • At 100% there is no comparison group: error-rate guards do not apply to the last step, Datadog guards still do.
  • Health URL guards (an HTTPS endpoint you host) are planned and not available yet.
Datadog application key
Reading a monitor needs an application key besides the API key. Add it to the Datadog connection; it is encrypted like the API key and never shown again.

Automatic steps

Every minute NanoFlags checks each running rollout. A step moves up when its hold time has passed and every guard passes. If a guard cannot be read, the step waits; after 15 minutes in a row the rollout pauses and tells you. If anyone changes the rule outside the rollout, turns the flag off or uses the kill switch, nothing more is published and the rollout pauses. After the last step's hold time the rollout completes.

Rollback

A breached guard rolls back immediately, at any step and even during the hold time. The rollback publishes a new revision in which only this flag is restored to its configuration in the safe revision (other flags stay as published), and resets the rule's percentage in the draft. The guard experiment is stopped. To try again, fix the cause and create a new guarded rollout.

The rollback does not use the kill switch: the kill switch serves the off value to everyone, while a rollback returns to the reviewed state from before the rollout. You can still use the kill switch at any time.

A rollback never undoes what a person did. If the flag was turned off or the kill switch was used, nothing is published and the flag stays off. If someone published another edit to the flag during the rollout, that edit is kept and only the rule's percentage goes back to its safe value. The rollout page says which case applied.

Pause, resume, stop

  • Pause stops the automatic steps; guards are not checked. Resume restarts the current step's hold time and sample.
  • Roll back now restores the safe configuration, as an automatic rollback does. Stop here ends the rollout and keeps the current percentage.
  • These need the publish permission in the environment. A plan that has not started can be aborted; its change is closed.

Notifications and audit

  • A rollback or an automatic pause e-mails the rollout's creator and the flag owner, and owners and admins (unless they turned release-failure e-mails off).
  • Slack: the event rollout.rolled_back. Webhooks: rollout.rolled_back, and rollout.updated for every other step and state change.
  • The audit log records the plan, its approval, every step (as revision.published with kind: "rollout"), every pause, resume, rollback and the end. The rollout page shows a live timeline of every check, with the readings of each guard.

With AI agents and co-signing

  • AI agents. Automatic steps and rollbacks publish only the rule's new percentage on top of what is already published. They never publish the draft, so edits an AI agent proposed stay pending: they still need their own change, approved by a person, and a rollout never waits for them.
  • Customer co-signing. Automatic steps and rollbacks cannot carry your co-signature, so guarded rollouts are not available in an environment that requires customer co-signing, and co-signing cannot be turned on while a guarded rollout there is unfinished. The kill switch keeps working in co-signed environments.

Plans

Guarded rollouts are included in the Scale and Enterprise plans. On other plans the page shows what they do; rollouts you already have stay readable and can always be paused, rolled back or stopped. If the plan no longer includes them, automatic steps stop and the rollout pauses. See Plans and limits.