Blog & guides
Scenario checks: unit tests for flag changes
A scenario is a saved assertion about a flag: this context gets this value. How NanoFlags evaluates scenarios against a draft, why a passing run is bound to the exact draft checksum, who may produce a run that counts, and how to run them in CI.
NanoFlags engineeringPublished 4 min read
On this page
Most flag incidents are not caused by the code behind a flag. They are caused by a configuration change: a rule moved above another one, a segment edited for one campaign that another flag also uses, a rollout percentage typed into the wrong environment. The change is small, it looks right in review, and nobody ran anything against it. Scenario checks are the missing test step.
What a scenario is
A scenario is a saved check with three parts: a flag, an evaluation context, and the value you expect. For example: "a signed-in user in the US gets true for checkout-v2", or "an account on the enterprise plan gets the relevance-v2 ranking". You write it once, in the dashboard, next to the flag.
NanoFlags evaluates scenarios with the same evaluation engine the SDKs use, against either the draft of an environment (what you are about to publish) or its published revision (what is live now).
What "pass" means
- The actual value deep-equals the expected value: object key order is ignored, array order matters, and there is no type coercion (
"1"is not1). - A scenario whose flag does not exist (
flag_not_found) or whose evaluation errors never passes, even when it expectsnull. A deleted flag cannot silently satisfy its own tests. - Each run records, per scenario, the actual value and the reason the engine gave (
rule_match, a rollout, the default), so a failure tells you which rule answered.
Scenarios as a publish gate
An environment can require scenarios. Then a publish is accepted only when every scenario in scope has passed on the exact draft being published: the run records the draft's checksum, and the database compares it at publish time. Edit the draft after the run, even by one character, and the run no longer counts. In scope are the scenarios of the change's flags, or every scenario marked "include in CI" when publishing without a change.
Two rules keep the evidence honest:
- For a change, runs made with a CI key count, and so do dashboard runs by someone other than the change's author and the people who edited its draft. The author's own dashboard runs are kept as evidence but do not satisfy the check.
- Editing a scenario's flag, context or expected value creates a new version of the scenario. Runs of an earlier definition stop counting, so a test cannot be weakened after it passed.
Requiring scenarios, and recording CI runs, are part of the Team plan and above; on Explore you can still write scenarios and run them from the dashboard. New Production environments on Team and Scale start with the requirement turned on.
Running scenarios in CI
The CLI runs every scenario marked "include in CI" and exits non-zero when one fails:
$ NANOFLAGS_KEY=nf_ci_… nanoflags scenarios run --source draft
RESULT SCENARIO FLAG EXPECTED ACTUAL REASON
pass US signed-in gets v2 checkout-v2 true true rule_match
FAIL EU stays on v1 checkout-v2 false true rule_match
1 passed, 1 failed · source draft · checksum 3f2a… · published r12In GitHub Actions, the NanoFlags action wraps the same command, records the run against the commit and writes the table to the job summary. Use a CI key: it can run scenarios and read configuration, but it cannot publish, so a leaked CI key cannot change what your users get.
name: Flags
on:
pull_request: # NOT pull_request_target
push:
branches: [main]
permissions:
contents: read
jobs:
scenarios:
runs-on: ubuntu-latest
if: github.event_name != 'pull_request' || github.event.pull_request.head.repo.full_name == github.repository
steps:
- uses: actions/checkout@v4
- uses: actions/setup-node@v4
with:
node-version: 22
- uses: motazalmaroai/nanoflags-sdk/integrations/github-action@main # pin to a commit SHA in production
with:
key: ${{ secrets.NANOFLAGS_CI_KEY }}
source: draftUse source: draft to check what you are about to publish, or source: revision to check what is live. Any other CI system can call the CLI directly:
export NANOFLAGS_KEY="$NANOFLAGS_CI_KEY" # from your CI secret store
nanoflags scenarios run --source draft --commit "$COMMIT_SHA" --run-url "$BUILD_URL"
# exit 0: all passed · 1: a scenario failed · 2: key or usage problem · 3: NanoFlags unreachableWriting scenarios that catch real mistakes
- Pin the edges, not the middle. One scenario per rule boundary: the first context that should match a rule, and the nearest one that should not (the EU user next to the US rule).
- Assert the default. A context that matches no rule and expects the default value catches a new rule that is broader than intended.
- Cover shared segments from every flag that uses them. A segment edit is the classic "small change, big blast radius"; scenarios on each dependent flag turn it into a visible failure.
- Test the off state. If a flag is turned off in an environment, a scenario expecting its off value documents that this is intentional.
- Use a stable targeting key for rollouts. Bucketing is deterministic (MurmurHash3 of the rollout's seed and the key), so a scenario with a fixed key always lands in the same bucket and stays reproducible.
Reference: Concepts → Scenarios and CI and GitHub Actions. The release safety page shows how scenarios work together with reviewed changes and approvals.
- scenarios
- testing
- ci
- release safety