Skip to content
NanoFlags
Start free

Reference

Shadow evaluation

While a change waits for approval, your SDKs in production evaluate its draft next to the live configuration and report how many evaluations would change. Approvers see the measured impact before they decide. Nothing is applied: your services keep the live configuration.

What it shows

On the change page, the Production impact (shadow) panel shows, for the last 30 minutes, per service and version: how many evaluations were sampled and what share would get a different value with the draft. For example:

In production over the last 30 minutes: this change would change new_checkout for 12.4% of evaluations in checkout-service (3.1.0). 0% elsewhere. 2 rules of the draft never matched.
  • Percentages appear once a flag has 100 sampled evaluations; before that the panel says not enough traffic yet.
  • “Rules that never matched” are draft rules no sampled evaluation matched: often a typo in a condition or an attribute your services do not send.
  • A new draft starts a new measurement; numbers of an earlier draft are marked as such.

How it works

  • When approval is requested (and while the change is approved but not yet published), NanoFlags signs the draft as a shadow bundle and the edge offers it next to the live bundle.
  • The shadow bundle is signed with its own signature domain and carries a shadow marker inside the signed bytes. SDKs never apply it as configuration, even if it were tampered with or served in place of a normal bundle: a bundle with a shadow marker is always refused as configuration, and a shadow bundle whose signature does not verify is ignored.
  • For a sample of the evaluations of the flags the draft changes (10% by default), the SDK evaluates the draft as well, synchronously and in memory, and counts. Your code always gets the live value.
  • Counters are sent with the SDK's regular telemetry report, at most once a minute. Memory is bounded (at most 100 flags, 10 value pairs and 50 rule counters per flag).
  • When the change is approved and published, sent back to draft or closed, or the draft changes, the measurement ends and the shadow bundle is withdrawn.
  • Browser and mobile keys receive the browser view of the draft (client-exposed flags only), like they do for published configuration. While a change is measured, the draft values of its client-exposed flags can therefore be read by anyone holding the public key. Keep shadow evaluation off where unpublished values must stay secret until they are published.

Shadow evaluation is advisory only. Approvals, scenario requirements, co-signing and publishing work exactly as before; nothing reads the measurement to decide anything.

What leaves your services

Only counters. Per flag: the number of sampled evaluations, how many would change value, how often each pair of values (live → draft) occurred, and how often each draft rule matched. They are labelled with the service name and version you configure in the SDK, and the SDK version.

Never sent or stored: evaluation contexts, attribute names or values, user keys or identifiers, IP addresses. Values in the counters are the flag's own configured values (longer than 100 characters: a hash). When your code's own fallback was used (the flag is new in the draft, an evaluation error, a value of the wrong type), the SDK sends the placeholder <fallback> instead of the value, and the server drops any value that is not one of the flag's configured values. Reports sent with browser keys only count for client-exposed flags and are shown separately; the Slack message and the AI review use server reports only. Counters are deleted after 7 days.

This is designed for data-protection laws such as the Saudi PDPL and the GDPR: no personal data is processed for this feature. It is off by default, turned on per environment by an owner or admin, and every change of the setting is recorded in the audit log (environment.shadow_enabled, environment.shadow_disabled).

Turning it on

Open Settings → Environments → Security, pick the environment (usually production) and choose Turn on under Shadow evaluation. Changes already waiting for approval are measured from then on. SDKs pick up a shadow bundle with their next configuration check: within about a minute when polling, within 5 minutes over the live connection.

SDK options

The JavaScript SDK (@nanoflags/sdk, and the React and React Native packages built on it) supports shadow evaluation. It needs telemetry (events) on; with events: false it is off too.

ts
import { createClient } from "@nanoflags/sdk";

const flags = createClient({
  sdkKey: process.env.NANOFLAGS_SDK_KEY!,
  baseUrl: "https://edge.nanoflags.io",
  service: "checkout-service",
  serviceVersion: "3.1.0",          // shown next to the service name
  shadow: { sampleRate: 0.1 },      // default; `shadow: false` turns it off
});
  • serviceVersion: your service's version, shown next to the service name on the panel.
  • shadow.sampleRate: share of evaluations of changed flags that are also evaluated with the draft (default 0.1).
  • shadow: false: never fetch or evaluate shadow bundles in this service.

Services on other SDKs keep working unchanged and simply do not report.

Where approvers see it

  • Change page: the Production impact (shadow) panel, with rows per service and version.
  • Slack: the approval message of the Slack app gets one line with the measured impact; it is refreshed once there is enough traffic.
  • AI change review: the automatic checks note measured changes (medium from 10% of sampled evaluations), and the AI summary takes them into account when they are available at review time.

Wire format (other SDKs)

For SDK authors. A fresh GET /v1/bundle answer (200 or 304) carries X-NanoFlags-Shadow: <sessionId>.<checksum> while a shadow bundle exists. GET /v1/shadow (same SDK key) answers {sessionId, checksum, baseRevision, bundle} with X-NanoFlags-Signature, X-NanoFlags-Key-Id and X-NanoFlags-Environment-Id; 404 when there is none. The signature is Ed25519 over

text
nanoflags-shadow-v1
<environment id>
<session id>
<checksum>
<sha256 hex of the bundle JSON text exactly as served>

with the keys of /v1/keys. Use the bundle only if the signature verifies and its shadow member has the same session id, checksum and environment id. Shadow-evaluate a flag when its configuration differs from the live one, or when it references a segment that differs, directly or through other segments (a segment differs when its definition does, when the draft removes it, or when it references a segment that differs). Report with the next POST /v1/events body, in a shadow member:

json
{
  "service": "checkout-service", "sdk": "js/0.5.0", "revision": 42, "events": [],
  "shadow": {
    "sessionId": "6f1c2a4e-8b1d-4c3e-9f00-1234567890ab",
    "checksum": "0123456789abcdef0123456789abcdef",
    "version": "3.1.0",
    "flags": [{
      "flag": "new_checkout", "samples": 400, "flips": 50,
      "pairs": [{ "live": "false", "shadow": "true", "count": 50 }],
      "rules": { "0": 120, "default": 280 }
    }]
  }
}

Limits per report: 100 flags, 10 pairs and 50 rule counters per flag, values as JSON text with object keys sorted, at most 100 characters, <fallback> for a side that used the app's fallback, flips ≤ samples. Compare values as JSON with sorted keys, so key order never counts as a change. Rule keys are 0-based rule indexes of the draft, or default.

Plans and limits

PlanShadow evaluation
Explore, SoloNot included
Team, ScaleIncluded
EnterpriseIncluded

Counters are kept 7 days; a measurement ends after 7 days at the latest. One session records at most 50 service and version combinations. See Plans and limits.