Skip to content
NanoFlags
Start free

Integrations

Data warehouse export

Export NanoFlags data to your own bucket every hour or every day: evaluation counts, the audit log, the flag catalog and the published configuration, as gzip-compressed newline-delimited JSON and optionally Parquet, with a manifest per export. Scale plan.

What is exported

Each export covers one UTC period (an hour or a day) of your whole organization. Storage: Amazon S3, Cloudflare R2, Google Cloud Storage (with an HMAC key) or another S3-compatible service.

Set up a bucket

  1. Create a bucket (or a prefix in an existing one) and an access key that can only write objects there. Reading and deleting are not needed.
  2. In NanoFlags, open Settings → Integrations → Data warehouse export as an owner or admin, choose the storage, bucket, optional prefix, schedule, formats and datasets, and enter the access key.
  3. NanoFlags writes a small test object (nanoflags/v1/_connection-test.json) to check the key. Export last period now runs an export at once.
StorageEndpointKey
Amazon S3From the region: s3.<region>.amazonaws.comIAM user access key with the policy below
Cloudflare R2From your account ID: <account>.r2.cloudflarestorage.comR2 API token with Object Read & Write on the bucket (its access key ID and secret)
Google Cloud Storagestorage.googleapis.comHMAC key of a service account with Storage Object Creator on the bucket
S3-compatibleYour https:// endpoint on a public host nameIts access key
Minimal S3 policy
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": "s3:PutObject",
      "Resource": "arn:aws:s3:::acme-exports/nanoflags/*"
    }
  ]
}

Files and manifest

text
<prefix>nanoflags/v1/
  evaluations/dt=2026-11-18/hour=07/part-00000.ndjson.gz
  evaluations/dt=2026-11-18/hour=07/part-00000.parquet
  audit_log/dt=2026-11-18/hour=07/part-00000.ndjson.gz
  flags/dt=2026-11-18/hour=07/part-00000.ndjson.gz
  config_snapshots/dt=2026-11-18/hour=07/part-00000.ndjson.gz
  _manifests/hourly/2026-11-18T07.json        (daily: _manifests/daily/2026-11-18.json, dt=… only)
  • Partitions are Hive-style (dt=…/hour=…), so engines can prune by date.
  • NDJSON files (.ndjson.gz): one JSON object per line, UTF-8, gzip. Every field of every row.
  • Parquet files (optional): flat columns; nested values (flag configurations, audit details, evaluated values) are JSON text columns. Timestamps are UTC milliseconds.
  • The manifest is written last: an export is complete when its manifest exists. It lists every file with its row count, size and SHA-256.
Manifest
{
  "format": "nanoflags-warehouse-export",
  "version": 1,
  "export_id": 1842,
  "org_id": "5b0c…",
  "granularity": "hourly",
  "period_start": "2026-11-18T07:00:00.000Z",
  "period_end": "2026-11-18T08:00:00.000Z",
  "generated_at": "2026-11-18T09:16:02.511Z",
  "formats": ["ndjson", "parquet"],
  "datasets": {
    "evaluations": { "available": true, "rows": 5230, "files": 4 },
    "audit_log": { "available": true, "rows": 18, "files": 2 },
    "experiments": { "available": false, "rows": 0, "files": 0 }
  },
  "files": [
    {
      "dataset": "evaluations",
      "key": "exports/nanoflags/v1/evaluations/dt=2026-11-18/hour=07/part-00000.ndjson.gz",
      "format": "ndjson",
      "rows": 5000,
      "bytes": 81233,
      "sha256": "9f2c…"
    }
  ]
}

Schedule, idempotency and retries

  • A period is exported once its data is final: about 2 h 15 min after an hour ends, 2 h 30 min after a day ends. Evaluation counts are rolled up every 5 minutes and accept events up to an hour late.
  • Rows are read in a fixed order and split into fixed-size parts, so exporting a period again writes exactly the same object names: a retried or repeated export overwrites, it never duplicates. Load by period (replace the partition) to stay idempotent on your side.
  • Failed exports are retried with backoff (5, 10, 20, 40 minutes and so on, 8 attempts). Refused keys or a missing bucket fail at once and mark the destination as failing; the settings page lists recent exports with their status.
  • Exports start from the moment the destination is set up (no backfill); after a pause, up to 48 hourly or 7 daily missed periods are caught up.

Datasets

DatasetOne row perMain fields
evaluationsHour × environment × flag × service × SDK × reason × rule × valueproject_key, environment_key, hour, flag_key, service, sdk, reason, rule_index, value, sampled_events, rated_events, weighted_events, sample_rate_min, sample_rate_max
audit_logAudit entry created in the periodid, created_at, project_key, environment_key, actor_id, actor_name, action, object_type, object_id, object_label, summary, detail
flagsFlag (catalog at export time)project_key, flag_id, key, name, description, type, purpose, permanent, client_exposed, review_date, archived_at, created_at, owner_name
config_snapshotsEnvironment: the revision live at the end of the periodproject_key, environment_key, revision, checksum, published_at, kind, flags, segments
experimentsExposure or metric (once experiments are enabled)Listed in the manifest as unavailable until then

sampled_events counts the evaluation events SDKs sent (sampled); weighted_events sums 1 / sample rate over the events whose rate is known, an estimate of all evaluations.

Load into BigQuery

BigQuery reads gzip NDJSON and Parquet from Cloud Storage. For S3 or R2, copy the files with the Storage Transfer Service, or export straight to a Cloud Storage bucket.

BigQuery
-- One period at a time; OVERWRITE PARTITIONS keeps a repeated load idempotent.
CREATE TABLE IF NOT EXISTS nanoflags.evaluations (
  org_id STRING, project_key STRING, environment_key STRING, hour TIMESTAMP, flag_key STRING,
  service STRING, sdk STRING, reason STRING, rule_index INT64, value JSON,
  sampled_events INT64, rated_events INT64, weighted_events FLOAT64,
  sample_rate_min FLOAT64, sample_rate_max FLOAT64, last_at TIMESTAMP
) PARTITION BY DATE(hour);

LOAD DATA OVERWRITE nanoflags.evaluations
  PARTITIONS (DATE(hour) = '2026-11-18')
FROM FILES (
  format = 'JSON',
  uris = ['gs://acme-exports/nanoflags/v1/evaluations/dt=2026-11-18/*.ndjson.gz']
);

-- Or with the bq tool, Parquet files and Hive partitions:
-- bq load --source_format=PARQUET --hive_partitioning_mode=AUTO \
--   --hive_partitioning_source_uri_prefix=gs://acme-exports/nanoflags/v1/audit_log/ \
--   nanoflags.audit_log 'gs://acme-exports/nanoflags/v1/audit_log/dt=2026-11-18/*.parquet'

Tip

An external table over nanoflags/v1/evaluations/* with Hive partitioning queries the files in place without loading.

Load into Snowflake

Snowflake
CREATE STAGE nanoflags_stage
  URL = 's3://acme-exports/nanoflags/v1/'
  STORAGE_INTEGRATION = acme_s3;          -- or CREDENTIALS = (...); R2 and GCS work through their S3 endpoints
CREATE FILE FORMAT nanoflags_ndjson TYPE = JSON COMPRESSION = GZIP;

CREATE TABLE IF NOT EXISTS nanoflags_evaluations_raw (v VARIANT, file STRING);

-- COPY remembers the files it loaded (64 days): running it again only loads new or changed files.
COPY INTO nanoflags_evaluations_raw (v, file)
  FROM (SELECT $1, METADATA$FILENAME FROM @nanoflags_stage/evaluations/)
  FILE_FORMAT = (FORMAT_NAME = nanoflags_ndjson)
  PATTERN = '.*[.]ndjson[.]gz';

CREATE OR REPLACE VIEW nanoflags_evaluations AS
SELECT v:project_key::string AS project_key, v:environment_key::string AS environment_key,
       v:hour::timestamp_tz AS hour, v:flag_key::string AS flag_key, v:service::string AS service,
       v:reason::string AS reason, v:value AS value, v:sampled_events::number AS sampled_events,
       v:weighted_events::float AS weighted_events
FROM nanoflags_evaluations_raw
QUALIFY ROW_NUMBER() OVER (PARTITION BY file, v:flag_key, v:hour, v:environment_key, v:service, v:sdk, v:reason, v:rule_index, v:value ORDER BY 1) = 1;

-- Parquet instead: CREATE FILE FORMAT nanoflags_parquet TYPE = PARQUET;
--   COPY INTO nanoflags_audit_log FROM @nanoflags_stage/audit_log/ FILE_FORMAT = (FORMAT_NAME = nanoflags_parquet)
--   MATCH_BY_COLUMN_NAME = CASE_INSENSITIVE PATTERN = '.*[.]parquet';

Load into Redshift

Redshift
CREATE TABLE IF NOT EXISTS nanoflags_audit_log (
  id BIGINT, org_id VARCHAR(36), created_at TIMESTAMPTZ, project_key VARCHAR(64), environment_key VARCHAR(64),
  actor_id VARCHAR(36), actor_name VARCHAR(256), action VARCHAR(128), object_type VARCHAR(64),
  object_id VARCHAR(128), object_label VARCHAR(512), summary VARCHAR(2048), detail SUPER
);

-- Replace one period in a transaction, so loading it again is idempotent.
BEGIN;
DELETE FROM nanoflags_audit_log WHERE created_at >= '2026-11-18' AND created_at < '2026-11-19';
COPY nanoflags_audit_log
  FROM 's3://acme-exports/nanoflags/v1/audit_log/dt=2026-11-18/'
  IAM_ROLE 'arn:aws:iam::123456789012:role/redshift-nanoflags'
  FORMAT AS JSON 'auto' GZIP TIMEFORMAT 'auto';
COMMIT;

-- A prefix COPY reads every object under it. When Parquet is also written, list the .ndjson.gz keys of the
-- NanoFlags manifest in a COPY manifest file ({"entries": [{"url": "s3://…", "mandatory": true}]}) and add MANIFEST,
-- or load the Parquet files instead with FORMAT AS PARQUET and the same manifest approach.

Security

  • The access key is encrypted (AES-256-GCM) by the NanoFlags notify service with a key the database does not hold, never shown again and never sent to a browser.
  • Only owners and admins configure the export; every change, test and failed export is in the audit log.
  • Endpoints must be https:// on a public host name; redirects are not followed.
  • Exports contain your organization's data only. The audit log includes member names; evaluation counts contain evaluated values but no evaluation contexts.
Plan
Data warehouse export is included in the Scale plan. See Plans and limits.