Feature flags: design

Status: items 1 to 5 built (SWFT-2393, epic SWFT-2391); item 6 (pilot) not started. This design replaces the Flagsmith plan the epic first described.

Why

Large PRs are slow to review and hard to verify. Product can only test a change after it merges into develop, and once there it is hard to revert. Feature flags let unfinished work merge in small PRs while it stays off. Product turns the flag on for a QA tenant, and turning it off replaces a revert.

Decisions

Decision

Choice

Why

Build or buy

Build. Keep the API close to OpenFeature (flag_enabled(name, context)) so a provider such as self-hosted Unleash could replace it later.

django-waffle is not tenant-aware and its tables end up in the wrong schema under django-tenants. A hosted service sees tenant and user IDs, which raises a BAA and vendor-review question. Self-hosted Unleash or Flagsmith (the epic’s first plan) adds a service to run and bring into SOC 2 scope. Every option needs the same snapshot-and-poll cache below anyway.

Scope

A global default, then a per-tenant override, then a per-user override. The most specific setting wins.

Per-tenant lets product test in QA. Per-user lets one person try a feature in a shared tenant.

Declaration

Flags are declared in code (swifty_backend_shared/feature_flags/flags.py) with owner, ticket, remove-by date and default. The database holds only on/off values.

A typo fails instead of reading as off, flag metadata goes through code review, and the expired-flag CI check needs no database.

Storage

FeatureFlag (one row per declared flag, holding the global value; empty means the code default) and FeatureFlagOverride (one tenant or one user) in swifty_shared (swifty_backend_shared, a shared app), plus an append-only FeatureFlagChange history table protected by a database trigger.

BaseAuditMixin records only timestamps, not who made a change, so SOC 2 needs its own history table.

Permissions

Two internal permissions, view_feature_flags and change_feature_flags, and no default model permissions. Only superusers see or grant them in the user and group admin, and only superusers can add a user to a group that holds one.

Clients use the admin console and can manage their own roles; turning flags on and off is an internal operation.

Flag lifetime

Each flag stores an owner, its Jira ticket and a removal date. A CI check fails once a flag is past its removal date.

Without this, old flags pile up.

Who switches flags in prod

Only people with an ops or admin role, and only through Django admin or the management command. Every change is logged. Dev and QA are open to the team.

Switching a flag on in prod is effectively a release, so it needs an approval record that fits the change management policy (SWFT-1944).

Existing Tenant.enable_* fields

Leave them as they are.

Those are permanent product settings per tenant, not temporary rollout flags.

When the lookup fails

Keep using the last snapshot that loaded successfully. A flag counts as off only when a process has never loaded a snapshot (cold start with the database unavailable).

A Redis or database outage must not quietly switch released features back off across the fleet.

Performance

The runtime is uvicorn with 4 workers per pod, several pods, and django-q2 workers. Redis is reached through django_redis with a 5-second socket timeout (swifty_django/settings.py, _make_cache). A flag check must therefore never call Redis or the database directly: a Redis hiccup would add up to 5 seconds to every request that checks a flag.

  • Each process holds every flag and override in memory. That is tens of flags and a few hundred overrides, a few KB, loaded with one query.

  • flag_enabled() is a dictionary lookup with no I/O.

  • Each process runs at most one Redis GET flags:version every 5 seconds, and reloads from the database only when the version changes.

  • Total cost: a few Redis GETs per second across the fleet, and one small query per process per flag change.

How a change propagates

admin / command ──► DB row saved (source of truth) + FeatureFlagChange row
                    └─ transaction.on_commit ──► Redis INCR flags:version
each process, every ≤5s ──► GET flags:version ── changed? ──► reload snapshot from DB
                    safety net: reload from DB every 60s regardless
Flutter ──► fetch flags at login, on app resume, and when the API's
            X-Flags-Version response header differs from what it has

How stale a flag can be: about 5 seconds normally, and at most 60 seconds while Redis is down.

Race conditions

  1. A flag changes during a request. Middleware takes a snapshot of the flags into a context variable when the request starts, and the whole request uses it. A request never runs half on the old code path and half on the new one.

  2. A flag changes during a django-q2 chain or any other long-running process. The process decides once, when it starts. It stores the flag values in the task arguments (or on the domain object, such as the prescription), and every later step reads them from there instead of calling flag_enabled() again. This is a firm rule.

  3. Pods disagree for a few seconds. That window can’t be avoided. The old code path must handle data written by the new one, because turning a flag off is the rollback. A flag rolls back code, not data. Migrations follow expand/contract (SWFT-1948).

  4. The UI and backend disagree. The backend always decides. A flagged endpoint returns a clear error when the flag is off, and the UI reloads its flags when it gets that error. A UI flag only hides or shows things; it doesn’t enforce anything.

  5. The version is bumped before the database commit. Bump the version only in transaction.on_commit, and when reloading, read the version before querying the database. A change that lands between the two is caught on the next poll. When two admins edit the same flag, the last write wins and the history table records both changes.

  6. A process starts cold. A new pod, or a django-q2 worker recycled after 100 tasks, loads the snapshot the first time it checks a flag.

Backend (item 2)

Code: swifty_django/swifty_backend_shared/feature_flags/.

Module

What it does

flags.py, registry.py

Declare flags; UnknownFlag for a name that isn’t declared

resolve.py

Pick the value: user, then tenant, then global, then code default

store.py

The per-process snapshot: 5-second version poll, 60-second reload, keep the last good snapshot on failure

service.py

flag_enabled(), capture_flags(), set_flag(), the version counter (feature_flags cache alias, 250 ms Redis timeout), and the post_migrate sync that adds a row per declared flag

middleware.py

One snapshot per request: Django middleware for the admin, and a WSGI wrapper for the API’s connexion app (swifty_api/api.py), which doesn’t run Django middleware

admin.py

Flag pages with overrides inline and a required reason; read-only change history

manage.py feature_flag list / set

For dev, QA and scripts; --actor and --reason required

flag_enabled(flag, tenant=..., user=...) uses connection.tenant when no tenant is passed. Outside a request (django-q2), checks use the process snapshot, so call capture_flags() once at the start of a chain.

Tests: feature_flags/tests/ (no database, in scripts/python_unit_tests.sh) and swifty_api/tests/test_feature_flags.py (Postgres). The API test conftest.py loads the snapshot before each database test, as a running process already has it, so query-count assertions don’t see the load.

API (item 3)

  • GET /swifty/1/feature_flags (swifty_api/controllers/feature_flags_controller.py) returns {"version": n, "flags": {"name": true, ...}}: every declared flag, resolved for the signed-in user in the request’s tenant. Any signed-in user may call it; flag names are visible to clients.

  • Every API response carries X-Flags-Version (the request’s snapshot version, exposed through CORS). It is 0 when the version is unknown.

  • The version only goes up. The Redis counter starts at the current time in milliseconds whenever the key is missing (first use, a flush, or Redis restarting empty), so it is always past any earlier count. Clients can therefore ignore a lower version: it comes from a process that hasn’t seen the latest change yet.

  • The values only show or hide UI. A flagged endpoint checks the flag itself.

Flutter (item 4)

Code: swifty_ui/lib/providers/feature_flags.dart.

  • FeatureFlagsNotifier.instance holds each pharmacy’s flags, keyed by domain, and is registered as a provider in main().

  • FeatureFlagVersionInterceptor is on every Swifty API client (createRetryingDio with endsExpiredSession). When a response’s X-Flags-Version differs from the version loaded for that host, it loads the flags with the same credentials as that request. Sign-in, switching pharmacy and a flag change therefore need no explicit load. A version lower than (or equal to) the one loaded or loading is ignored, as is 0. A failed load keeps the old flags and waits 60 seconds before trying that version again.

  • context.flagEnabled(Flags.x) and FeatureFlagGate read the flags for the selected pharmacy. A flag is off until loaded. Sign-out clears them.

  • Page tests: pumpPage(..., flags: {...}).

Process and compliance (item 5)

  • AGENTS.md has the rules: when to use a flag, declaring it in both places, PR size, “the backend decides”, “decide once” for chains, expand/contract, tests, and removal.

  • CI (test_flags.py, part of scripts/python_unit_tests.sh) fails when a flag is past remove_by, or when Flutter’s Flags names a flag that flags.py doesn’t declare. CI runs it for Django changes and for changes to swifty_ui/lib/providers/feature_flags.dart.

  • Change management (for SWFT-1944; needs the policy owner’s decision). Proposed wording: “A feature flag and both code paths it selects are reviewed and approved in the PR that adds it. Turning a flag on or off in production is a standard change: it is made by someone holding the Change feature flags permission, through the admin, with a recorded reason. The append-only flag history records who, when, what and why. Turning a flag off to stop a problem follows the same path, with the incident as the reason.” Open question for the policy owner: does a production flip also need a second person’s approval? Today the system records the change but doesn’t require a second approver.

Plan

  1. Design decisions (this document)

  2. Django backend: models, flag_enabled(), snapshot cache, request middleware, admin and management command, tests

  3. API: an endpoint for the active flags, the OpenAPI spec, regenerated clients, and the X-Flags-Version header

  4. Flutter: load flags, a FeatureFlags provider and helper, pumpPage support

  5. Process and compliance: prod change approval (SWFT-1944), PR size rule, a removal ticket per flag, AGENTS.md section

  6. Pilot: one real feature behind a flag end to end