# Feature flags: design Status: items 1 to 5 built (SWFT-2393, epic SWFT-2391); item 6 (pilot) not started. This design replaces the Flagsmith plan the epic first described. ## Why Large PRs are slow to review and hard to verify. Product can only test a change after it merges into `develop`, and once there it is hard to revert. Feature flags let unfinished work merge in small PRs while it stays off. Product turns the flag on for a QA tenant, and turning it off replaces a revert. ## Decisions | Decision | Choice | Why | | --- | --- | --- | | Build or buy | Build. Keep the API close to OpenFeature (`flag_enabled(name, context)`) so a provider such as self-hosted Unleash could replace it later. | `django-waffle` is not tenant-aware and its tables end up in the wrong schema under django-tenants. A hosted service sees tenant and user IDs, which raises a BAA and vendor-review question. Self-hosted Unleash or Flagsmith (the epic's first plan) adds a service to run and bring into SOC 2 scope. Every option needs the same snapshot-and-poll cache below anyway. | | Scope | A global default, then a per-tenant override, then a per-user override. The most specific setting wins. | Per-tenant lets product test in QA. Per-user lets one person try a feature in a shared tenant. | | Declaration | Flags are declared in code (`swifty_backend_shared/feature_flags/flags.py`) with owner, ticket, remove-by date and default. The database holds only on/off values. | A typo fails instead of reading as off, flag metadata goes through code review, and the expired-flag CI check needs no database. | | Storage | `FeatureFlag` (one row per declared flag, holding the global value; empty means the code default) and `FeatureFlagOverride` (one tenant or one user) in `swifty_shared` (`swifty_backend_shared`, a shared app), plus an append-only `FeatureFlagChange` history table protected by a database trigger. | `BaseAuditMixin` records only timestamps, not who made a change, so SOC 2 needs its own history table. | | Permissions | Two internal permissions, `view_feature_flags` and `change_feature_flags`, and no default model permissions. Only superusers see or grant them in the user and group admin, and only superusers can add a user to a group that holds one. | Clients use the admin console and can manage their own roles; turning flags on and off is an internal operation. | | Flag lifetime | Each flag stores an owner, its Jira ticket and a removal date. A CI check fails once a flag is past its removal date. | Without this, old flags pile up. | | Who switches flags in prod | Only people with an ops or admin role, and only through Django admin or the management command. Every change is logged. Dev and QA are open to the team. | Switching a flag on in prod is effectively a release, so it needs an approval record that fits the change management policy (SWFT-1944). | | Existing `Tenant.enable_*` fields | Leave them as they are. | Those are permanent product settings per tenant, not temporary rollout flags. | | When the lookup fails | Keep using the last snapshot that loaded successfully. A flag counts as off only when a process has never loaded a snapshot (cold start with the database unavailable). | A Redis or database outage must not quietly switch released features back off across the fleet. | ## Performance The runtime is uvicorn with 4 workers per pod, several pods, and django-q2 workers. Redis is reached through `django_redis` with a 5-second socket timeout (`swifty_django/settings.py`, `_make_cache`). A flag check must therefore never call Redis or the database directly: a Redis hiccup would add up to 5 seconds to every request that checks a flag. - Each process holds every flag and override in memory. That is tens of flags and a few hundred overrides, a few KB, loaded with one query. - `flag_enabled()` is a dictionary lookup with no I/O. - Each process runs at most one Redis `GET flags:version` every 5 seconds, and reloads from the database only when the version changes. - Total cost: a few Redis GETs per second across the fleet, and one small query per process per flag change. ## How a change propagates ``` admin / command ──► DB row saved (source of truth) + FeatureFlagChange row └─ transaction.on_commit ──► Redis INCR flags:version each process, every ≤5s ──► GET flags:version ── changed? ──► reload snapshot from DB safety net: reload from DB every 60s regardless Flutter ──► fetch flags at login, on app resume, and when the API's X-Flags-Version response header differs from what it has ``` How stale a flag can be: about 5 seconds normally, and at most 60 seconds while Redis is down. ## Race conditions 1. **A flag changes during a request.** Middleware takes a snapshot of the flags into a context variable when the request starts, and the whole request uses it. A request never runs half on the old code path and half on the new one. 2. **A flag changes during a django-q2 chain or any other long-running process.** The process decides once, when it starts. It stores the flag values in the task arguments (or on the domain object, such as the prescription), and every later step reads them from there instead of calling `flag_enabled()` again. This is a firm rule. 3. **Pods disagree for a few seconds.** That window can't be avoided. The old code path must handle data written by the new one, because turning a flag off is the rollback. A flag rolls back code, not data. Migrations follow expand/contract (SWFT-1948). 4. **The UI and backend disagree.** The backend always decides. A flagged endpoint returns a clear error when the flag is off, and the UI reloads its flags when it gets that error. A UI flag only hides or shows things; it doesn't enforce anything. 5. **The version is bumped before the database commit.** Bump the version only in `transaction.on_commit`, and when reloading, read the version before querying the database. A change that lands between the two is caught on the next poll. When two admins edit the same flag, the last write wins and the history table records both changes. 6. **A process starts cold.** A new pod, or a django-q2 worker recycled after 100 tasks, loads the snapshot the first time it checks a flag. ## Backend (item 2) Code: `swifty_django/swifty_backend_shared/feature_flags/`. | Module | What it does | | --- | --- | | `flags.py`, `registry.py` | Declare flags; `UnknownFlag` for a name that isn't declared | | `resolve.py` | Pick the value: user, then tenant, then global, then code default | | `store.py` | The per-process snapshot: 5-second version poll, 60-second reload, keep the last good snapshot on failure | | `service.py` | `flag_enabled()`, `capture_flags()`, `set_flag()`, the version counter (`feature_flags` cache alias, 250 ms Redis timeout), and the `post_migrate` sync that adds a row per declared flag | | `middleware.py` | One snapshot per request: Django middleware for the admin, and a WSGI wrapper for the API's connexion app (`swifty_api/api.py`), which doesn't run Django middleware | | `admin.py` | Flag pages with overrides inline and a required reason; read-only change history | | `manage.py feature_flag list` / `set` | For dev, QA and scripts; `--actor` and `--reason` required | `flag_enabled(flag, tenant=..., user=...)` uses `connection.tenant` when no tenant is passed. Outside a request (django-q2), checks use the process snapshot, so call `capture_flags()` once at the start of a chain. Tests: `feature_flags/tests/` (no database, in `scripts/python_unit_tests.sh`) and `swifty_api/tests/test_feature_flags.py` (Postgres). The API test `conftest.py` loads the snapshot before each database test, as a running process already has it, so query-count assertions don't see the load. ## API (item 3) - `GET /swifty/1/feature_flags` (`swifty_api/controllers/feature_flags_controller.py`) returns `{"version": n, "flags": {"name": true, ...}}`: every declared flag, resolved for the signed-in user in the request's tenant. Any signed-in user may call it; flag names are visible to clients. - Every API response carries `X-Flags-Version` (the request's snapshot version, exposed through CORS). It is 0 when the version is unknown. - The version only goes up. The Redis counter starts at the current time in milliseconds whenever the key is missing (first use, a flush, or Redis restarting empty), so it is always past any earlier count. Clients can therefore ignore a lower version: it comes from a process that hasn't seen the latest change yet. - The values only show or hide UI. A flagged endpoint checks the flag itself. ## Flutter (item 4) Code: `swifty_ui/lib/providers/feature_flags.dart`. - `FeatureFlagsNotifier.instance` holds each pharmacy's flags, keyed by domain, and is registered as a provider in `main()`. - `FeatureFlagVersionInterceptor` is on every Swifty API client (`createRetryingDio` with `endsExpiredSession`). When a response's `X-Flags-Version` differs from the version loaded for that host, it loads the flags with the same credentials as that request. Sign-in, switching pharmacy and a flag change therefore need no explicit load. A version lower than (or equal to) the one loaded or loading is ignored, as is 0. A failed load keeps the old flags and waits 60 seconds before trying that version again. - `context.flagEnabled(Flags.x)` and `FeatureFlagGate` read the flags for the selected pharmacy. A flag is off until loaded. Sign-out clears them. - Page tests: `pumpPage(..., flags: {...})`. ## Process and compliance (item 5) - `AGENTS.md` has the rules: when to use a flag, declaring it in both places, PR size, "the backend decides", "decide once" for chains, expand/contract, tests, and removal. - CI (`test_flags.py`, part of `scripts/python_unit_tests.sh`) fails when a flag is past `remove_by`, or when Flutter's `Flags` names a flag that `flags.py` doesn't declare. CI runs it for Django changes and for changes to `swifty_ui/lib/providers/feature_flags.dart`. - **Change management (for SWFT-1944; needs the policy owner's decision).** Proposed wording: "A feature flag and both code paths it selects are reviewed and approved in the PR that adds it. Turning a flag on or off in production is a standard change: it is made by someone holding the Change feature flags permission, through the admin, with a recorded reason. The append-only flag history records who, when, what and why. Turning a flag off to stop a problem follows the same path, with the incident as the reason." Open question for the policy owner: does a production flip also need a second person's approval? Today the system records the change but doesn't require a second approver. ## Plan 1. Design decisions (this document) 2. Django backend: models, `flag_enabled()`, snapshot cache, request middleware, admin and management command, tests 3. API: an endpoint for the active flags, the OpenAPI spec, regenerated clients, and the `X-Flags-Version` header 4. Flutter: load flags, a `FeatureFlags` provider and helper, `pumpPage` support 5. Process and compliance: prod change approval (SWFT-1944), PR size rule, a removal ticket per flag, `AGENTS.md` section 6. Pilot: one real feature behind a flag end to end