Secrets and app config

TenancyEngine is the secret authority for your application. Secrets are versioned, encrypted with a per-version data key that is itself wrapped by a master key held outside the database, and every read or write is audited (who, which version, when -- never the value).

All runtime routes use your organization API key and an explicit ?environment= of Development, Staging or Production.

Reading secrets (secrets.read)

GET /api/v1/runtime/applications/{applicationId}/secrets?environment=Staging
X-TenancyEngine-Api-Key: te_...

Returns the current version of every secret this application environment owns or has been granted:

{ "secrets": [ { "key": "Stripe:SecretKey", "value": "...", "version": 3, "source": "owned",
                 "createdAtUtc": "...", "updatedAtUtc": "..." } ] }

GET .../secrets/{key}?environment= returns one secret (404 when it is not owned or granted). source is owned or granted; an owned secret shadows a granted one with the same key.

The secrets.read scope is explicit: it is not included in mcp.full, admin, config.read or any other scope, because it returns real credential values that can be shared across applications. The routes share the public API rate limit.

Sharing one secret between two applications

A secret shared by two apps is one record plus a grant, so the apps can never drift apart. Grants are managed in the console API by an organization admin or platform operator (permission platform:applications:configure on the owning application); the grantee must be in the same organization.

Console and admin routes (never return values)

Under /api/v1/console/applications/{applicationId}/secrets (requires a signed-in user with application configure permission, platform:applications:configure). None of these routes ever returns a secret value, and every one of them is audited.

Route	Purpose
GET ?environment=	List the secrets this environment owns, with rotation state (below).
PUT /{key}?environment=	Create the secret or add a new version immediately. Body { "value", "kind", "description", "rotationPolicyDays" }; kind is generated, paste or te-issued. The value is write-only. For a controlled cutover with a grace window use PUT /{secretId}/value or POST /{secretId}/rotate.
POST /{secretId}/rotate?environment=	Start a rotation (generated and te-issued kinds).
PUT /{secretId}/value?environment=	Paste-once rotation (paste kind).
PUT /{secretId}/policy?environment=	Set or clear the automatic rotation policy.
GET /{secretId}/rotations?environment=&take=	Rotation history, newest first.
POST /{secretId}/grants?environment=	Grant another environment read access. Body { "granteeApplicationEnvironmentId" }.
DELETE /{secretId}/grants/{granteeEnvironmentId}?environment=	Revoke a grant.
GET /audit?environment=&secretId=&take=	Access audit entries.

The list (GET ?environment=)

[ {
  "id": "guid", "key": "PortalSso:SigningKey", "kind": "generated",          // generated | paste | te-issued
  "description": null, "rotationPolicyDays": 90,
  "applicationEnvironmentId": "guid",
  "currentVersion":  { "number": 3, "activatedAt": "2026-10-01T12:00:00Z" },
  "previousVersion": { "number": 2, "expiresAt": "2026-10-02T12:00:00Z" },  // null once the grace window has ended
  "pendingVersion":  null,                                                    // { number, createdAt, promoteAt } while a rotation waits to promote
  "currentMasterKeyId": "...",
  "lastRotatedAt": "2026-10-01T12:00:00Z",      // when the current version became current
  "nextRotationDueAt": "2026-12-30T12:00:00Z",  // null when there is no policy
  "rotationStatus": "Idle",                     // Idle | InProgress | Failed
  "lastError": null,                            // set when rotationStatus is Failed (a reason code, never a value)
  "grants": [ { "environmentId": "guid", "appName": "Billing Portal", "environment": "Staging" } ],
  "grantedEnvironmentIds": [ "guid" ],
  "canAutoRotate": true,
  "canAutoRotateReason": "TenancyEngine generates the next value and rotates it for you.",
  "createdAt": "...", "createdAtUtc": "...", "updatedAtUtc": "..."
} ]

rotationStatus is Failed while the most recent rotation failed and no later one succeeded.

Rotation routes

• POST /{secretId}/rotate, body { "graceMinutes": number? } (optional body; default 1440 = 24 h, maximum 43200). For generated and te-issued secrets. Mints the next value on the server, keeping the shape of the old one (base64 of N bytes stays base64 of N bytes, hex stays hex). Returns 202 { "rotationId", "id", "secretId", "status", "fromVersion", "toVersion", "promoteAt", "error" }. status is Succeeded when the rotation was due immediately and has already been promoted (the default), InProgress while it waits for its distribution window. 409 when a rotation of that secret is already in progress, 422 when the kind cannot rotate itself (paste) or the secret has no current value, 404 for an unknown secret, 400 for a bad graceMinutes.
• PUT /{secretId}/value, body { "value": string, "graceMinutes": number? }. Paste-once rotation for paste secrets: the value becomes a Pending version and goes through the same promote flow. Same 202 shape and 409/404/400; 422 when the secret is not a paste kind.
• PUT /{secretId}/policy, body { "rotationPolicyDays": number | null } (1 to 3650, or null to switch automatic rotation off). Returns the updated list item. 400 when out of range.
• GET /{secretId}/rotations?take= returns [ { "id", "startedAt", "finishedAt", "status", "fromVersion", "toVersion", "actor", "trigger", "graceMinutes", "error" } ]. status is InProgress, Succeeded or Failed; trigger is Manual, Scheduled or Paste; actor is user:{id}, api-key:{id} or system (the scheduler).

How a rotation runs

1. A Pending version is created (generated server side, or the pasted value). It is never served as the current value.
2. After the distribution window (Secrets:Rotation:DistributeSeconds, default 0 = immediately) the Pending version becomes Current and the old Current becomes Previous with expiresAt = now + grace. During the window apps can already read the Pending value as {Key}Next.
3. After the grace window the Previous version is revoked: its wrapped data key and ciphertext are destroyed.
4. A failure at any step records Failed with a reason (pending_version_missing, current_version_changed, promotion_failed:<ExceptionType>) and leaves the Current version untouched. A value that an operator writes with PUT /{key} while a rotation is waiting fails that rotation rather than being overwritten.

A second rotation started while the previous one's grace window is still open supersedes it: the older Previous version is revoked immediately.

A rotation policy (rotationPolicyDays) starts a rotation of generated and te-issued secrets automatically once nextRotationDueAt has passed, with actor system. A scheduled rotation that failed is not started again for an hour. Paste secrets show their due date but are never rotated without a person pasting the new value.

Which process runs it. The rotation state machine is the SecretRotations table, advanced by SecretRotationJob, an IScheduledJob on the platform's JobScheduler, which takes the Postgres advisory IDistributedJobLock before each tick, so with N replicas exactly one advances it. The same pass promotes due rotations, revokes expired Previous versions and starts scheduled rotations. The routes above advance a due rotation immediately under the same lock, so a manual rotation finishes within the request when no distribution window is configured. The Temporal runtime (ADR-0021) is not used for this: it is gated and AutomationRuntime:Enabled is false by default, and rotation does not need it. Settings under Secrets:Rotation: DefaultGraceMinutes, MaxGraceMinutes, DistributeSeconds, SchedulerEnabled, JobIntervalSeconds (30), MaxAttempts (3), ScheduledRetryBackoffMinutes (60).

Reading the previous and next values

GET /api/v1/runtime/applications/{id}/secrets returns, in addition to each secret's Current value, an entry {Key}Previous (source previous, with expiresAtUtc) while the grace window is open and {Key}Next (source next) while a rotation waits to promote. PortalSso:SigningKey therefore also yields PortalSso:SigningKeyPrevious for the length of the grace window. A verifier accepts Current, Previous and Next and signs with Current only. An owned secret with the same key shadows a derived entry.

Rotating an organization API key with overlap

POST /api/v1/console/api-keys/{id}/rotate (organization API key admin permission, as for create and revoke), body { "graceHours": number?, "organizationId": guid? } (default 24, maximum 720; organizationId only matters for platform-wide users). It mints a successor with identical scopes, application binding, application allowlist, IP allowlist and expiry rule (an expiring key gets the same lifetime from now), links the two keys and sets the old key's expiry to now + grace; an old key that already expires sooner keeps its sooner expiry. Both keys authenticate until then. The response is the reveal-once shape of creation: { "key": {...}, "plainKey": "te_...", "predecessor": {...} }; the new plaintext is never retrievable again. Key views carry successorKeyId and predecessorKeyId. Revoked, expired, already-rotated (successor still active) and environment-bound workload keys are refused with 400/404. The action is audited as apikey.rotate.

App config routes (config.read / config.write)

/api/v1/runtime/applications/{applicationId}/config[/{key}] is unchanged for callers. It now runs on the same store, so config values are versioned, envelope-encrypted and audited too. Config routes only see values written through them; a config.read key cannot read a secret created through the secrets API (it gets a 404, and a config write to such a key returns 409).

Loading secrets into configuration (SDK 1.1.0)

TenancyEngine.Sdk 1.1.0 adds GetSecretsAsync / GetSecretAsync on TenancyEngineClient and a configuration provider so an app does not have to call them itself:

// After the sources that carry the bootstrap values (appsettings, environment, user secrets):
builder.Configuration.AddTenancyEngineSecrets("SecretsProvider:TenancyEngine");

Setting (under the section)	Meaning
Enabled	Explicit switch. With none, the provider is on only when BaseUrl is set.
BaseUrl, ApplicationId, Environment	Which store and which application environment. There is no default environment.
ApiKey	An org API key with secrets.read, used as given.
WorkloadCredential	A credential from the GitHub OIDC exchange (below); renewed in memory before it expires. Set exactly one of ApiKey or WorkloadCredential.
PollIntervalSeconds	Reload interval, default 300.
CredentialRefreshIntervalSeconds	How often a workload credential is renewed until its real expiry is known, default 1200.
FailOnStartupError	Default: fail unless ASPNETCORE_ENVIRONMENT/DOTNET_ENVIRONMENT is Development.
CredentialStatePath	SDK 1.1.1. Where a renewed workload credential is saved so a restarted process continues its renewal chain (see Durable renewal). Empty (default) saves nothing.

Key mapping. __ becomes :, so the secret PortalSso__SigningKey (or PortalSso:SigningKey) is read as configuration["PortalSso:SigningKey"] and binds to PortalSso.SigningKey in an options class. The provider is added last, so it wins over appsettings and environment variables for the keys it owns.

Behaviour. The first load is synchronous with retries; outside Development a failed first load throws TenancyEngineSecretsException and the app does not start. After that it polls, raises the configuration change token when a value or the set of keys changed (so IOptionsMonitor consumers update), and keeps the last successful copy in memory only. A failed poll (network error, 5xx, even 401) leaves the values in place and is logged without any value. No secret value is ever written to disk. Options that are read once (IOptions<T>) need a restart to see a rotated value: consume secrets through IOptionsMonitor<T> (or IOptionsSnapshot<T>) and read CurrentValue at the point of use, as TenaBill's SSO services do.

Deploy-time workload identity (GitHub OIDC)

A deploy job can obtain a credential for exactly one application environment without any stored secret.

1. An operator registers the job in the console route POST /api/v1/console/applications/{id}/workload-identities?environment=Staging with the exact repository, ref and jobWorkflowRef (for example cyntrix-inc/tenancy-infrastructure, refs/heads/main, cyntrix-inc/tenancy-infrastructure/.github/workflows/deploy.yml@refs/heads/main). Wildcards are rejected. GET lists, DELETE /{registrationId} removes the entry and revokes every live credential it issued, GET /issuances is the audit.
2. The workflow (with permissions: id-token: write) requests an ID token with audience tenancyengine-secrets and exchanges it:

- run: |
    OIDC=$(curl -sH "Authorization: bearer $ACTIONS_ID_TOKEN_REQUEST_TOKEN" "$ACTIONS_ID_TOKEN_REQUEST_URL&audience=tenancyengine-secrets" | jq -r .value)
    curl -s -X POST https://auth.stage.saasruntime.com/api/v1/workload-identity/github/exchange       -H "Authorization: Bearer $OIDC" -H 'Content-Type: application/json'       -d '{"applicationId":"<id>","environment":"Staging"}'   # -> { "credential": "te_...", "expiresAtUtc", "chainExpiresAtUtc" }

(.NET jobs can use TenancyEngineWorkloadIdentity.RequestGitHubIdTokenAsync / ExchangeGitHubTokenAsync.) 3. The deploy step injects the credential as SecretsProvider__TenancyEngine__WorkloadCredential.

SaaSRuntime verifies the RS256 signature against GitHub's JWKS (token.actions.githubusercontent.com), the issuer, the audience, expiry (60 s skew, and a token may not claim a lifetime over an hour), and that repository, ref and job_workflow_ref match an active registration for that application environment. A token can be exchanged once (its jti is recorded), so a captured token cannot be replayed. Failures return only 401 invalid_token or 403 not_authorized; the reason is logged, never returned.

The credential is an ordinary org API key record: prefix te_, scope secrets.read only, bound to the application and, new in this release, to one environment (it gets 403 for any other environment and for every route that needs another scope), expiring after an hour. Reusing the key record gives expiry, revocation, rate limiting and the secret-read audit (by key id) for free; a JWT would have needed its own signing key, which is circular for a secret store. These keys are hidden from the console key list.

Renewal. Only a workflow can get an OIDC token, so a running container renews with the credential itself: POST /api/v1/runtime/workload-credentials/refresh returns the next credential in the same chain (the SDK provider does this before expiry and on a 401). The previous credential is left to expire on its own so replicas sharing the injected credential can each renew. A chain ends after an absolute 30 days, measured from the OIDC exchange, and every refresh re-checks that the registration still exists. Security reasoning: a stolen credential is worth at most its remaining lifetime unless the thief keeps refreshing, which is bounded by the 30-day cap, visible in the issuance audit (method, caller IP, chain id), and stops the moment the registration is removed.

Durable renewal (a restarted container still starts)

A container that restarts after its injected one-hour credential expired (crash, out-of-memory kill, host reboot, docker restart) used to be unable to start. Three layers fix that without a stored long-lived secret:

1. Every deploy and every reconfigure exchanges a fresh OIDC token (scripts/exchange_workload_credential.py in tenancy-infrastructure, called by every app's own deploy-staging.yml and release.yml through the shared actions/app-release action, so the OIDC subject is the app's repository, and by reconfigure-root-secrets.yml), so a recreated container always starts from a credential issued by that run. The credential reaches the container through a compose variable named in scripts/workload-identity-apps.json (variable, for example TENABILL_WORKLOAD_CREDENTIAL) that the app's service maps to SecretsProvider__TenancyEngine__WorkloadCredential, together with SecretsProvider__TenancyEngine__CredentialStatePath=/tmp/tenancyengine-secrets/state.cred. The whole mechanism is off until the repository variable TE_WORKLOAD_IDENTITY_ENABLED is true and an app has a catalog entry; the catalog is empty until P5, so nothing changes for running apps.
2. The SDK keeps the latest renewed credential (SDK 1.1.1, CredentialStatePath) in a small owner-only file, tied by fingerprint to the credential the deploy injected (a redeploy therefore ignores older state). A restart resumes from it. The file holds a credential only, never a secret value.
3. A bounded restart window on the server. With WorkloadIdentity:RestartRefreshWindowMinutes greater than 0, a workload credential that expired no more than that long ago may call the refresh route (and only that route) to get the next credential of its chain. Default 0 (off, exactly P2 behaviour); set it per lane (for example 1440) when the first app depends on the provider. It cannot outlive the 30 day chain, the registration is re-checked, ordinary keys never get it, and each use appears in the issuance audit.

When all of that is exhausted the provider fails startup with a precise message: the credential was rejected, and that the deploy or the reconfigure workflow injects a fresh one.