Skip to content
Incident7 minstable

Rotating a leaked secret

A key in a commit, a log, or a client bundle is already compromised. Revoke first, rotate, redeploy, purge, and audit — in that order — because scrubbing git history does not un-leak anything.

When to use this

  • A credential — API key, database password, token — has been exposed: committed to git, written to a log, shared in a channel, or shipped in a client bundle.
  • You're not sure whether it was exploited, which is exactly when to assume it was.

Prerequisites

  • Access to the secret provider's console to revoke and re-issue.
  • Access to the secret store / CI variables where the new value will live.
  • Git history rewriting tooling (git filter-repo or BFG) and the ability to coordinate a force-push.

Procedure

  1. 1

    Revoke first — assume it is already compromised

    The instinct is to rotate; the correct first move is to revoke. A new key does nothing to stop abuse of the old one — only revocation kills it. Do this before anything else, before you even start scrubbing the repo. Every minute the old key is live is a minute it can be used.

  2. 2

    Issue a new secret into the secret store, never into code

    Generate a fresh credential and place it in your secret manager or CI environment variables. It must never re-enter source code — the whole incident began with a secret living somewhere it shouldn't. Reference it from the environment at runtime.

    New value goes to the vault / CI, not the repo
    # Example: store in your platform's secret manager, not .env in git.
    fly secrets set STRIPE_KEY=sk_live_new...        # or:
    gh secret set STRIPE_KEY --body "sk_live_new..." # CI
    # Code reads it from the environment: config('services.stripe.key')
  3. 3

    Redeploy and confirm the cutover

    Redeploy so every running instance picks up the new secret, then prove the switch: a call authenticated with the old key must now fail, and the app must work end-to-end on the new one. Until you've seen the old key return 401, the rotation isn't done.

  4. 4

    Purge from git history — knowing it does not un-leak

    Rewrite history to remove the secret, force-push, and have collaborators re-clone. Be clear-eyed about what this does and doesn't achieve: it tidies the repo, but forks, mirrors, clones, and CI caches may still hold the value. The purge is hygiene; the revocation in step 1 is the protection.

    Rewrite history, then force-push
    # Remove the offending file/blob from all history.
    git filter-repo --path config/secrets.php --invert-paths
    git push --force --all
    # Tell everyone to re-clone; old clones still contain the secret.
  5. 5

    Audit for abuse and add scanning to prevent recurrence

    Check the provider's access logs for any use of the old key between exposure and revocation — that window is your blast radius. Then close the door: add secret scanning as a pre-commit hook and a CI gate so the next leak is blocked before it lands.

    Block the next leak at commit and in CI
    # Pre-commit + CI secret scanning (e.g. gitleaks).
    gitleaks protect --staged   # pre-commit: blocks the commit
    gitleaks detect --source .  # CI: fails the build on any hit

Verify it worked

  • The old secret is dead: a request authenticated with it returns 401/403.
  • The new secret works end-to-end through the redeployed application.
  • Provider access logs show no unexplained use of the old key after revocation.
  • Secret scanning now blocks the pattern both at commit time and in CI.

If it goes wrong — rollback

  • There is no rolling back a rotation — never restore the leaked secret. If the new one breaks something, issue another new one and redeploy.
  • If revocation took down a legitimate integration, that integration simply needs the new key; fix forward, don't reinstate the old.

Want this run for you?

We implement and operate procedures like this in production systems where a misstep is expensive. Fixed scope, fixed price, defined delivery date.

Request a Fixed-Scope Architecture Blueprint