Security · Incident response ·

Rotate API Keys After a Hosting Breach: Northflank Guide

Northflank’s status page tells customers to assume every environment variable stored on the platform was decrypted, and to rotate all keys. If you also run apps on other hosts, or keep secrets in GitHub Actions, the same question applies: which keys do you actually have, and in what order should you rotate them? This guide is a names-only inventory and rotation order you can follow for any host breach. As of October 2026, the incident page still showed an open incident when we checked, so look at it again before you finish.

If you want the next step, read Gemini API key reported as leaked? Rotate, cap, find the leak.

Quick fix

  • Rotate billable keys first. AI and cloud keys (OpenAI, Anthropic, Gemini, AWS) can run up a bill in hours, so they go before everything else.
  • Inventory names, never values. List every variable name on the host, in secret groups, in CI and in local .env files. Don’t paste values into chats or tickets.
  • Create the new key, update the host, redeploy, then revoke the old key. Don’t leave a leaked key active “for a few days” to be safe.
  • Check usage logs back to before the incident, and keep checking. Northflank asks you to review AI provider usage and billing logs for unexpected activity.

What the primary sources say

As of October 2026 (sources checked 31 Oct 2026), Northflank’s security incident page lists these updates. Times are as shown on the page, in UTC.

  • 30 Oct, 13:40: “Rotate environment variables immediately.”
  • 30 Oct, 13:58: “Assume all environment variables have been decrypted. Disable/rotate all keys.”
  • 30 Oct, 15:12: “Secret rotation on Postgres and MongoDB has been enabled for all users.” Northflank also recommends disabling public expose on addons, and lists secrets on workloads, secret groups and global secrets.
  • 30 Oct, 19:28: the investigation is ongoing with “no further confirmed findings”, and the guidance is to assume all environment variables have been compromised and rotate all secrets and credentials, “including those in workloads, secret groups and global secrets”, and to “Disable public access to addons where enabled.”
  • 30 Oct, 23:38: “Users have reported unauthorised use of AI API keys supplied to the Northflank platform.” Northflank asks you to revoke existing credentials with their issuing providers, generate replacements, update the affected secrets and workloads in Northflank, review AI provider usage and billing logs, and report suspicious usage to Northflank support and the relevant provider.
  • 31 Oct, 01:04: “We have enabled audit logs for all customers, including free users.”

The same 23:38 update says these reports “may be consistent with the broader pattern of attacks targeting AI credentials” and that similar IP addresses appear in Wiz’s “Black October” report. In the pages we read, neither Northflank nor Wiz gives a root cause or a count of affected customers for the Northflank incident, so we don’t state one.

Wiz’s research post, Black October for AI Developer Platforms (30 Oct 2026), says at least nine AI developer platforms disclosed incidents in October, with Northflank listed on 30 Oct. It says “all public reports highlight targeted exploitation, explicitly hunting stored environment variables, with a focus on those granting AI access”. Wiz describes two enterprise cases it could attribute to one of these vendor breaches. In both, the first unauthorized activity happened within 30 minutes of the reported credential access at the vendor. In one case a stolen key made approximately 800k Bedrock calls over thirty hours, and in the other a restricted key made approximately 1.7 million calls over thirteen hours. Wiz says that at list price, these two cases alone “likely ran up over $100k in usage”.

Step 1: build a names-only inventory

You can’t rotate what you haven’t listed. Start from Northflank’s own list and widen from there. Write each variable name, where it lives, and which provider issued it into a sheet. Leave the value column out.

On the host. Northflank’s secrets docs say you can set build arguments and runtime variables on individual services and jobs, or create a secret group that multiple resources in a project inherit. Variables set directly on a service override same-named variables from a secret group. Go through each of these, because Northflank’s guidance names workloads, secret groups and global secrets separately:

  • Runtime variables on every service and job
  • Build arguments on every build
  • Secret groups, and which resources inherit them
  • Global secrets
  • Addon credentials (Postgres, MongoDB) and any addon with public access enabled
  • Secret files mounted into containers

In GitHub Actions. The gh secret list command lists secrets at repository, environment, organization and user level, and its JSON fields are name, visibility, selected repos and updated time, not values:

gh secret list --repo OWNER/REPO
gh secret list --repo OWNER/REPO --env production
gh secret list --org YOUR_ORG

On your machine and in other hosts. This prints variable names from .env files without their values (it skips lines that start with export):

grep -hE '^[A-Za-z_][A-Za-z0-9_]*=' .env* | cut -d= -f1 | sort -u

Repeat the export-names step for every other host you use. Anything that was ever pasted into a hosting dashboard counts, even if the app that used it is gone.

Step 2: put each key in a priority tier

Rotate in this order. The order is our judgement, based on how fast each kind of key can cost you money or data, and on Northflank and Wiz both pointing at AI credentials.

  1. Billable AI and cloud keys: OpenAI, Anthropic, Gemini, AWS access keys (including Bedrock).
  2. Payment keys: Stripe secret and restricted keys, and webhook signing secrets.
  3. Databases and addons: connection strings and passwords, including the Postgres and MongoDB rotation Northflank enabled.
  4. Source-control and CI tokens: GitHub personal access tokens, deploy keys, registry tokens.
  5. Everything else: email and SMS provider keys, analytics keys, session and signing secrets.

If you can’t tell what a variable does, treat it as live and look up the owning service by name before you delete it.

Step 3: rotate without leaving the old key alive

For each key the order is the same: create the replacement, update the host, redeploy, confirm the app works, then revoke the old key. With a breach, do the last step straight away instead of waiting for a quiet period.

OpenAI. OpenAI’s production best practices say to create a replacement, update your applications to use it, and revoke the old key once you’ve verified the replacement works. It also recommends setting an expiration date when you create a project API key.

Gemini. Google’s Gemini API key docs give a leak checklist: generate a new key, update your application, disable or delete the compromised key once the new one is verified, and audit usage in the Google Cloud console billing logs and API usage.

AWS access keys. AWS’s update access keys guide creates a second key, updates your applications, checks the old key’s last-used information, and deactivates the old key before deleting it. Deactivating is reversible, which makes it a safe first step. AWS describes waiting several days to watch the old key’s last-used information, and that is where we’d shorten it for a breach: deactivate as soon as the new key is deployed and the app works, and delete once you’ve confirmed nothing else needs it.

aws iam create-access-key --user-name YOUR_USER
aws iam get-access-key-last-used --access-key-id OLD_KEY_ID
aws iam update-access-key --user-name YOUR_USER --access-key-id OLD_KEY_ID --status Inactive

Stripe. Stripe’s API keys docs list a compromised secret or restricted key as a reason to rotate. Rotating revokes the key and generates a replacement. If you pick an expiration date, both keys work until then, up to 7 days. Choosing “Now” deletes the old key. For a breach we’d choose Now. Stripe also says webhook signing secrets aren’t API keys and are found per endpoint in the Webhooks section of the Dashboard, so list them in your inventory separately and check Stripe’s webhook docs for how to replace them.

Anthropic. Keys are created in the Claude Console. Anthropic’s Admin API docs say the Admin API can list keys and change a key’s status to inactive, but it can’t create keys. If you manage an organization, that’s the quickest way to disable a batch of old keys once their replacements are live.

Databases. Northflank says secret rotation on Postgres and MongoDB is enabled for all users. Rotate those credentials, then update every secret group and service that holds a connection string. If an addon has public access enabled, Northflank recommends disabling it.

Step 4: confirm the old keys are dead

Rotation only counts when the old value stops working.

  • Send one harmless read request with each old key and expect an authentication error. Stripe’s docs say an expired key returns an authentication error. Check the equivalent for each provider before you rely on it.
  • Check “last used” data. AWS shows last-used information per access key. Stripe lets you open request logs per key.
  • Search for leftovers. The same variable name can sit in an old secret group, a preview environment or a staging copy. Search each by name.

Step 5: review usage and bills, going back before the incident

Northflank asks customers to review AI provider usage and billing logs for unexpected activity. Wiz describes what stolen-key abuse looked like in its two cases, which can help you recognise it:

  • A single, short test call from a residential IP address, within a one-minute sweep that tried keys across multiple organizations
  • A trickle of 14 to 120 calls a day from the same small client, which Wiz reads as a job verifying the keys still worked
  • Enumeration of Bedrock and EC2 quotas, followed by bulk traffic
  • In one case, an attempted account data retention change on the key

Wiz also publishes a table of IP addresses and hosting network ranges (ASNs) used to test stolen keys. It says it doesn’t have evidence linking them directly to the Black October attacks, so treat a match as a lead and not as proof. If you find suspicious usage, Northflank asks you to report it to Northflank support and to the provider. Some providers let you ask for a bill review, but none of the sources above promise a refund or waiver.

Step 6: make the next rotation smaller

  • Limit blast radius. Wiz’s two cases show the limits of a restricted key: the restricted key in Case B still saw approximately 1.7 million calls, and more than a million of them succeeded. Per-key spend limits and budget alerts are worth setting before the next incident.
  • Use expiring keys where the provider supports it. OpenAI recommends expiration dates on project keys, and Stripe documents scheduled rotation.
  • Keep an inventory sheet. The first rotation is slow because nobody knows what exists. The second one shouldn’t be.
  • Check your agent rules. If a coding agent can read your env files, assume it could print them. Add a rule or a hook that blocks reads of .env files before the next incident.

Checklist and kits

Disclosure: these are our own paid products. The Host Breach Rotation Kit is designed to include a names-only secret inventory, a priority-order rotation plan and a read-only repo check. It costs $12 (plus VAT where applicable). For the AI-key part of a leak, the API Bill Panic Kit is designed to help with OpenAI, Anthropic and Google keys: a checklist to limit spend and rotate keys, a scanner for leaked keys in git history, files and built bundles, and a neutral template for asking a provider to review a bill. It doesn’t inventory your whole host or cover AWS or Stripe keys. It costs $9 (plus VAT where applicable). Everything in this guide works without either one.

Sources

As of October 2026 (sources checked 31 Oct 2026):

  • Northflank status, Security incident (updates 30 Oct to 31 Oct 2026): https://status.northflank.com/cmvv0rn5a092f14rutwccodpp
  • Northflank status page: https://status.northflank.com
  • Wiz Blog, Black October for AI Developer Platforms (30 Oct 2026): https://www.wiz.io/blog/black-october-for-ai-developer-platforms
  • Northflank docs, Inject secrets: https://northflank.com/docs/v1/application/secure/inject-secrets
  • GitHub CLI manual, gh secret list: https://cli.github.com/manual/gh_secret_list
  • OpenAI API docs, Production best practices (API keys): https://developers.openai.com/api/docs/guides/production-best-practices
  • Google AI for Developers, Using Gemini API keys (leak response checklist): https://ai.google.dev/gemini-api/docs/api-key
  • AWS IAM docs, Update access keys: https://docs.aws.amazon.com/IAM/latest/UserGuide/id-credentials-access-keys-update.html
  • Stripe docs, API keys (rotate an API key): https://docs.stripe.com/keys
  • Anthropic docs, Admin API (API keys): https://platform.claude.com/docs/en/manage-claude/admin-api

Northflank, Wiz, OpenAI, Anthropic, Claude, Google, Gemini, Amazon Web Services (AWS), Amazon Bedrock, Stripe, GitHub and other names belong to their owners. Names are used only to identify products and services. Stack & Ship is not affiliated with or endorsed by any of them.

Related posts