Railway breach: what to rotate first (env vars, DB, keys)

On October 21, 2026, Railway told every customer to rotate the secrets stored in their environment variables. Its investigation found that an attacker had reached internal Railway systems, pulled encrypted environment variables, and decrypted a subset of them. Railway says the attacker was hunting for AI provider API keys, but environment variables on a typical vibe-coded app also hold the database password, the Stripe secret key, the Supabase service role key, OAuth client secrets, and the session signing secret.

This guide puts those in the order that limits damage fastest, shows how to rotate each one on Railway without an outage where that is safe, and how to check each provider for misuse. If your keys leaked another way (a commit, a screenshot, a frontend bundle), start with the first hour after an API key leaks instead.

Checked on October 23, 2026 against Railway's security disclosure, its preliminary incident report, the incident status page, Railway's own rotation guides and docs, and coverage by BleepingComputer and TechCrunch, all linked below. The investigation is still open, so check Railway's status page for anything newer. VibeLocked is independent and not affiliated with Railway or any provider named here.

The short version

1. Get deploys working first. Railway reset GitHub connections and revoked API tokens, project tokens, and CLI sessions. A rotated variable only reaches your app after a redeploy.
2. Database: turn off public networking if you do not need it, then Regenerate the password and redeploy every service that uses it.
3. Money and admin keys: Stripe secret and restricted keys, Supabase secret or service_role keys, cloud keys (AWS, GCP).
4. AI provider keys: OpenAI, Anthropic, OpenRouter, xAI Grok, Gemini. These were the attacker's target, and many were already revoked by the providers.
5. GitHub and registry tokens, OAuth client secrets for Sign in with Google or GitHub.
6. Signing secrets: session and JWT secrets (AUTH_SECRET, JWT_SECRET) and webhook secrets (STRIPE_WEBHOOK_SECRET).
7. Everything else in your variables: email, SMS, storage, analytics, monitoring keys.
8. Check each provider for misuse back to October 20, and update every other copy of each old value.

What Railway says happened

From Railway's security disclosure (October 21), its preliminary report (October 22), and the incident status page:

What	Railway's statement
Attack window	October 20, 13:58 UTC to October 22, 03:32 UTC (October 20, 6:58am PT to October 21, 8:32pm PT). The end time is when Railway confirmed the attacker no longer had access to internal systems.
How	A sustained, targeted campaign. Railway found privilege escalation in its build infrastructure, then the attacker reached credentials for Railway's CI environment and ran an unauthorized query of encrypted environment variables, a subset of which were decrypted.
Goal	AI provider API keys, which Railway believes were wanted for distillation attacks. Railway says the attacker looked for workspaces with many environment variables or many AI keys.
Scale	At most 1.85% of workspaces, about 100,000, had some environment variables potentially compromised. Because the data was envelope encrypted, Railway cannot say exactly which variables in an affected project were read.
Who was told	Railway emailed workspaces with definite impact and those it rated higher risk, a group larger than the 1.85%. On October 23 it said those notifications were complete, and that workspaces outside those groups are not confirmed safe.
Your account	Not logged into. Railway has not ruled out access to user source code.
Running apps	Stayed online. New builds and deploys were paused, then restored in stages through October 22.

BleepingComputer reported the same core facts: an unauthorized party "may have accessed encrypted environment variables and, for some of them, obtained the decrypted contents", and Railway urged all users to rotate environment variables and private registry credentials (BleepingComputer, TechCrunch).

What Railway already did for you

• Asked providers to revoke keys. On October 21 Railway sent potentially decrypted keys to GitHub, GitLab, Anthropic, OpenAI, OpenRouter, SpaceXSI, Stripe and others, and for some providers sent every key stored on the platform. If your AI feature or GitHub deploys broke, a revoked key may be why.
• Reset storage bucket credentials. The ACCESS_KEY_ID and SECRET_ACCESS_KEY of every bucket in every environment were regenerated. Your services keep the old values until you redeploy them (bucket and GitHub guide). The audit log shows these resets with the actor type Railway Staff.
• Reset GitHub tokens. The tokens Railway held for your GitHub connection were invalidated, and GitHub tokens stored as registry credentials were removed from Railway. Railway notes those registry tokens still work on GitHub until you revoke them yourself.
• Invalidated Railway API tokens, project tokens, and sessions. Tokens created before the incident stay invalid (recover failing deploys).

What Railway did not do: it cannot rotate your database password, your Stripe key, your OAuth secrets, or your signing secrets. Those are yours.

Step 0: Get deploys working again

Containers read variables only at startup, so every rotation below ends with a redeploy. Fix the pipeline first, using Railway's recover failing deploys guide:

• GitHub deploys: reconnect GitHub in Account Settings, confirm the Railway GitHub App still has access to your repository, then disconnect and reconnect the repository in the service's Settings, Source. In a team project, the member who originally connected the repository is the one who must reconnect.
• CLI deploys: run railway logout, then railway login.
• CI deploys: create a new project token (RAILWAY_TOKEN) or account or workspace token (RAILWAY_API_TOKEN) with the same scope as before, and replace the old value in your CI provider's secrets.
• Commits pushed during the pause were not queued. Your service runs its last successful deployment until you trigger a new one.

While you are in each service, check whether it has a healthcheck path that touches the database. With a healthcheck, Railway only switches traffic to a new deployment after it returns 200, so a deploy with a bad credential fails safe and the old deployment keeps serving (zero downtime rotation).

Step 1: Database credentials and connection strings

The database holds your users' data, and a Railway database with public networking turned on is reachable from anywhere with the password. Railway's guide for PostgreSQL (with matching guides for MySQL, MongoDB and Redis, linked in Sources) gives this order:

1. Find every consumer. In the database service's Variables tab, services that use a reference such as ${{ Postgres.DATABASE_URL }} pick up the new password on their next deploy. Services with a pasted connection string do not. Note them.
2. Check public access. In the database's Settings, Networking, turn off Public Access if nothing outside the project needs it. A private database is only reachable on the project's private network.
3. List outside consumers. Anything using DATABASE_PUBLIC_URL (your laptop, CI, a BI tool, a cron job on another host) needs the new value by hand.
4. Regenerate. Open the database's Database tab, Config, and click Regenerate next to the password. Railway changes the password inside the database and updates the variables together. It restarts the database, which drops every open session, including any an attacker had.
5. Redeploy every service that uses it. Regenerate redeploys only the database. Replace pasted strings with the reference variable while you are there.
6. Verify. Deploy logs should show no password authentication failed errors.

Railway warns against editing POSTGRES_PASSWORD or PGPASSWORD by hand or running ALTER USER yourself, because each changes only one side and leaves the variable and the database out of sync.

Zero downtime or not? Railway documents a dual role flow: create a second database role, move the apps to it, then drop the old one. Railway's incident guide says to use it only if you believe the credential was not compromised. If you think it may have been, cutting off the old password matters more than a short outage, so Regenerate first.

Database hosted elsewhere? If your Railway app connects to Supabase, Neon, PlanetScale or another host, the connection string in your Railway variables is still exposed. Reset the database password in that provider's dashboard, then update the variable on Railway and redeploy.

Redis often holds sessions. Railway's Redis guide says that if sessions or tokens live in Redis, invalidate them by flushing the session database or rotating your app's signing secret.

Step 2: Payment, database admin, and cloud keys

These keys can move money or bypass every rule in your app.

Stripe secret and restricted keys. Railway's Stripe guide follows Stripe's own advice: rotate first, investigate second. On the API keys page in live mode, choose Rotate key. Select an expiration of Now for a confirmed leak, or a short window long enough to redeploy for a precautionary rotation. Rotate every other secret and restricted key on the account too. Then update the variable on Railway and deploy.

Supabase secret or service_role key. It bypasses Row Level Security, so anyone holding it can read and change every row. Create a new secret key (sb_secret_...), put it in your server variables, redeploy, then delete the old one. For a legacy service_role JWT, replace it with a new secret key and disable the legacy keys. The steps and the effect on sessions are in what to do with an exposed Supabase key. The public anon or publishable key does not need emergency rotation; anon key vs service role key explains why.

AWS and Google Cloud keys. Railway published guides for checking AWS and GCP access. Deactivate and replace access keys and service account keys, and review the cloud audit logs for activity you did not cause.

Step 3: AI provider keys

This is what the attacker came for. Railway published usage check guides for OpenAI, Anthropic, OpenRouter and xAI Grok. The pattern is the same for each:

1. Revoke the old key at the provider, even if the provider already emailed you that it was revoked. Deleting it from Railway only removes your copy.
2. Create a new key, one per app and environment, with a spend limit or credit limit set at the provider.
3. Update the variable on Railway and deploy. Check the logs for successful requests.
4. Review usage by key for the attack window: token volume, request counts, and models your app does not use.

Our AI tool breach guide has the provider by provider steps for OpenAI, Anthropic, Gemini and OpenRouter, including hard spend limits, so this guide does not repeat them.

Step 4: GitHub, registry tokens, and OAuth client secrets

GitHub tokens in your variables. If you stored a GitHub personal access token, deploy key, or SSH key as a variable (for example to pull a private package or open pull requests), revoke it on GitHub and create a new one with the smallest scope that works.

Registry tokens. Services that pull private images from ghcr.io stored a GitHub token as a registry credential. Railway removed those from its side, but the token itself still works until you delete it on GitHub. Create a new one with read:packages, add it in the service's Settings, Source, Registry Credentials, and redeploy each service (Railway's guide). Do the same for Docker Hub or other registries, and check their audit logs.

Supabase personal access tokens. If a CI job or script on Railway runs the Supabase CLI or Management API, a token starting with sbp_ may sit in a variable such as SUPABASE_ACCESS_TOKEN. It is separate from your project keys, and an older full access token reaches every Supabase organization and project you belong to. Delete it on the Access Tokens page in your Supabase account settings, then create a scoped token limited to the projects and permissions that job needs, with an expiry. The scoped personal access tokens section lists the permissions for common CLI jobs, and the first hour after an API key leaks covers the rest of the rotation.

Check GitHub for persistence. Railway's GitHub guide lists what to review: your security log, authorized OAuth apps and GitHub Apps, SSH and GPG keys, personal access tokens, and on each repository the deploy keys, secrets, webhooks, and collaborators. An attacker with push access can add a webhook or secret that outlives a password reset. Organization owners should export the audit log before revoking anything.

OAuth client secrets (GOOGLE_CLIENT_SECRET, GITHUB_CLIENT_SECRET, AUTH_GOOGLE_SECRET). With the client secret, someone can impersonate your app in OAuth flows. Google lets you rotate without downtime: on the client's page, Add Secret, update your app, then Disable and delete the old secret (Google: manage OAuth clients). For other providers, generate a new client secret in their developer settings, update the variable, redeploy, then delete the old one.

Step 5: Session, JWT, and webhook signing secrets

These do not call any provider, so no provider will revoke them for you, and they are easy to forget.

Session and JWT secrets (AUTH_SECRET, NEXTAUTH_SECRET, JWT_SECRET, SESSION_SECRET, COOKIE_SECRET). Anyone with this value can forge a session cookie or token and sign in as any user. Rotating it has a side effect: every session signed with the old secret stops working, so users sign in again.

• If you believe the secret may have been decrypted, accept the forced sign out. It also kicks out any forged sessions.
• If you are rotating only as a precaution, some libraries accept several secrets. Auth.js takes an array with the new secret first and the old one second, so current sessions keep working until they expire (set and rotate AUTH_SECRET). Remove the old one afterwards.
• If you use Supabase Auth, Clerk, or Firebase Auth, the session signing key lives with that provider, not in your Railway variables. Rotate what is in your variables (their secret keys), and follow the provider's docs before touching signing keys.

Webhook signing secrets (STRIPE_WEBHOOK_SECRET, Clerk, Resend, GitHub). With the secret, someone can send fake events that your handler trusts, such as "payment succeeded". Stripe lets you Roll secret on the endpoint and either expire the old secret immediately or keep it valid for up to 24 hours, during which Stripe signs each event with both (Stripe webhooks). Roll it, update the variable on Railway, deploy, then confirm deliveries succeed. While you are there, look for webhook endpoints you did not create; Railway's Stripe guide notes an attacker added endpoint receives a copy of every event. How to check that your handler verifies signatures at all: securing Stripe webhooks.

Step 6: Everything else in your variables

Open each service's Variables tab, in every environment you use (production, staging, PR environments), and go down the list. Anything that authenticates to something is a secret: email (Resend, SendGrid, Postmark), SMS, storage outside Railway, search, monitoring, analytics write keys, encryption keys. Rotate each at its provider.

Two Railway features make the next rotation easier (zero downtime rotation):

• Shared variables and references. Store a key once as a project shared variable and reference it with ${{ shared.API_KEY }}. One edit then covers every service, and one deploy moves them all together.
• Staged changes. Edits are staged until you click Deploy, so you can update several variables across services, review the diff, and apply them in one deploy.

Rotate without an outage when you are rotating as a precaution: create the new key, update the variable, deploy, confirm requests succeed, then revoke the old key. When a key is known to be exposed, revoke first and accept the short outage.

Step 7: Check for misuse

Look back to at least October 20, 2026, 13:58 UTC (6:58am PT), the start of the attack window.

Where	What to look for
Railway audit log	Variable reads, registry credential changes, and deployments in the window that were not made by your team or by Railway Staff. Export entries you need, retention depends on your plan.
Database	Roles you did not create (SELECT rolname FROM pg_roles;), new extensions, unexpected row counts or recently changed rows, sessions you do not recognize in pg_stat_activity. Preserve evidence before restoring from a backup.
Buckets	Objects you did not upload, or deletions you did not make. Railway notes buckets have no automatic backups.
Stripe	The old key's request logs (kept even after rotation), payments and refunds you did not make, payouts and bank accounts, webhook endpoints, team members.
AI providers	Usage grouped by key, spend for the period, members and keys you did not create.
GitHub	Security log, authorized apps, new tokens or keys, repository webhooks and deploy keys.
Cloud	AWS and GCP audit logs for access from unknown identities or places.

If you find abuse, keep screenshots and exports, then contact that provider's support. Railway asks anyone with known compromised use of a token to email support@railway.com to help its investigation.

Step 8: Find every other copy of the old values

The values that sat in Railway also sit in other places. Each copy must get the new value, or keep working until you replace it:

• Local .env files and backups, and anything that reads DATABASE_PUBLIC_URL
• CI secrets (GitHub Actions and others) and other hosts that call the same APIs
• Other Railway environments and services with pasted values instead of references
• Shell history, agent and MCP configs, and old commits

The find every copy section of our AI key breach guide has search commands that print file names only, never key values. If a secret was ever also in your frontend code or a public file, rotation is only half the fix; see keep secrets out of frontend code and exposed .env, .git, and source maps.

Common AI-generated mistakes

• A database URL pasted into each service instead of a reference variable, so a regenerated password silently breaks one service, or leaves it on the old one.
• Public networking left on for a database only the app uses.
• Updating the variable without redeploying, so the running container keeps the old value.
• Revoking the old key in Railway only, by deleting the variable, while it still works at the provider.
• One JWT_SECRET or AUTH_SECRET shared by staging and production.
• No healthcheck, so a deploy with a bad credential takes traffic.
• Forgetting the webhook secret and OAuth client secret because nothing broke.

Checklist

• Deploys work again: GitHub reconnected, new Railway tokens in CI, CLI logged in again
• Every database password regenerated, public networking off where not needed, every consumer redeployed
• External database passwords (Supabase, Neon, others) reset at that provider
• Stripe secret and restricted keys rotated, webhook endpoints reviewed
• Supabase secret or service_role key replaced and the old one deleted or disabled
• Any Supabase personal access token (sbp_) deleted and replaced with a scoped token that has an expiry
• AWS and GCP keys replaced and audit logs reviewed
• Every AI provider key revoked at the provider, replaced with one key per app with a spend limit
• GitHub tokens and registry tokens revoked on GitHub and replaced; GitHub security log and repository settings reviewed
• OAuth client secrets rotated
• Session or JWT secret rotated, and I chose deliberately between a forced sign out and a secret array
• Webhook signing secrets rolled and deliveries confirmed
• Every other key in every environment rotated
• Bucket services redeployed with the new credentials
• Provider logs checked back to October 20, 2026
• Local .env files, CI secrets, and other hosts updated with the new values
• Healthchecks and reference variables set up for the next rotation

Copy-paste prompt for your AI coding tool

Paste this into Cursor, Claude Code, Lovable, or Bolt. It asks for variable names and file paths only. Never paste secret values, never paste your Railway variables, and do not let the tool print them.

I host this app on Railway and need to rotate every secret after the October 2026 Railway security incident. Do not print, log, or ask me for any secret values. 1) List every environment variable name this codebase reads (process.env, import.meta.env, os.environ, config files, Dockerfiles, CI workflows), grouped as: database connection strings, payment keys, database admin keys (for example Supabase service role), cloud keys, AI provider keys, OAuth client secrets, session or JWT signing secrets, webhook signing secrets, and other API keys. Show variable names and the file paths that use them only. 2) For each signing secret, say what breaks for users when it changes, and whether the library supports more than one secret during rotation. 3) Flag any connection string or key that is hardcoded, read in frontend code, or logged. 4) Suggest a healthcheck endpoint that verifies the database connection without exposing details. 5) Give me a rotation order and a checklist I can follow in each provider's dashboard. Do not change any files until I approve the plan.

Rotate the values yourself in each provider's dashboard and in Railway. More prompts are in the security review prompt guide.

Check what your live app exposes with LeakCheck

Rotation fixes the secrets Railway held. It does not tell you whether the same secrets also leak from your app itself. LeakCheck scans any live web app or public GitHub repo, read only. On a live app it looks for secret key patterns in the JavaScript your site ships to visitors, exposed files, and, for apps that use Supabase or Firebase, whether the public key can read data. On a public repo it checks the files for leaked keys.

Its limits, honestly: LeakCheck cannot see your Railway environment variables or Railway's internal systems, and it cannot tell you whether your keys were among those decrypted in the incident. It only sees what anyone on the internet can see. A clean result means nothing secret is visible from outside, not that rotation is unnecessary.

Sources

• Railway: Security Disclosure, Wednesday October 21st
• Railway: Preliminary Report, Oct 21st Security Incident (timeline, scale, vendor revocations, FAQ)
• Railway Status: incident updates (token invalidation, bucket resets, notifications complete)
• Railway: Redeploy buckets and reconnect GitHub after the credential reset
• Railway: Recover failing deploys (GitHub, CLI, CI tokens)
• Railway: Rotate PostgreSQL database credentials
• Railway: Rotate MySQL database credentials
• Railway: Rotate MongoDB database credentials
• Railway: Rotate Redis database credentials
• Railway: Check for unexpected usage in Stripe
• Railway: Check for unexpected OpenAI usage
• Railway: Check for unexpected Anthropic usage
• Railway: Check for unexpected OpenRouter usage
• Railway: Check for unexpected Grok usage
• Railway: Check for anomalous GitHub access
• Railway: Check for anomalous AWS access
• Railway: Check for anomalous GCP access
• Railway docs: Rotate API keys and database credentials without downtime
• Railway docs: Database View (Regenerate)
• BleepingComputer: Railway warns devs to rotate secrets after potential breach
• TechCrunch: Developer platform Railway urges all users to revoke secret keys after cyberattack
• Stripe: Webhooks (roll endpoint signing secrets)
• Stripe: API keys
• Supabase: API keys (rotating a leaked key)
• Auth.js: JWT reference (secret arrays)
• Google: Manage OAuth clients (client secret rotation)