Rotating Nine Secrets After Railway's October Breach
In short
- Rotating every credential Mistboard kept on Railway took nine rotations, 22 deleted variables and about two hours of hands-on work spread over 19 hours, plus a 12-minute database outage I caused myself.
- My largest exposure was the Postgres password: the database had a public TCP proxy, so a leaked connection string would have worked from anywhere on the internet.
- The outage came from changing that password while the running containers still held the old one. A second database role lets old and new overlap, and makes the same rotation zero downtime.
- A fingerprint check caught a stale release token that would have blocked the next deploy, without printing either value.
At 4:55 p.m. Pacific on October 21, an email from Railway arrived with the subject “Urgent: Railway Security Disclosure - Rotate Credentials”. The first instruction was “Immediately revoke and reissue all environment variables, including any API keys, stored in Railway.”
Mistboard is a xiangqi and variants site I run alone. It runs on Railway as three services: web (the Node server and the site), engine-worker (the game engines behind the bots) and Postgres. Every secret those services use was a Railway environment variable.
Railway reported a breach and paused builds for a day
Railway’s disclosure says it “detected anomalous activity involving systems that store customer environment variables,” and that an unauthorized third party “may have accessed encrypted environment variables and, for some of them, obtained the decrypted contents.” The impact “may be limited to a subset of users,” who will be emailed. The email I got was the general notice, so I treated every value as leaked. Two days later a second email narrowed it: my workspace’s variables matched a pattern the attacker used to search for AI and cloud provider API keys, though Railway could not say whether any were decrypted. By then every secret in it had been rotated.
The paused builds shaped everything that followed. Revoking a key while the running container still holds it breaks that feature until a new build is live, and for most of a day no new build could go live.
The database password was my largest exposure
I ranked every secret by what someone holding it could do to the site or to its players.
| Secret | What a holder could do | What I found |
|---|---|---|
| Postgres password, with a public TCP proxy | Read or write any table from anywhere | No added roles, no unexplained admin sessions |
| Email API key | Send mail as mistboard.com, such as fake sign-in codes | Send log clean |
| LLM API key (forum translation) | Run inference on my bill | Every call on the key in 30 days matches a translation the server stored |
| Analytics personal API key | Read the site’s analytics through the full API | Rolled |
| Deploy key for a private engine repository | Read the engine source | Replaced, old keys deleted |
| Payment keys | Little: billing isn’t launched | Rolled |
| Two internal service tokens | Call the site’s own internal endpoints | Regenerated |
The services reach the database over Railway’s private network, but the database also had a TCP proxy on a public port, left on so admin scripts could run from a laptop. With that proxy, the connection string was a working login from any machine. Removing it took a click, broke nothing in production, and was one of the first things I did. Admin scripts now reach the database through a short-lived SSH tunnel into the private network.
A copy of the database would hold no passwords. Sign-in is a one-time code sent by email, and the site stores sign-in codes and session tokens only as hashes, so nothing in a read would let someone sign in as a player.
Nineteen hours from the email to the last check
| Time (PDT) | Who | What happened |
|---|---|---|
| Oct 21, 00:24 | Railway | First status entry; builds paused |
| Oct 21, 04:33 | Railway | Dashboard, API and CLI down; builds paused in all regions |
| Oct 21, 16:55 | Railway | Disclosure email: rotate everything |
| Oct 21, 17:30 | Me | Tracker opened; inventory of every secret the code reads |
| Oct 21, 17:40 | Me | Database review: no added roles, no unexplained admin sessions |
| Oct 21, 18:04 | Me | LLM key disabled; forum translation off until the next deploy |
| Oct 21, 18:08 | Me | Public database proxy removed; email send log read clean |
| Oct 22, about 00:10 | Railway | Builds back for paid plans, then hours of slow builds |
| Oct 22, 10:31 | Me | Analytics, payment and webhook keys rolled at the providers |
| Oct 22, 10:38 | Me | Eight new values pushed; web and engine-worker rebuild, 20+ minutes each |
| Oct 22, 10:52 | Me | 22 dead variables deleted |
| Oct 22, 11:04 | Me | engine-worker build cloned the engine repository with the new deploy key; old keys deleted |
| Oct 22, 11:23 | Me | Database password regenerated; outage starts |
| Oct 22, 11:25 | Me | Postgres back; engine-worker crashes on the restart |
| Oct 22, 11:35 | Me | web live on the new password; outage ends |
| Oct 22, 11:38 | Me | engine-worker live on the new password |
| Oct 22, 11:50 | Me | Sign-in code arrives on the new email key, old key deleted; bot game and forum translation work |
| Oct 22, 11:55 | Me | Stale release token found and fixed; rotation complete |
The hands-on time was about 40 minutes that evening and 90 the next morning. The first evening went to what needed no build. Everything else waited for the build queue.
For most providers I overlapped old and new. The email key stayed live until a sign-in code arrived through its replacement, and the old deploy keys were deleted after a build had cloned the engine repository with the new key. Postgres was the exception.
I did the work with Claude Code. Claude kept the tracker, built the inventory from the code, ran the database and log reviews, wrote the scripts and ran every check. I did every step at a provider: rolling each key, typing each new value into the macOS Keychain, removing the proxy, deleting variables and pressing Regenerate. Claude saw secret names, never values: a script carried each value from the Keychain to Railway with its output scrubbed, so no secret appeared in a transcript, a log or a file.
Changing the database password broke every database read for 12 minutes
Railway’s Regenerate button changes the database password and the variable together. I expected about a minute of failed connections while web and engine-worker redeployed from images they had already built. Railway’s docs say otherwise: a redeploy “rebuilds the image from the deployment’s original source code.” A restart reuses the existing deployment, variables included, so it came back with the old password.
Postgres came back clean in two minutes, and every query from the running containers failed. The site itself stayed up: pages loaded, but anything that reads the database failed. Live games dropped, and studies, forums, chat and the game viewer returned errors until a fresh web build went live. Railway’s build queue was still recovering from the incident, so the outage lasted exactly as long as a slow build, and with the old password gone there was no faster way out.
A second login role makes the same rotation zero downtime. Create the new role, point the services at it, deploy, and disable the old role once the new deployments are healthy. Old and new work side by side for as long as the build takes (#614).
engine-worker made it worse. When Postgres restarted, the worker’s database client threw an error nothing was listening for, and the process exited instead of reconnecting (#612). A database restart is routine (maintenance, failover, a plan change), and no test had ever put the worker through one.
A fingerprint check caught a stale token before the next release
Releases drain live games before a restart, and the drain call carries a token. The release script looked in the Keychain under an old item name first and the canonical name second. The rotation wrote the new value under the canonical name and left the old item in place, so the next release would have sent the old token, got a 401, and been unable to drain.
After two releases failed on drain 401s in September, I wrote drain:token-check. It fingerprints every local copy of the token in the order the release reads them and compares each with the value inside the production container, without printing any of them. Four hours before the next scheduled release, it reported the token as shadowed: production’s value was on the laptop, but an older copy was read first. Deleting the old item fixed it.
I knew about the two names when I wrote the check. I built a detector, kept the hazard, and the rotation recreated it. The fix is a single name, with the push script running the check after every rotation (#613).
Twenty-two variables were dead
The first inventory came from the code: every environment variable name the server and scripts read. The dashboard told a different story. Twenty-two variables were dead (flags for retired game variants, install switches for engines I no longer build), and three secret names the code reads were set nowhere. About 30 pairs of feature flags remain, a pair per game variant on both services. To find nine secrets, I read every variable by name.
CI had nothing to rotate. The scheduled workflow that calls production authenticates with a short-lived GitHub OIDC token, which the server verifies, so no credential sits in the repository’s secrets.
Seven rules I’m keeping
- Overlap every credential. The new credential works before the old one dies, on every provider, the database included.
- Time a redeploy before a cut-over. If the old credential dies first, the build time is the outage. Read what the platform means by “redeploy” and “restart”, then time both.
- Every long-running process survives its database restarting. Test it by killing the client’s socket and asserting that the next query succeeds.
- Store each secret under a single name. A second copy under an old name will eventually be read first. Where a value can’t be printed, compare fingerprints, and run the comparison in the tool that does the rotation.
- Keep the secret list short and apart from config. A rotation should read a list of nine. Feature flags belong in code or in a single list (#615).
- Close inbound paths nobody needs. The public database proxy saved a step for admin scripts and left the database reachable from the internet, behind its password, for months.
- Prefer credentials that aren’t stored. OIDC meant zero rotations in CI.
The next step is a timed rehearsal: restore a Postgres dump on another host and boot web against it, so recovering anywhere else is a known number of hours.