Skip to content

Technology and money in plain English · Since 2012

Tech
Guide

Railway Breach: How to Rotate Your Secrets in the Right Order

Railway says an intruder decrypted some customer environment variables. Here is what the company has confirmed and the order in which to rotate your secrets.

Key facts

  • Railway says an attacker accessed internal systems from Oct. 20 to early Oct. 22 (UTC). As of
  • Railway says at most 1.85% of workspaces may have had variables compromised. As of
  • Railway has not ruled out access to user source code. As of
  • Running apps stayed online; only new builds and deployments were paused. As of
In this story4 sections
Rows of server racks in a data center, the kind of infrastructure at stake in the Railway breach
Image: Brett Sayles / Pexels
Listen to this article
5 min listenTech

Last checked . Picks are based on the prices, rates, fees and terms each provider publishes, checked at the source. We never rank products by commission. See how we’re funded.

If you ran anything on Railway, assume the secrets stored there are exposed and replace them, starting with the provider keys that cost money or unlock other systems. That’s the blunt advice from the Railway breach disclosure, and it applies even if the company never emails you.

What Railway has confirmed about the breach

Railway published a preliminary report on Oct. 22. In it, the company says an attacker got into parts of its internal systems, including build infrastructure, between Oct. 20 and early Oct. 22 (UTC). The attacker got in with credentials for its CI environment. Then the intruder decrypted stored artifacts that held more API access, ran an unauthorized query against encrypted environment variables, and decrypted a subset of them. Railway believes the person was after AI provider API keys.

The scope of the Railway breach is where things get murky. Railway says at most 1.85% of workspaces may have had variables compromised. It couldn’t tell which variables were decrypted, though, so it told upstream providers about “all of the keys we stored.” If a vendor revoked one of your keys, that alone doesn’t mean it was used. Railway also says it isn’t ruling out access to user source code. Running apps stayed up, and only new builds and deployments were paused for a while.

Advertisement

How to rotate secrets after the Railway breach

Railway tells customers to rotate the keys they kept on the platform, and it’s most urgent for accounts created before Oct. 21. GitGuardian, a secrets-security firm, published a walkthrough that turns that advice into a workable order. Read together, the two point to the same sequence.

After the Railway breach, start by listing what you stored: every variable, in every service and environment. The Railway command-line tool can print them as key-value pairs, and a temporary local file is fine for that as long as you delete it afterward.

Then go to the source. Revoke and reissue each credential with the company that issued it, such as AWS, Google Cloud, GitHub, Stripe, or your AI model vendor, and only then paste the new value into Railway. Reverse the order, and the old key stays alive in the one place an attacker may already have a copy.

Don’t stop at the obvious API keys. Database passwords, private registry logins like Docker Hub, session secrets, and webhook signing keys belong on the list too, and Railway asks customers to close public network access to their databases. Sealed variables, the ones you can’t read back through the dashboard or API, aren’t clearly addressed in the disclosure, so replacing them is the safer call.

Next, go looking for trouble. GitHub audit logs, registry audit logs and the usage pages on your cloud and AI accounts can show activity you didn’t cause, going back to Oct. 20. A sudden jump in AI usage charges would be the likeliest early sign. Last, redeploy the latest commit once the keys are swapped, because anything pushed during the pause never shipped, reconnect GitHub if asked, and recreate your command-line and CI credentials.

What is still unknown

Railway hasn’t explained how the attacker got in or who it was. TechCrunch reported that the company gave no count of affected users and plans to publish a post-mortem. Until it lands, the broadest reading of the Railway breach is the safest one.

Editor’s take

Environment variables are convenient, and that’s the problem. One dashboard holding every key your app needs is a tempting target, and this attacker went straight for the keys that unlock paid AI services. The takeaway from the Railway breach isn’t really about Railway. Give each environment its own keys, set spend caps at the provider, and use short-lived tokens when a service offers them. Lock the accounts that issue those keys behind a hardware security key. And watch how fast you can rotate, not the headline, because the next disclosure, like the Quest Diagnostics breach, won’t give you advance notice either.

See something that needs fixing? Report an error

Get The Jolt Journal in your inbox

The day’s top stories in tech, finance and AI, picked by our editors. Free.

Choose your newsletters

Up next

Keep reading

Add The Jolt Journal as a preferred source on Google

Add The Jolt Journal to your Home Screen to get alerts

  • 52 Stories published

The Jolt Journal Tech Desk covers artificial intelligence, software, devices, cybersecurity and the platforms that shape how we live and work. Every story is checked against its sources and reviewed by our editor before publication.

Covers

Most Popular

More in Tech

See all

Leave a comment

This site uses Akismet to reduce spam. Learn how your comment data is processed.

Breaking-news alerts

We’ll send a notification to this device when a big story breaks. Choose what you want, then allow notifications when your browser asks. You can change this or turn alerts off any time.

Send me