Key rotation 101
What is key rotation?
Key rotation is the practice of changing access keys on a regular schedule to reduce the likelihood that a key can be abused if it gets compromised.
Why rotate API keys?
Over time, secrets sprawl. They end up being exposed in places they shouldn’t be due to human mistakes and operational practices during a product’s lifecycle. Detecting when a secret is exposed is important, but rotating API keys on a regular basis is an essential layer of security to ensure that exposed secrets can’t be used for long and are less likely to work when they are discovered.
What happens if you never rotate an API key?
The longer a key is around, the more chances it has to get leaked or compromised. Not only that, but if an API key is never rotated it is guaranteed to work for any malicious party that finds it. Rotating your keys on a regular basis reduces the window of opportunity in which a key can be abused.
Key rotation best practices
Document from the beginning!
Having well-documented API keys will help you avoid a hard time when you need to rotate a key. The first thing you should do on this front is to record where your keys are being used. If a key is compromised, you’ll want to quickly identify the affected applications and data. Even if it’s not for a critical security issue, it will make the general process of rotating the key a lot smoother.
The next thing to do is record who/what has access to an API key. When a developer leaves the company, even if it was on good terms, they should no longer have access to internal systems. Keeping a detailed record of which keys an individual has access to will make it possible to quickly rotate all the relevant keys.
How often should you rotate API keys?
As a best practice, you should rotate API keys at least every 90 days. If you have a strong automated process for rotating keys, you could rotate much more often than that. We will get into automation later, though.
Important events may require you to rotate keys as well. Example events could be a developer leaving the company or a secret being compromised or exposed in plain text. Having a well-established process for key rotation is especially important for time-sensitive events like these.
Mackenzie Jackson
How do you rotate a key without downtime?
To rotate a key with zero downtime, you’ll need to create and deploy a new key before revoking the old one. If possible, monitor logs to ensure that the new key is being used after it has been deployed. Once the new key is being used by your application, you can revoke the old key.
What if the service only allows one active key at a time?
If the service you’re using only supports one active key at a time, you have a couple of options. The option you go with will depend on whether you can choose a custom secret to replace your old one:
- If you can manually set the new key, you can have your application use BOTH the new key and the old one that is being rotated. Just have your app try the new key first, and then fall back to the old one if it fails. Once you have manually rotated it to the new custom value, your app will automatically start using the new key.
- If you cannot manually set the new key, your only option to avoid downtime is to create a new application or account with its own key on the service side. Keeping track of multiple client credentials for the same application isn’t clean nor desirable, so this should be a last resort.
How do you automate key rotation?
Assuming you know how to rotate a secret manually, it can be automated if the service exposes key management through their API. To enable automated key rotation, the service’s API needs to let you:
- Create new keys
- Revoke old keys
The final implementation of your automation is going to depend on the service you are working with and how they do key management. If you know the process and have access to the proper APIs, you can do it! One last thing – don’t forget to validate active and inactive keys as part of your automation!
Key rotation examples
Now that we know what key rotation should look like, let’s look at a couple real examples.
Rotating a GitHub App Token
Let’s start with an easy one. Rotating a GitHub App key is straightforward because GitHub allows apps to have more than one active key at the same time. You can find your registered apps under Settings > Developer settings > GitHub Apps.
To rotate the key for an app, you need to click the “Edit” button and scroll all the way down until you see the “Private keys” section.
As you can see, we are able to generate a new private key without revoking the old one. In fact, GitHub doesn’t even allow us to revoke the old key until there is a new one available!
For scenarios like this, this is what our key rotation process looks like:
- Generate a new private key
- Deploy the new private key to our application
- Revoke the old private key
When we generated the new private key for our app, we didn’t need to worry about our app getting locked out because our old key stayed active. Services that allow multiple active keys are the easiest to work with when rotating secrets.
Rotating an Airbrake project key
Not all key rotations will be that easy, unfortunately, so let’s also walk through a difficult key rotation: Airbrake project API keys. Airbrake is a service that allows you to monitor your app for errors and performance issues.
You can find your Airbrake project’s API key right in the settings page on the right-hand side.
As you can see from the screenshot, each Airbrake project only has one active key, and the “Regenerate API Key” button will instantly revoke the current key and replace it with a new one. If our goal is to rotate the key with no down time, this puts us between a rock and a hard place.
We can’t choose what the next API key will be, so we can’t preload the next key into our app alongside the old key (the trick that we covered earlier). Our only option to avoid downtime is to create another Airbrake project, and deploy that new project’s ID and key to our application. This gets really messy because the new project won’t have all of the settings we changed in the old one. We could actively maintain two identical projects and switch between the two each time we need to rotate the API key, but that isn’t a good solution either.
In situations like these, our only option may be to regenerate and deploy the new API key as quickly as possible and accept a lapse in coverage for Airbrake monitoring. If you are a developer who is implementing authentication for your service’s API, take this as an example of what not to do to your API consumers. Allow for multiple active keys in the applications you develop.
Conclusion
We’ve covered a lot of ground in this article from the what’s, why’s, and how’s of key rotation to some real-world examples. To wrap it all up, I’ll capture all the advice above into a concise best practices list. If you want to be great at API key rotation, you need to:
- Record where your keys are being used
- Record who/what has access to an API key
- Rotate keys at least every 90 days
- Rotate keys when developers leave
- Rotate keys when they are leaked or compromised
- Create and deploy a new key before revoking the old one
- (Try the backup plans if the previous step isn’t straightforward)
- Automate rotation if the service exposes key management through their API
- Validate that keys are active/inactive as expected
- Allow for multiple active keys in the applications you develop
Hopefully at this point you feel like you have a concrete understanding of key rotation, and maybe you’re even starting to think about how to be better at this in your own software development lifecycle.
If you feel like you’re a long way from greatness in this area, just start with good documentation and take it one app at a time. You’ve got this!
