Secrets leakage is a growing problem affecting companies of all sizes, including GitHub. They recently made an announcement on their blog regarding an SSH private key exposure:

[Last week GitHub] discovered that GitHub.com’s RSA SSH private key was briefly exposed in a public GitHub repository.

The company reassured the public explaining that the key was only used to secure "Git operations over SSH using RSA", meaning that no internal systems, customer data, or secure TLS connections were at risk. They reacted immediately by detecting the incident and changing the key:

"At approximately 05:00 UTC on March 24, out of an abundance of caution, we replaced our RSA SSH host key used to secure Git operations for GitHub.com."

The impact has therefore been limited both in time and in scope: "This change only impacts Git operations over SSH using RSA. If you are using ECDSA or Ed25519 encryption, then you are not affected.", according to them.

This is further evidence that secrets sprawl is not just being driven by inexperienced developers or new teams. In our State of Secrets Sprawl 2023 report, we uncovered more than 10,000,000 new secrets that were pushed to public GitHub repos. This is a 67% increase in the number of secrets we detected over the previous year's report, while GitHub itself only saw a 27% increase in new accounts.

The increase in detected hardcoded credentials is driven by many factors, but it is clear that the developers involved in the incidents range in seniority from novice to expert and from organizations of all maturity levels. From our report, we discovered that 1 in 10 committers exposed a secret in 2022. If you have exposed a secret publicly, you are certainly not alone. GitHub serves as a good reminder we must stay vigilant in our security practices, no matter how large our team.

Let's take a look at what the risks are in this situation, how GitHub handled the remediation process, and some simple steps to avoid exposing your own private keys publicly.

The Risks Of Leaked Private SSH Keys

Later in their post, GitHub said: "We [replaced our RSA SSH host key] to protect our users from any chance of an adversary impersonating GitHub or eavesdropping on their Git operations over SSH. This key does not grant access to GitHub’s infrastructure or customer data."

While we can take comfort in this last part, no customer data is exposed. It is also good that there is no known risk of a takeover of their infrastructure, especially given how many developers and so much of the internet relies on it day to day.

But there is a serious risk from "adversary impersonating GitHub or eavesdropping on their Git operations."  This is what is referred to as a "man-in-the-middle attack," where the end user can not tell the difference between the legitimate other party and the attacker.

With so many developers and services relying on GitHub SSH communications to be truly secure, it made a lot of sense for the company to revoke and replace their SSH key. In that same article, they also published what you need to do if you are affected and how to tell if you are affected.  If you get a warning when connecting to GitHub via SSH, WARNING: REMOTE HOST IDENTIFICATION HAS CHANGED! you will need to remove the old key or manually update your ~/.ssh/known_hosts file.

Your GitHub Actions might be affected as well. GitHub's rotation of their private SSH key will mean workflow runs will fail if they are using actions/checkout with the ssh-key option. This might feel familiar to CirlceCI customers, who experienced a similar unexpected workflow interruption due to a leakage incident.

Reactions And Risk From Mass Replacement

As expected, the developer community has had many public reactions to the event. No one, it seems, is happy about the situation., but some people are helping spread the word without adding much commentary. Other conversations speculated on how this could have happened and further potential security issues that might come into play.

One concern that popped up in several conversations was the potential for a new man-in-the-middle attack. Attackers know that keys will be replaced by many developers by using the command ssh-keygen -R github.com without manually verifying the new fingerprint is the expected value. It is entirely possible that a bad actor could insert their own fingerprint in certain situations.

This is a great reminder that we should be embracing Zero Trust, moving away from 'trust, but verify' to a stance of 'verify, only then trust.'

A Lesson In Incident Remediation

While it is an unfortunate situation that GitHub has needed to rotate this certificate, affected so many customers, and put out such a notice, we want to highlight some best practices they used to respond to the event.

While we don't know how long the secret was exposed, we can assume it was fairly recently.

Acting quickly and deliberately, without panicking, is essential in secrets remediation. Given the timestamp they published, we can assume this incident caused a sleepless night for the team involved, and they had to weigh many factors. It seems they thought the security risks merited the key rotation, even if it would potentially affect a large swath of users.

We can also applaud them for communicating very quickly and publicly. While most people might not be subscribed to the GitHub blog, they pushed the news into multiple channels as publicly as they could. While some in the community felt there was a little bit too much 'marketing spin' when using terms like "we have no reason to believe" or "an abundance of caution," the GitHub team has been straightforward in their communications throughout all of their security incidents. We will take a little soft peddling versus not getting an overview that includes remediation steps.

How To Prevent Leaking A Credential

Nobody is perfect, and we all make mistakes from time to time. Expecting humans always to deliver flawlessly only sets you up for disappointment, especially when dealing with something as conceptually as complex as Git. While we all love the world's favorite version control system, most developers can tell you it is a bit too easy to do something wrong, such as pushing to the wrong remote repository.

While we don't know precisely what caused this particular incident, we at GitGuardian have a lot of experience with similar leakage situations and want to use this opportunity to remind you of some best practices.

Never Add Credentials To Version Control

This one might seem obvious, but all of us touching code are sometimes guilty of this. There are times, especially when debugging something when you just need to test a credential works. This is true of passwords, API keys, and, relevant to this, certificates.

You might add a new cert directly in the project with the full intention of moving it someplace safe or modifying your .gitignore file, but then life happens, and you get distracted. One git add -A and git commit later, your cert is now in your git history. One 'git push' later, and it is now out in a shared repo.  

This is where tools like git hooks and ggshield can really come in handy. Setting up a pre-commit hook is very quick and simple. Once set up, every attempt at committing code will trigger a scan that will halt the operation if a secret is discovered in any of the tracked files. Catching a secret before it becomes part of your git history is the safest and cheapest place to remediate the situation.

Double Check That Is The Right Remote

One of the strengths of Git is the ability to easily push all your changes to any remote repository you have permission to connect. Unfortunately, this can get confusing rather quickly. Imagine you have two repos, one public and one private, with remote names   proj-1-p and proj-1-pr respectively. You are only one letter away from potentially pushing to the wrong place. Accidents happen. It is a very good idea to make those origin names more explicit.

Another factor a lot of developers deal with is their own shell aliases. For example, it is common to alias gpo as a shortcut for the often typed git push origin. It can be very easy to go into automatic mode and use a shortcut that might send the wrong branch to the wrong place. When in doubt, type it out.  

Rotate Secrets Often

While rotating your SSH key might not be necessary for this event, do you know when the last time was that you rotated it? Was it this decade? If you don't know, then right now is a good time to go remedy that. The longer any valid credential lives, the more opportunities exist for that credential to be found and misused.

We have found that teams that rotate keys more often are at the top of the pyramid, expert level when it comes to secret management maturity. Going through the exercise of rotating credentials regularly when there is no emergency will prepare you to better handle times when there are those added pressures.

Stay Vigilant And Safe

Leaks happen to us all sometimes, even to massive platforms like GitHub. While there likely are many workflows that have been affected and many developers who need to update their known_hosts files, thanks to GitHub's clear communication on the incident, there is a clear remediation path, and we know the overall scope of the incident.

This is a good time to reflect on your own secrets management and detection strategy as well. GitGuardian is here to help.  You can help keep credentials, like certificates, out of your source control with ggshield. And if a secret does get pushed into a shared repo, the GitGuardian platform can let you know immediately, as well as help your team coordinate through the remediation process.

Stay safe out there; we are all on the internet together.