Skip intro and jump to guide.

After reading about leaked secrets and exposed credentials everywhere, your security lead became too tempted to peek at the private repositories in your organization. You start running a couple of regex searches like /AKIA[0-9A-Z]{16}/ thinking you won’t find much, when lo and behold, a handful of what looks like valid AWS API keys are highlighted.

You are now confident more secrets are lurking in this git repo. One branch, one regex doesn't do, so you and your security lead decide to take things further and run automated secrets scanning with GitGuardian (or Gitleaks, or TruffleHog) on every single repository. It only takes a few minutes before you find out how big the problem is, thousands of occurrences are detected. You gaze at each other in disbelief and realize that you now have to deal with a giant pile of security debt.

We hear this very often at GitGuardian. Many Proof of Concept exercises we run at large companies with hundreds of developers and repositories share this similar story where thousands of results are surfaced, revealing years of accumulated poor credentials hygiene. And if you find it hard to believe, read our latest reports in which we analyze the Twitch, Nvidia, and Microsoft leaked repositories.

Why is the threat of secrets-in-code different?

🔍
Exposed secrets are a unique breed of vulnerabilities found in source code. Regardless of whether the code is compiled and in runtime or not, hardcoded secrets represent a risk in themselves. Attackers who gain initial access to a repository can traverse all its branches and commit history to look for valid secrets. It doesn't matter if the secret is found on the main branch or a short-lived bugfix branch, as long as it is valid and gives access to something.

With this in mind, we should consider every occurrence as a dormant threat that cannot be left untreated. But how can you pull this off when there are more incidents than your security engineers can ever handle?

Prioritizing your incidents

Start with a discovery phase

Code security practices, and secrets management, in particular, differ from one engineering team to the next one. This usually translates to uneven repository hygiene and will show when examining the sources impacted the most by incidents.

Analyze your historical incidents to identify high-risk repositories
Analyze your historical incidents to identify high-risk repositories

We recommend you start by determining the repositories that contain the most secrets and try linking them to the applications or services your development teams are maintaining before you decide what to tackle first.

Continue your discovery phase, but this time, focus on the secrets. Explore secrets exposure trends and evolution over time and try identifying the types of secrets that leak the most as well as the services they give access to.

Analyze your historical incidents to identify very leaky secrets
Analyze your historical incidents to identify leaky secrets

Define your severity levels

Before diving headfirst into remediating your historical incidents, we suggest you set a few rules for prioritizing them first. Here are some elements to consider before deciding on the severity of an incident and the priority it should get:

Remediating your incidents

Before you kick off this phase, you need to establish precise remediation processes for your historical incidents. Ask yourself the following questions:

  • Which team/role will be responsible for remediating the incidents?
  • Which team/role will be held accountable for tracking remediation progress and reviewing open incidents?
  • Which team/role will be consulted during the remediation process (e.g. DevOps engineers might have to intervene to avoid service outages while rotating secrets).
  • How much autonomy are we willing to give to developers to resolve incidents?

Scale remediation with the help of developers

It’s no secret that security engineers are heavily outnumbered by developers (with an average ratio of 100:1) and cannot possibly handle all the vulnerabilities generated by the latter's activity. To scale remediation, you need to pull your developers closer to the process and make sure they contribute their fair share. Here’s how you can go about this:

Step 1. Identification and alerting

For every historical incident in your pile, identify the developer involved to share with them incident details. Simply use the commit author email here.

Step 2. Feedback collection

Once you have listed all incidents and their matching developers, you need to automate incident details sharing en masse and ask developers to submit their feedback. This will give you additional information such as:

  • Is this secret a test environment credential? Or is it a false positive?
  • Does it give direct access to sensitive services or data?
  • Is it used in a production environment?
  • Has it already been revoked by the developer?

Step 3. Resolution

You now have everything you need to prioritize and handle your incidents.

  • If you decide to trigger the full remediation workflow, ask development and DevOps teams to coordinate and:
    • Start by revoking and rotating the secret,
    • Replace the secret in your code with environment variables or by invoking the correct calls to centralized secrets storage,
    • Optionally, rewrite the git history to remove all traces of the incident (we recommend against doing this, git can get very messy!).
  • Otherwise, simply close the incident. Make sure you add a flag (false positive, test key…) to avoid losing any context in case you need to come back to the incident.

Step 4. Verification

When you are informed that the steps were all followed, you can mark the incident as ‘resolved’. You should also try to automate calls (HTTP, SSH, or other) to the associated host to verify if the secret is no longer valid and obtain proof of remediation.

As all teams advance and become more comfortable with their collaboration and remediation processes, we suggest your security teams embrace the trust but verify principle. In the long run, developers should be empowered to resolve their incidents with little to no intervention from security teams.

Here's an example of an automated workflow you can try replicating.

Automated workflow for enabling developer-driven remediation (incident sharing and resolution)
Automated workflow for enabling developer-driven remediation (incident sharing and resolution)

Don’t forget ‘orphan’ incidents

Your remediation workflows need to factor in an important detail: is the developer involved in this incident still on staff? If the answer is no, you will have to do without their help or knowledge. Here’s what to do when faced with this situation:

Step 1. Identification

Start by verifying (when possible) the developer is no longer here:

  • Try pinpointing the team the developer belonged to when they were active (by querying your LDAP for example).
  • Identify repo admins or users with write access (at the incident location).

Step 2. Assignment

Once all teams impacted by orphan incidents are identified, assign the historical incidents in bulk to engineering leads, repo admins, or other relevant team members.

Alternatively, if you have a “Security Champions” program in place, try leveraging this helpful resource. Security Champions can take on the task of remediation on behalf of the developers no longer on staff.


Before you go

Remediating years of historical incidents of hardcoded credentials may seem unrealistic at first. But with careful preparation, reasonable resources, and the right dose of automation, we believe it can be perfectly pulled off.

In this guide, we explored some of the complex aspects of remediating secrets-in-code incidents. Ultimately, the goal of secrets detection and remediation in AppSec programs is to prevent more and remediate less. Focusing on resolving past incidents while doing nothing to avoid new ones will not result in lowering the total count and, in turn, reducing the amount of effort needed for remediation in the future.

Keep your eyes on our blog, we will soon release a guide on reducing the risks of secrets exposure in the SDLC!