Skip the intro and jump to the guide.

After reading about leaked secrets and exposed credentials everywhere, your security lead became too tempted to peek at the private repositories in your organization. You start running a couple of regex searches like /AKIA[0-9A-Z]{16}/ thinking you won’t find much, when lo and behold, a handful of what looks like valid AWS API keys are highlighted.

You are now confident more secrets are lurking in this git repo. One branch, one regex doesn't do, so you and your security lead decide to take things further and run automated secrets scanning with GitGuardian (or Gitleaks, or TruffleHog) on every single repository. It only takes a few minutes before you find out how big the problem is, thousands of occurrences are detected. You now realize that you now have to deal with a giant pile of security debt.

We hear this very often at GitGuardian. Many Proof of Concept exercises we run at large companies with hundreds of developers and repositories share this similar story where thousands of results surfaced, revealing years of accumulated poor credentials hygiene. And if you find it hard to believe, read our latest reports in which we analyze the Twitch, Nvidia, and Microsoft leaked repositories.

Why is the threat of secrets-in-code different?

🔍
Exposed secrets are a unique breed of vulnerabilities found in source code. Whether the code is compiled and in runtime or not, hardcoded secrets represent a risk in themselves. Attackers who gain initial access to a repository can traverse all its branches and commit history to look for valid secrets. It doesn't matter if the secret is found on the main branch or a short-lived bugfix branch as long as it is valid and gives access to something.

With this in mind, we should consider every occurrence as a dormant threat that cannot be left untreated. But how can you pull this off when there are more incidents than your security engineers can ever handle?

Prioritizing your incidents

Start with a discovery phase

Code security practices, and secrets management, in particular, differ from one engineering team to the next. This usually translates to uneven repository hygiene and will show when examining the sources impacted the most by incidents.

Analyze your historical incidents to identify high-risk repositories

We recommend you start by determining the repositories that contain the most secrets and try linking them to the applications or services your development teams are maintaining before you decide what to tackle first.

Continue your discovery phase, but this time, focus on the secrets. Explore secrets exposure trends and evolution over time and try identifying the types of secrets that leak the most and the services they give access to.

Analyze your historical incidents to identify leaky secrets

Deploy honeytokens when dealing with a large secrets debt

Deploying GitGuardian honeytokens can serve as a crucial element in your secrets remediation strategy, especially when dealing with a substantial secrets debt.

As you embark on the journey to clean up these historical secrets, a process that might take months or even years, these honeytokens will act as an alarm system, offering protection during this vulnerable period. By deploying honeytokens strategically across your critical repositories, you create an effective alert mechanism.

How to Create and Use Honeytokens: Step-by-Step Instructions
Learn how to create, test and deploy GitGuardian honeytokens to detect security breaches, strengthen supply chain security, and prevent code leakage. Find out where to place honeytokens to effectively deceive attackers and protect your assets.
GitGuardian Blog - Automated Secrets DetectionThomas Segura

In the event of a breach, the honeytoken triggers an immediate alert, providing essential information for your security team to investigate and address the breach swiftly. This not only brings peace of mind but also helps in prioritizing secrets incidents.

Breached honeytokens signal compromised assets, guiding your team to prioritize and rotate critical secrets in those assets promptly, ensuring a proactive and efficient remediation process. This approach allows you to maintain a vigilant defense against potential security threats while steadily addressing secrets sprawl.

Define your severity levels

Now, before diving headfirst into remediating your historical incidents, we suggest you set a few rules for prioritizing them first. Here are some elements to consider before deciding on the severity of an incident and the priority it should get:

Remediating your incidents

Before starting this phase, you must establish precise remediation processes for your historical incidents. Ask yourself the following questions:

  • Which team/role will be responsible for remediating the incidents?
  • Which team/role will be accountable for tracking remediation progress and reviewing open incidents?
  • Which team/role will be consulted during the remediation process (e.g., DevOps engineers might have to intervene to avoid service outages while rotating secrets).
  • How much autonomy are we willing to give to developers to resolve incidents?

Scale remediation with the help of developers

It’s no secret that security engineers are heavily outnumbered by developers (with an average ratio of 100:1) and cannot possibly handle all the vulnerabilities generated by the latter's activity. To scale remediation, you must pull your developers closer to the process and ensure they contribute their fair share. Here’s how you can go about this:

Step 1. Identification and alerting

For every historical incident in your pile, identify the developer involved to share with them incident details. Simply use the commit author email here.

Step 2. Feedback collection

Once you have listed all incidents and their matching developers, you need to automate incident details sharing en masse and ask developers to submit their feedback. This will give you additional information such as:

  • Is this secret a test environment credential? Or is it a false positive?
  • Does it give direct access to sensitive services or data?
  • Is it used in a production environment?
  • Has it already been revoked by the developer?

Step 3. Resolution

You now have everything you need to prioritize and handle your incidents.

  • If you decide to trigger the entire remediation workflow, ask development and DevOps teams to coordinate and:
    • Start by revoking and rotating the secret,
    • Replace the secret in your code with environment variables or by invoking the correct calls to centralized secrets storage,
    • Optionally, rewrite the git history to remove all traces of the incident (we recommend against doing this, git can get very messy!).
  • Otherwise, simply close the incident. Make sure you add a flag (false positive, test key…) to avoid losing context if you need to return to the incident.

Step 4. Verification

You can mark the incident as' resolved' when you are informed that the steps were followed. You should also try to automate calls (HTTP, SSH, or other) to the associated host to verify if the secret is no longer valid and obtain proof of remediation.

As all teams advance and become more comfortable with their collaboration and remediation processes, we suggest your security teams embrace the trust but verify principle. In the long run, developers should be empowered to resolve incidents with little to no intervention from security teams.

Here's an example of an automated workflow you can try replicating.

Automated workflow for enabling developer-driven remediation (incident sharing and resolution)
Automated workflow for enabling developer-driven remediation (incident sharing and resolution)

You can learn more about how GitGuardian can help you automate granting access to developers to their own incidents in this 1-min video by Dwayne McDaniel, our Security Advocate.

Don’t forget ‘orphan’ incidents

Your remediation workflows need to factor in an important detail: is the developer involved in this incident still on staff? If the answer is no, you will have to do without their help or knowledge. Here’s what to do when faced with this situation:

Step 1. Identification

Start by verifying (when possible) the developer is no longer here:

  • Try pinpointing the team the developer belonged to when they were active (by querying your LDAP for example).
  • Identify repo admins or users with write access (at the incident location).

Step 2. Assignment

Once all teams impacted by orphan incidents are identified, assign the bulk of the historical incidents to engineering leads, repo admins, or other relevant team members.

Alternatively, try leveraging this helpful resource if you have a “Security Champions” program. Security Champions can take on the task of remediation on behalf of the developers no longer on staff.

Before you go

Remediating years of historical incidents of hardcoded credentials may seem unrealistic at first. But with careful preparation, reasonable resources, and the right dose of automation, we believe it can be perfectly pulled off.

This guide explored some of the complex aspects of remediating secrets-in-code incidents. Ultimately, secrets detection and remediation in AppSec programs aim to prevent more and remediate less. Focusing on resolving past incidents while doing nothing to avoid new ones will not result in lowering the total count and, in turn, reducing the effort needed for remediation in the future.

Keep your eyes on our blog, we will soon release a guide on reducing the risks of secrets exposure in the SDLC!