In our previous article of the two-part series, Implementing a Secrets Detection Program for the Enterprise, we explored some of the complex aspects of dealing with a security debt consisting of thousands of hard-coded secrets.
Digging through the remote past is a critical step toward mitigating the threats of secret sprawl in the SDLC, but a secrets detection program shouldn't just end here. As you progress and your program matures, you will want to prevent more and remediate less.
This article will discuss the approach followed by one of GitGuardian's enterprise customers to implement a secrets detection program and stop poor secrets management practices at the source.
Let's dive in.
The setting
A quick overview of the company
GitGuardian partnered with a U.S multinational cybersecurity company to deploy its enterprise secrets detection and remediation platform for more than 5,000 developers.
This company has a global footprint, serving thousands of customers worldwide. Its core products and services address network and endpoint security, but the offering extends to cloud-native application protection and security operations.
Why secrets detection
Security leadership at this cybersecurity multinational was well aware of the risks of exposed secrets. A multitude of factors eventually led to giving secrets detection and remediation a high priority in the company's AppSec program:
- The surge in high-profile incidents and supply chain attacks involving leaked secrets or source code (Solarwinds, Codecov, Daimler-AG, Nissan, United Nations, State of New York).
- Finding exposed credentials via external pentests, code reviews, and internal red teaming exercises. No organization is immune to this.
- Robust M&A activity (external growth through company acquisitions), introducing new technical environments and codebases.
- Regular customer requests to provide attestations that applications and source code are continuously scanned for vulnerabilities (see NIST Minimum Testing Standards for Software Vendors).
Objectives of the secrets detection program
Different organizations and stakeholders (CISO, VP of security, Technical Security Leads, and Directors of Engineering) may set various goals for a secrets detection program. However, one goal all organizations have in common is reducing the risks of secrets exposure.
In addition, this cybersecurity multinational was looking to
- Shift security testing left in the software development process and extend to the right with visibility and control for AppSec teams.
- Prove return on investment (ROI), on top of a lower Total Cost of Ownership (TCO) compared to a homegrown solution based on open-source detection.
- Find a way around talent shortage in application security and contain the need for recruiting new full-time equivalents with the help of automation.
The journey to a zero secrets-in-code policy
Sizing up the problem
The company ran its first Proof Of Concept (POC) exercise on a sizeable perimeter, a codebase to which ~1,000 developers have been actively contributing. The complete history scan revealed thousands of hard-coded credentials.
GitGuardian also showed through deeper analysis that developers were introducing, on average, a few hundred credentials every month to this perimeter – providing strong motivation for what the security leadership later dubbed "stopping the bleeding."
Goodbye, remediation. Hello, prevention.
GitGuardian's native integrations with Version Control Systems (GitHub, GitLab, Bitbucket) and CI/CD tools surface real-time incidents and policy breaks for security teams. This downstream detection gives visibility and triggers Incident Response workflows (to remediate the hard-coded credentials). It's crucial to understand scanning is informative at this stage and does not introduce any security gates.
Since this AppSec team is short-staffed and already managing other vulnerabilities caught by SAST and SCA scanners, it realized it needed to introduce guardrails for developers to block push events with commits containing hard-coded secrets. In other words, stopping the bleeding. GitGuardian helped the team set up ggshield (command-line interface) with pre-receive git hooks.
This implementation is the most powerful. It is deployed once for self-hosted GitHub or GitLab instances and delivers push protection for all incoming code contributions. Here's how it works:
- When GitGuardian catches a secret, the developer must remove it from their local git commit before attempting another push.
- If the developer believes it to be a false-positive or is willing to take the risk and push what seems to be a non-sensitive secret, they can do this with the help of a break-glass option.
A Proven Return on Investment (ROI)
With this mechanism, GitGuardian Internal Monitoring is now reducing the number of incidents by about 80% without adding unacceptable friction to the development workflow.
It also frees considerable resources for the AppSec team since it no longer has to deal with hundreds of verified incidents every month. On average, security engineers now have to review ~50 alerts for checks bypassed by developers – requiring the complete revocation and rotation of the secrets.
What's next for this company
Getting rid of the security debt
With the prevention of new incidents, the AppSec team can now focus on security debt and dig through historical incidents to mitigate the threat of secrets hard-coded before implementing GitGuardian.
To learn more about how to plan and conduct this initiative, we recommend reading part 1 of this guide, Implementing a Secrets Detection Program for the Enterprise – Investigating, prioritizing, and remediating thousands of incidents.
Ziad Ghalleb
Deploying additional pre-release tooling for developers
Adding automated secrets detection to pre-receive hooks gives security teams peace of mind knowing that almost no secrets will come through the gate. But it comes with a tiny expense for developers that have to rewrite their local git history to remove the secret.
To avoid this, developers can opt for a pre-commit git hook to scan the git index before inserting the changes in the commit history. When ggshield (GitGuardian CLI) detects secrets in this staging area, developers can remove them from the text file (no git commands needed) before committing again.
One thing to note, however, is that centralized security teams cannot enforce this level of protection. Instead, they need to rely on the development teams' goodwill and bet on the slick developer experience GitGuardian delivers.
Before you go
Over time, the GitGuardian team has acquired sensible expertise in deploying secrets detection programs for large organizations.
At the start of 2021, the ten largest organizations using GitGuardian Internal Monitoring averaged 1,380 developer licenses. One year later, this number has almost doubled to reach 2,688 developer licenses per deployment, a testimony to the maturity of GitGuardian's platform and support.
Want to learn more about how GitGuardian can help your organization keep your secrets protected? Get in touch with one of our experts or go ahead and sign up for a free trial.
