Secure by Default: Integrating GitGuardian in Vermeer's Software Development Lifecycle

My name is C.J. May, and I’m a Senior IT Security Analyst at Vermeer Corporation. We are a manufacturing company for agricultural and industrial machinery. Vermeer has over 3500 employees globally. Of those 3500, about 65 are active developers and 7 work in cyber security. We write a lot of code for our machines, and we also develop software in-house to support our dealers and our customers. Over the past few years, we’ve overhauled our software development lifecycle (SDLC), and that includes improvements to security.

Our aim is to make our software “Secure by Default,” which means that security isn’t something that gets tacked on at the end of a finished software product – it’s built into our processes at every stage of the SDLC. That way, the end product has already undergone security review before it is even deployed, and we have already captured most of the vulnerabilities.

There are multiple components to our DevSecOps stack, but I’m here to talk about secret scanning. In this article, I’ll explain which secret scanning tools we tried in the past, the pain points we uncovered, why we switched to GitGuardian and the value we have gotten since.

Where we Started

When our software development process started its overhaul, we didn’t have money immediately budgeted for security tools. Open-source scanners were our first solution to identifying secrets in our code. After some testing, we ended up settling on Gitleaks. 

Gitleaks is a command-line tool that will scan your entire git history for secrets. I liked that the rules in Gitleaks were easily customizable, which enabled me to maintain our own forked ruleset that ignored false positives and added internal detections.

At the time we started using Gitleaks, they didn’t have any GitHub Actions (they have since created one), so I helped maintain an open-source action that used Gitleaks. I deployed the action to all of our repositories, and we had secret scanning in all of our Pull Requests as well as a historical lookback of what we had missed before!

Tools vs. Platforms

Gitleaks turned out to be a dependable standalone tool, but over time I realized that there is a lot more to leaked secrets than having a functioning scanner. First of all, there was quite a bit of work involved in maintaining the custom Gitleaks action. I had to use GitHub’s API to deploy it to all of our repositories. There were also some edge cases that broke it on a few occasions, which led to developer frustration and debugging time on my part.

The custom action was also completely decentralized, which meant that I had to build out reporting functionality so we could review leaks and ensure that they were being taken care of. Another challenge that came from the lack of central management was dealing with false positives and remediated secrets. When there were detections that we no longer wanted to see, we used Gitleaks’ .gitleaksignore file. Disabling detections using the file worked fine, but there wasn’t a built-in way to document why a detection was being ignored.

Another challenge that developed over time was how to handle our historical leak cleanup. Prior to using Gitleaks, our codebase’s only secret scanner was our developers’ eyes. This meant that we had some secrets in our code already that needed to be cleaned up. We could go through each one of our 1000+ repositories and run a full history scan with Gitleaks, but that would take a long time, and we would have no way to prioritize and find the most important secrets first.

Overall, there was a lot of extra time spent on engineering in order to support our secret scanning needs with an open-source tool. As time passed, we asked for budget to improve the security of our SDLC. Once that was approved, we decided to improve our secret scanning processes with a comprehensive platform that would be more cohesive and include all the nuts and bolts we were previously engineering ourselves.

Evaluating Secret Scanning Platforms

Before we set out on our exploration, we did some research and defined success criteria for the platforms we would be evaluating. It’s kind of a long list, but others who are in a similar position may find it helpful, so I am keeping it unredacted:

• Detect secrets with low False Negative & False Positive rates
• Full Git history scanning
• Coverage and automation at all stages of the SDLC (Code, CI, Artifacts, Cloud)
  • IDE plugin
  • Pre-commit hook
  • Push protection
  • Repository scanning
  • Pull Request check
  • Container image scanning
  • Filesystem scanning
• Public leakage monitoring
• Credential validity checks
• Custom secret detection via regex
• Ease of deployment
• Ease of management
  • Ignoring false positives
    • Ignore reason logging
    • File path exclusions
    • Regression handling
  • Handling true positives
    • Active credential validation
    • Comments
    • Assignees
    • Jira integration
    • Same-secret grouping
• Minimal friction with developer workflows

We evaluated multiple secret scanning products. We also checked out full SAST solutions, and we found that some SAST products could detect exposed secrets. However, they weren’t as accurate, and they would handle a leaked secret like any other discovered vulnerability, which didn’t help us prioritize historical leaks. They also lacked features like validity checks and public leak monitoring.

In the end, GitGuardian emerged as the clear leader in the secret scanning space from a feature perspective. As I’ve already explained, though, features alone aren’t valuable if they don’t enable cohesive processes for the end users. With that in mind, I’ll get into our experience with GitGuardian since we became customers.

The Value of GitGuardian

GitGuardian’s 350+ secret detectors are the best at identifying secrets without flagging a bunch of things that aren’t actually sensitive. We evaluated other tools that had decent secret scanning as well, but for us, what sets GitGuardian apart is the user experience for everyone involved.

On the developer side of things, GitGuardian is seamlessly integrated into the tools that our developers already use, such as GitHub Pull Requests and Git pre-commit hooks. GitGuardian has built-in guides so that developers know what to do when a scan does detect a secret. They also let you modify the message with your own information or links to documentation. Having tight integrations into existing processes reduces context-switching for our developers, which lets them focus on the things they’d rather be doing.

On the management side, GitGuardian has relieved every pain point we previously had when we were patching together our own secret scanning setup with Gitleaks. GitGuardian’s platform centrally monitors all of our repositories in one place, which enables us to keep track of the secrets that have been dealt with. We can also assign leaks to individuals with fine-tuned access policies so they can leave comments and track the leak through to remediation.

GitGuardian also helped us tackle our historical leak cleanup in an organized and efficient manner. After we scanned all of our GitHub repositories, I went through and tuned our severity rules. Now all of the secrets that GitGuardian finds are ranked in a way that is most relevant to our business.

Conclusion

Overall, I’ve been very impressed with the way GitGuardian has performed for an organization of our size and complexity. Not only that; I’ve gotten a lot of engineering time back since we acquired GitGuardian’s internal monitoring. If we had gone with GitGuardian from the beginning, I probably would have saved over 100 hours of setup and maintenance that we put into an open-source solution.

GitGuardian reduces the burden on your developers and platform engineers by reducing complexity, integrating smoothly into existing processes, and providing exactly what you need, where, and how you expect it. GitGuardian’s platform has already become a valuable part of our DevSecOps SDLC, and we greatly appreciate the time it saves us and the value it provides.