Secrets management takes more than just tools

Dwayne McDaniel -

Everyone wants good security. According to Gartner, worldwide spending on security is forecast to grow 11.3% in 2023 to reach more than $188.3 billion. It is clear that security is a priority and that teams are spending budget to find solutions to code security issues.

And yet, secrets sprawl remains a growing problem. That will not come as a surprise to anyone who has read the GitGuardian State of Secrets Sprawl report, where we found 10 million secrets on public GitHub repos in 2022. In 2023, we saw reports that over 50% of attackers gained initial access with compromised credentials across all investigated breaches. Nobody wants to be the next Samsung or Nvidia, leaking thousands of secrets when a code leak occurs. No board of directors wants to be featured on the front page of a major newspaper for a security breach like Uber experienced. No team wants to have their GitHub admin credentials found, granting access to dozens of GitHub orgs and hundreds of repos, like one recent car company recently experienced.

Dealing with secrets sprawl is top of mind for many organizations, but this is not an issue where secrets management tools alone can save the day. So what can solve secrets sprawl? Fortunately, we have been working with a lot of organizations since 2017 and have some keen insight into how the companies with the best security posture are getting it right.

A good lock alone is no guarantee of security

A hard-to-pick deadbolt on your front door is meaningless if someone leaves the door propped open. If someone forgets to set the lock when they are in a hurry, it defeats the most heavy-duty hardware. If they leave the key out front of the door, that lock will be very quickly opened and therefore useless.

Just having the lock does not mean that your team is aware they should lock that door. The lock is certainly a needed component of good physical security, but the device unto itself does not automatically mean you have a secure facility.
Good security requires three fundamental pillars:

  • People - Kept aware of the issues and properly trained on the tooling and processes.
  • Processes - Clear documented procedures for creating, storing, accessing, and rotating secrets.
  • Tools - Credentials storage and management, as well as detection and leak remediation.

If we think of our example of locking the front door, there are many things we need to consider and steps to take to ensure the door is properly secured. For example, we would need to:

  • Create awareness that having an unlocked door is an issue.
  • Buy and install the lock.
  • Create clear documentation on when and how to use the lock.
  • Train the team on properly using the lock, adding this training to any new team member onboardings.
  • Automatically check if the door is unexpectedly unlocked.
  • Periodically rotate the key.
  • Continually review our processes to ensure they are up to date.

Just as acquiring the lock itself is no guarantee of security, no one piece of our plan in isolation would make much sense. Just writing a process without communicating it would never do anyone any good. Training the team on using the lock without actually installing the lock would be silly.

Let's take a closer look at those three pillars, People, Processes, and Tools, needed for good secrets management.

Good secrets management takes people

Every security issue has a people component. Just as in our example with the lock, if no one knows what security issues your team is working to solve, what security tools you have, or how to use them, then the likelihood of making progress to a more secure organization is very low.

There are two major components to keep in mind when thinking about education for teams within your company:

  • Awareness - Explain the issue, what risks it brings, and what can be done to address it.
  • Training - How to use the specific tools that are available to solve the issue.

This is essentially the "why" and the "how" of security. While these certainly can be combined into one training, any internal sessions that only list a set of rules without explaining the larger context are not likely to be too effective. It is vital to get everyone on the same page, especially since security crosses team boundaries.

Raising awareness across teams

One of the largest issues organizations face is that the teams responsible for creating secrets, mostly developers, are not the same teams charged with the security of those secrets. While dependent on one another, in a lot of organizations, devs and security teams work in silos, only communicating during the security testing phase of the software development lifecycle. An adversarial relationship can emerge. Developers overall want to write more secure code, and all security folks want developers to follow security best practices, so how do we get there?

Developers and security teams work differently. Developers tend to jump straight in and figure things out as they go, learning as they build. Security teams tend to read a lot of cybersecurity news and 30+ page PDFs, keeping ahead of the latest threats and vulnerabilities. While it is impractical and unrealistic to expect developers to keep up with every new security bulletin that emerges, it does not mean developers don't care about evolving threats. Communicating any major and central security concerns does not have to be a dry and burdensome task.

One way to raise awareness effectively for your teams is to hold "lunch and learns." These tend to be less formal gatherings where devs get to learn the latest about a certain issue. The goal would be to get everyone to use the same language about the problem set and possible solutions. Getting the conversation started without the added pressure to learn a new tech or tool at the same time can really help the drive toward your security goals.

Raising awareness across all teams is at the heart of many security champion programs. You can hear more about security champions from Tanya Janca during her appearance on The Security Repo Podcast.

Training for security success

The word 'security training' can cause an unpleasant reaction in some individuals. Many folks will immediately envision an overlong video explaining something about weak password creation or not clicking on any strange attachment. But if you have properly raised awareness and have invested in automation wherever possible, training for how to use a tool or following the right processes can be a more productive and rewarding experience.

Automation is really key in getting any tool adopted. While training can derail down technical rabbit holes and deep dives into all possible menu items, if the focus is instead on how to use the tool to drive productivity most effectively, teams tend to be more receptive. Instead of how to run every iteration of the command line tool, try focusing on creating git hooks. Instead of training on how to manage low or info-only alerts, you show how automation can automatically ignore and resolve those, as is the case with GitGuardian playbooks.

Process training can also be a useful time to review those processes and policies. What new questions came up? What new threats have emerged since the last time this training was given? Updating your processes is equally as important as training people to use them.

Training should be engaging and show how using the tools and processes achieves the goals that they were made aware of. The overarching goal of any training should be to improve communication. The more effectively a team communicates, the better they will be at solving issues.

Having the team aware of the issue and then trained on the tools means that when a request for feedback comes in, it is a routine operation. A simple collaboration between teams and not a butting of heads. More effective collaboration was one of the drivers behind our feedback system inside the GitGuardian dashboard. Getting feedback is a simple matter of inviting the developer involved to add some context to the incident and potentially resolve the incident right then and there. We want everyone working as a team to solve problems.

Consistent processes make for consistent results

If you want to make a cake, you should follow the directions for a cake. Sounds pretty simple, right? You wouldn't just hand someone all the ingredients for a cake, walk away, and then honestly expect a cake when you return. To get the results you want, you need to explain what you want and provide some instructions.

But when it comes to cybersecurity, a lot of teams simply give tools to team members and expect good results. If you want good results, then you need to create and communicate good processes. No matter what you are doing, be it creating a new credential for a service in your DevOps ecosystem or rotating a secret for a DB, there is likely a preferred, safe way to accomplish the task.

Initially, using a whiteboard to create a desired flowchart can be a great way to think through processes. If you turn these into Kanban or swimlane-based flow diagrams, that can make a great basis for any written procedures as well. No matter how you create or think of your processes, they are only ever good if you effectively communicate them. Communication involves both the documentation process and the training and distribution components.

Having your process laid out in an easy-to-find location can help ensure they get read and followed. This is why we include the Remediation section in the GitGuardian Dashboard Incident detail view. This customizable section can list your specific remediation steps, linking to your documentation and tools you expect the team to leverage.

The right tool for the job matters

Having the right processes and the best training on earth can only take you so far. When you are making a cake, you won't get very far without the ingredients, the cookware, and an oven, no matter how much training you have or how organized your recipe is. Getting the right tools in place means asking what issues you are trying to solve.

Just buying generic 'security tools' might help with some general security, but focusing on a more specific problem set means you have a higher chance of solving what is more critical. Obviously, at GitGuardian, we see the issue of secrets sprawl as one threatening all of our apps, our supply chain, and critical infrastructure. CISA and NIST agree secrets management needs to be addressed.

A vault only works if you lock valuables in it

Secrets managers such as Vault by HashiCorp or Doppler, or platform-based vault systems like Azure Key Vault or AWS Secrets Manager, are the bedrock of good secrets security. The benefits of using one of these tools are extensive. They provide encryption at rest and in transit, allowing programmatic access, thereby freeing devs from ever having to know or ever encode API keys.

Also, importantly, vault systems can provide access and change management logging so you can see who has added or modified any credential, as well as when keys were accessed. This is a vital component of secrets management.

But you only get these benefits from the consistent use of vaults throughout the organization. Developers are often given free rein, at least in the early days of a project, and they might have implemented their own secrets manager. They might have instead relied on `.env` files, which are not encrypted locally, which can have some major implications.

It is only when you get everyone aware of the benefits of using secrets management correctly and at scale that you can begin to have any training or process make headway in your organization. The tool, just sitting there, bypassed by parts of, or whole, applications, means secrets are not being managed. When you do invest in the tools, remember to hold off from marking the secrets management security box on your checklist until it is consistently and widely used.

Scanning by itself is not good enough

When you think of secret detection, a vital component of secrets management, you might immediately think of code scanning and using regular expressions. These are indeed needed puzzle pieces, but they only tell a small, initial part of the story. Just as important as detecting the secret is what you do with it once it is detected. What other information do you have? How do you communicate the findings and remediation path to everyone else?

These questions were top of mind when we developed the GitGuardian dashboard. Once a scan is run, any incidents are collected into a single, API-driven workspace view. The dashboard gives you context for each discovered secret, telling you where it was discovered when it was introduced to the codebase, and for many of the secrets we detect, if the secret is valid. This is true for both historical scans, which look through each commit and branch, and for active scans, when a new commit is added to a monitored repository.

Once all the needed information is gathered, the team can set to work on the remediation process, which can be conveniently customized and listed in the incident details view. The dashboard makes it easy to communicate with the developer involved, allowing them to provide vital context about this secrets incident, as well as the other security team members to the status of the incident.

Communication is the cornerstone of good teamwork. Tools should make communication easier and more efficient, allowing faster responses when incidents do occur. One of the pillars of security posture is incident response. This includes how you plan, execute, and how you incorporate your learnings back into your procedures and training. Great security posture means having the right tools, which makes it easy for everyone to follow the right processes.

The GitGuardian Secrets Management Maturity Model

With so much going on in the secrets management space, it can be hard to keep track of what to do next. It would be great to have a roadmap, helping you determine where you are on the path to better secrets management and what steps might prove the most valuable to your organization. If you have been looking for such a way finder, we have some good news; we built one.

Working with our customers, we have developed the Secrets Management Maturity Model, which you can download and share for free right now.

Our model lists 5 distinct levels of secret management maturity: from zero, the uninitiated, with no tooling or process in place, all the way to level four, the expert level, which can list automated rotation and advanced tooling standardized throughout their org among their achievements.

The model acknowledges that the state of secrets management in modern software development is always a mixture of various, many ad-hoc, solutions. The model maps to four major areas:

  • Developer environment
  • Source code management
  • CI/CD pipeline & artifacts
  • Runtime environments

At every level, we also delve into what processes and practices we see successful and the not-so-successful companies put in place.

The model is meant to be a helpful tool to identify your current state and help you find ways to improve. We know that every organization has a different story and is likely somewhere between states at any given time. It is never meant to shame or judge anyone. We hope you can use it to have better conversations internally about improving your security practices.

If you are curious how your team stacks up against other practitioners out there, we collected the results of our survey of over 500 IT decision-makers into a report as well. The Voice Of Practitioners report is full of data about how many others have gone through a secrets leak (spoiler, 75%), what common practices they believe their teams undertake (manual vs. automated code reviews), and what they plan to do about improving the situation.

Evaluate your secrets' security maturity

We want to help everyone improve their security. We know it can be daunting to start the self-evaluation process, determining where you are on the road to secrets management maturity. This is why we have put together a 5-minute self-evaluation quiz. Results are anonymous, and no identifying information will be asked. From there, you will be able to give an educated opinion on what you can do to improve your overall code security from a secrets management perspective.

If you are worried that your secrets have leaked into public GitHub repos, we can help you get a handle on that situation. GitGuardian Public Monitoring is always scanning all public repos on GitHub, with over a billion scans in 2022. We are happy to work with you through a free secrets audit and evaluate your exposure level.

If you want to talk about GitGuardian Honeytoken for intrusion and leak detection, or Infra as Code Security, we would be glad to chat. And, of course, if you want to get started with the GitGuardian Secrets Detection platform and ggshield, the GitGuarian CLI, we would be happy to work with you. You can even get started for free right now.

GitGuardian is making the right tools for code security, and we would be happy to help you figure out the right processes and training to make the whole team more mature when it comes to secrets management and detection.