Jason Haddix is someone that needs little introduction in the world of security. He has a long history of successful exploits in bug bounty, gaining the number 1 spot in Bugcrowd in 2014, he transitioned into security leadership, becoming the CISO and head of security at Ubisoft from 2019 - 2022, a company with more than 25,000 employees. Today he combines his tech leadership skills with his penetration testing skills as the CISO at BuddoBot, a world-class penetration testing organization.

Jason isn’t unfamiliar with the tactics and techniques of hackers, not only has he been able to break into many organizations himself, but during his time at Ubisoft, the notorious hacking group Lapsus$ broke into their infrastructure. Fortunately, the outcome wasn’t as severe as many previous victims, thanks to Ubisoft's internal security, but the experience still left him with some fundamental learnings that he shared with me during the Security Repo Podcast.

“The TTPs (tactics, techniques, and procedures) of the attackers like Lapsus is to move around your systems silently.” Jason Haddix

“The vaccine is to have a secrets management program so they can’t pivot” Jason Haddix

A Comprehensive Secrets Management Program

I had the opportunity to sit down with Jason recently, and I asked him what advice he would give to organizations to enhance their security? His answer was to build a comprehensive secrets management program.

Following the Lapsus$ breach, Jason told me he spoke to more than 40 other CISOs, including many from organizations that the gang had targeted as well, after discussing the issue with them he came up with a 4 step guide to build a complete secrets management program. These steps are to Detect, Prevent, Respond, and Educate. In this article, we will break down each step, diving into Jason’s advice.

“One thing I learnt from Lapsus is how important a secrets management program is” - Jason Haddix

Detect

“You have to have a detection mechanism” - Jason Haddix

Leaked secrets, or what we often call secrets sprawl, is a vulnerability that affects nearly all organizations. What makes this challenging is that we often don’t even know we have a problem because secrets are out of sight. They could be buried inside your git history, inside messaging systems like Slack, your internal documentation, or even inside ticketing systems like Jira.

“You need a tool where you can find them latently where they are now” - Jason Haddix

The first step of Jason’s secrets management program is to identify where your leaked secrets are. This includes finding them inside your repos, including all your branches and history, inside your communication channels, inside your internal documentation, and any other tools where your team is sharing or storing data. Humans are very bad at identifying secrets inside code, logs, and other data systems, so it is important to have a tool that can perform this task. While there are many, GitGuardian specializes in identifying secrets in large enterprise systems, not just historically but also in real-time. Regardless of what tool you choose, some advice Jason did give was to make sure you are able to build your own custom regex to identify not just the standard secrets that come out of the box but the secrets you create internally.

“We built our own custom regex for the secrets that were only ours” - Jason Haddix

Secrets detection can also be built into many different areas of the organization and is not something that is just done once. As Jason explained, at Ubisoft, they built secrets detection into their red team's mandate as well as putting it into all their build pipelines.

Prevent

“Detect is to stop the bleeding, and prevent is to help you build for the future” - Jason Haddix

Once you have detection in place to catch the secrets that have leaked, you need to put in a mechanism that is able to prevent the problem in the first place. Jason explained that the most efficient tool they found to be able to prevent secrets from leaking was using pre-commit hooks for all developers. A pre-commit hook is a script that is run during the git workflow and has the ability to identify secrets and block any commits that may contain them. This is powerful because it means the secrets never make it into the servers, meaning it doesn’t become sprawled and compromised. To learn more about git hooks, see GGShield.

“The most efficient thing we found was using a pre-commit hook in the developer lifecycle” - Jason Haddix

Not only are pre-commit hooks a great way of preventing leaks, but they can also be used to educate. Jason explained they made sure that if the hook were triggered, meaning there were secrets leaked, the hook message would also contain information on what is the preferred method of storing and handling secrets which leads into the “Respond” section.

Respond

“You can’t just block them from doing it if you don’t give them a correct way to handle secrets properly” - Jason Haddix

One of the most important steps in building a secrets management program is to have tools to store and share secrets along with clear policies on how to use them. Secrets need to be locked down, with fine-grain access control and logs on who has accessed the secrets and where they have ended up. Secrets managers are the obvious tool to do this. While there are many tools out there, one item that Jason specifically brought up when evaluating tools was being able to rotate secrets (revoke existing secrets and replace them with new ones). Even with the best strategies, you can never be certain that a secret won’t end up sprawled somewhere, so it is important that secrets are frequently rotated.

“A great policy for vaulting secrets was rotating them automatically” - Jason Haddix

Rotating secrets is much more difficult than it sounds, especially at the scale of a large organization and considering that revoking access to a secret could potentially cause system outages. Some tools have a built-in automation of secrets, these tools create secrets for limited-time use, sometimes even one-time use, and then destroy the secrets automatically. This can’t be done for all secrets at this stage but is a great additional layer of protection.

Educate

“We had to go on a tear educating all developers and employees about not sharing secrets verbatim” - Jason Haddix

All the tools in the world won’t help you if your staff does not know how to use them correctly or does not understand the risk associated with hardcoding secrets. Having continued learning sessions about secrets, not just for developers but for all corporate passwords for all employees.

If you are wondering where to start with educating developers, this blog and the GitGuardian YouTube channel have some great content design for developers on how to and how not to handle secrets.

If you want to learn how to educate developers in your organization, reach out to GitGuardian for a free Lunch and Learn session on secrets management.

Final Words

Building out a comprehensive secrets management program takes time and effort. But it is extremely important as modern attacks are leveraging secrets not just for initial access but to move laterally and elevate privileges once inside systems.

Of course, every secrets management program is going to be different for every organization, but Jason’s 4 steps program is a great place to start. If you don't go further, you may consider looking at our Secrets Management Maturity Model, where you can evaluate your current policies and tools against leaders in the space.

Notes: This conversation was part of an episode of GitGuardian's Podcast The Security Repo, Haddix, BuddoBot, and UbiSoft are not affiliated with GitGuardian.