Sandusky, Ohio, is most likely known for Cedar Point, home to some of the largest and fastest roller coasters on Earth. Being the epicenter of family entertainment in the Midwest, the area has also become home to other amusement parks, particularly indoor water parks. One of these parks has been home to an annual community-led, developer-focused conference since 2006, and this year marked 19 years of CodeMash.

Around 1,100 attendees gathered for four full days of talks and workshops with a focus on leveling up everyone's skills and keeping the community up to date with the latest technology advancements. One of the things that makes CodeMash special is that it is very family-friendly, actively inviting kids to participate in all sorts of interesting sessions through its KidzMash program, which serves as a day camp while parents are off digging into technical sessions.

Happening entirely indoors at the Kalahari Resort, one of the largest indoor water parks in the world, meant no one had to go far for social events or, for those staying in the hotel, even brave the harsh Ohio winter. The parties and special events really made this community-driven event feel very connected, and I know I left with some of my favorite memories from any in-person event. 

Workshops beyond 101s

The first 2 days of the event are the "pre-compiler sessions," 4-hour long workshops covering a wide range of topics. Sessions covered everything from Linux server troubleshooting to GitHub Actions, to security training like web app pentesting, to front-end topics like building a web component library. 

As one of the attendees put it, since so many attendees have a lot of experience in tech, "these sessions really feel like level 201 or 202 classes." It was correctly assumed, at least for the sessions I took part in, that we all knew how to SSH into a server or could get Docker up and running on our local machines without much help. Being able to go beyond the basics quickly led to some of the most engaging workshop experiences you can have. 

Conversations with the community

Long-time readers of our Conference recaps might be expecting a summary of three or four select sessions. Instead, I want to share some of what I learned from the community in my conversations about non-human identity governance and secrets security. Given the average attendee of CodeMash was somewhere mid-career, with a lot of in-depth knowledge and some amazing stories, I took advantage of our time together to learn more about what they thought about these topics. I am happy to share what I heard. 

What do you mean by NHI?

When I asked folks, "What do you think about NHI governance?" most people were not sure what I meant by the acronym "NHI." In our world of jargon acronym soup, especially in the security relm, it is easy to forget that not everyone uses, or is used to, the same specific language, even though the root idea is something everyone is familiar with. 

For example, with some attendees, as soon as I said the whole term "Non-Human Identity," they knew exactly what I meant. Some other folks were more familiar with the term "machine identity" or "workload identity" and could talk about that with confidence. Once we were past the basic acronym confusion, roughly half of the people I talked to said, "You mean, like service accounts?" Yes, exactly like that, but so much more.

When I was using NHI, I meant any entity that is not a person that needs to authenticate to another non-human entity to get some work done. It was a good reminder never to assume you are clearly communicating when using any acronym, especially when an idea and its naming structure is still emerging in our space. 

NHI governance is not a new idea, but it is also not common at scale

Given this is a developer-forward community, the idea of making workloads, machines, or services talk to one another as part of our application is not new, nor is it unusual to talk about. However, thinking about how we create, observe, rotate the secrets of, and eventually decommission these NHIs is not something most people have given much thought to. 

Some of the folks who have thought about this have scoped it to their specific project or application. It was rare to meet anyone thinking about this issue at the scale of the full enterprise, though a few people at larger organizations said they knew of ongoing projects in this area.

While most people I talked to thought their org should have an accounting of the NHIs throughout the environments, most people thought it was likely someone else's worry, not something they were particularly concerned about. It reminded me of the tragedy of the commons, where we all falsely assume someone is going to take care of it. Hopefully, we, as an industry, can change this mindset.

No clear consensus on who should own Non-Human Identities

As my conversations progressed beyond what NHI and NHI governance mean, the next topic was who should own these NHIs. The most common clarifying question I got when I brought this up was, "What do you mean by own?" When we think of the term ownership, at least from a security or operational perspective, we most often mean the person who owns the responsibility when an NHI needs to have its credentials rotated. Who is on the hook for the risk if and when the secret gets leaked or used by an attacker? 

Once we were clear on what I was asking, here is what the people I talked to at CodeMash first answer was when asked about NHI ownership: 

• 20% thought Developers who add them to the application should own NHIs
• 12% thought DevOps or the Platform Team should own NHIs
• 10% thought CISO/SecOps are ultimately responsible for NHIs
• 6% thought Other/CxO, including IAM leads, should own NHIs
• 52% said "I don't know" or had no answer

As the NHI governance space continues to emerge and evolve, we will see a shift toward a more common agreement on who truly owns NHIs, but it is clear for now that this is a wide-open area where much education is needed.

Vault sprawl is common, but talking about it is not (yet)

In my talk on "Secrets Security End-to-End," I brought up the issue of vault sprawl, the phenomenon where an enterprise uses multiple secrets management platforms in an uncoordinated way. Using an enterprise vault system like CyberArk's Conjur, Vault by HashiCorp, or AWS Secrets Manager is always recommended best practice, but the larger the enterprise, the more likely they are to have introduced multiple solutions without any policies or procedures to prevent redundancy and related issues like rotating all instances of a secret. We highlighted this issue in our 2024 Voice of the Practioner report, where we found out that organizations maintain an average of six distinct secrets manager instances.

After my talk, I was surprised by a few conversations where people said they knew they had many more than six vault systems in place, but my talk was the first time someone had publicly said anything about that being less than ideal. One attendee said he knew, from the number of mergers and acquisitions they had gone through, that each application they now had to manage security for brought with it its own secret management tools. This means security has to deal with a different platform or method for almost every application their company runs. This is a clear issue for the enterprise to deal with if we are serious about taking on NHI governance and ending secrets sprawl.

At GitGuardian, as we are building our NHI security offering, we are coming to understand that while having secrets live outside the vault is very bad, there is a world where secrets living in multiple vaults can also cause a lot of issues. We are actively helping customers map the location and other relevant data points about all their mission-critical secrets, no matter where they reside. 

CodeMash is all about making human connections

While it might sound a little unusual to go to Northwest Ohio in the coldest part of winter, there was a warmth that came from being around so many like-minded people in the community that felt like walking on sunshine. While everyone, including newbies and students, was certainly welcome, I left thinking that being in a place where the average attendee was a little further on their career path led to some deeper conversations about the current state of tech and some very insightful conversations about what is coming next. There really was a sense of excitement about learning in the air. I was glad to be able to share my knowledge and hear what my peers had to say.

If joining your colleagues and peers for an event sounds appealing to you, you don't need to wait until next January. There is likely a community event, meetup, or conference somewhere near you soon and we encourage you to go join their conversation. You may even see GitGuardian there.