DevOpsDays Chicago 2022 - Cloud security, hacking containers, community, and much more...

I was thrilled to participate in DevOpsDays Chicago 2022 as my first in-person event as a Developer Advocate at GitGuardian. I am very excited to tell you all about this awesome event that happened from September 21 to 22 at the University of Illinois, Chicago’s The Isadore & Sadie Dorin Forum. Over 350 attendees, vendors, and volunteers gathered to take stock of the state of DevOps and share our knowledge and love of building in the cloud.

Celebrating 8 Years of DevOpsDays Chicago

Chicago is a city in the middle of everything, and not just geographically. It is also the home of many corporations and a vibrant technology community. DevOpsDays Chicago brought together developers, operations teams, SREs, and InfoSec leads to sharing their knowledge and experiences with the goal of helping us all adopt better DevOps best practices and be more secure.

Due to the pandemic, DevOpsDays Chicago went to a single-day virtual event in 2020 and was canceled in 2021. Absence truly does make the heart grow fonder, as every attendee I talked to mentioned how glad they were that the event was back. The community of DevOps pros was eager to share stories, and commiserate about the challenges modern cloud native software development brings and newly learned best practices.

For folks who were not able to make it in person, much of the event was also streamed for free. DevOps professionals around the world were able to tune in live for the single track of 30-minute talks, the shorter, 5-minute Ignite talks, as well as the afternoon workshops, including mine.

Open Spaces Make DevOpsDays A Unique Event

There was one important part of the event that was not shared over the live stream: the Open Spaces portions of DevOpsDays. This is one of the biggest reasons people I talked to cited for attending the event in person. Open Spaces is a way to run an ‘unconference’, where the agenda is set by the conference attendees and sessions run as small, interactive group discussions in breakout rooms.

It starts with attendees volunteering topics of interest. Some of the topics this year included: Achieving Awesome Observability, DevOps Feedback Loops, Book Recommendations, Chaos Engineering, GitOps In Practice, and even Volunteering and Running DevOpsDays.

After collecting the suggested topics the organizers assign breakout rooms and fit everything within allotted time slots. While on the surface it might sound a little chaotic to have the conference attendees self-organize into roundtable discussions, it actually is quite a smooth process thanks to some simple rules:

1. The people who show up are the right people
2. Whatever happens, happens
3. Whenever it starts is the right time
4. When it's over, it's over
5. Law of Mobility - If you want to move to a different open space, then move
6. Bring your best self

I think this is a really wonderful way to let people share their knowledge and experiences. I really learned a lot during the Open Spaces. I hope more events adopt this unconference approach as it is a very empowering experience for all involved.

Container Security Conversations At DevopsDays

The sessions covered a wide range of very interesting topics, from Lesley Cordero’s Effective Observability in a Microservices Architecture to Abby Allen’s Parenting makes me a better Product Manager. All were great and well delivered by the speakers. One major thread that ran through a lot of the talks and in many of my conversations was container security. While every talk is worth watching, I will highlight a few talks that stood out as articulating the underlying security discussions at the event.

Developers Securing Their Clouds

During his talk “Stories from the trenches – democratizing security with modern development” CTO at oak9, Aakash Shah, asked the attendees who held the title of a full-time security engineer. Not one hand was raised. He then asked who was a DevOps engineer and almost all the hands went up. The room was full of DevOps professionals very concerned about security!

In his talk Aakash’s talk, he discussed a AAA model for how security for devs must be laid out. While he goes into much more detail in his talk, those A’s stand for:

• Accessible - Translate security best practices into user stories, avoid jargon
• Actionable - Fit security into sprints, instead of 40+ page requirement docs
• Applicable - Understand business use case, base action plans on reality

It can be overwhelming to think of all the possible ways to approach security, but keeping those AAA’s in mind when discussing and implementing security solutions can help everyone work smarter and safer. Keeping user stories and business use cases front and center of the discussion is critical, especially while containers, and tools like Kubernetes. drive the complexity of modern cloud-native architectures up exponentially.

He also warned that it is tempting to “just dump yet another tool” to deal with security issues. While tools absolutely are necessary, he advocated for a more developer productivity mindset, where continuous education and better collaboration between security and development teams are more important than any single piece of technology.

Learning How To Hack Containers

Eric Smalling, Developer Advocate at Snyk gave one of the most chilling and eye-opening workshops I have ever attended “Hands-on hacking containers and ways to prevent it.” Rather than just a lecture on best practices, Eric walked us through how a hacker might systematically look through a container, escalate privileges and own namespaces, and potentially a whole cluster!

He safely did this against a demo environment they set up using Synk Lab’s Kubernetes Goof repository. The "About" section of the repo reads “Kubernetes Stranger Danger.” It is a free and open source tool you can use to learn how Kubernetes clusters can be attacked and show why some best practices need to be enforced.

One of the larger dangers he pointed out was how often people forget to set correct permissions for namespaces. It is pretty easy to just give a namespace full access rights while also treating them like private secure places to store scripts that might contain secrets. If they are accessed this potentially spells disaster for the whole cluster and application.

Another alarming danger was container privilege escalation. Once a bad actor realizes they can, they will own as much as they can, which might be ‘the world’ for your application. He stressed if there was one big takeaway from his talk it would be “don’t allow privilege escalation!”

Integrating Security Into The DevOps Culture

Building DevOps is not something you can buy off the shelf, it is about adopting the right culture and methodologies before you invest in tech. Similarly, if you approach security as just an add-on and do not address the culture or methodology, you can not ever fully deliver DevSecOps, according to Senior Developer Relations Engineer at New Relic, Daniel Kim during his talk “Building security into the DevOps pipeline at scale”

We can do this from the start by defining threat models in the planning phase, and bringing in the security team while the application still just exists on a whiteboard. During the coding phase, developer errors can be addressed by implementing the right tooling. His example was hardcoding secrets, which can be prevented by using Git Hooks (which we can help you achieve).

Catching issues early will help make sure your build and deployments go off as planned. Adding security testing through the build phase, in your CI/CD pipeline, will mean the security team won’t be seen as a blocker, needing to do full security audits per build. One of the tests Daniel discussed in depth was Software Composition Analysis to ensure dependency libraries do not present threats.

Daniel also addressed a large area of concern for developers and security; the fast evolution of tools and their accompanying threats. While containers and Kubernetes allow us to build amazing things at scale, we must recognize legacy security tools and approaches, like firewalls often are not keeping up with reality. This is where baking security into your DevOps culture comes in; the more the security teams are part of the development planning, the easier it is to identify potential threats and apply the best tools to keep everyone safe.

Learning From Each Other

I had a blast at DevOpsDays Chicago 2022. It was great to teach people some advanced Git with my workshop “Git - Beyond Just Committing,” but I think the greatest education that happened throughout the event came from conversations in the hallway track and during Open Spaces. I know I walked away with a new appreciation for topics like GitOps automation and container egress testing.

Congratulations to the awesome organizing team and volunteers for making DevOpsDays 2022 happen in person. It was a lot of work and it is very appreciated by the community!

While we might not all agree on particular tools or what we even mean by Chicago Style Pizza, we all came together to agree that the future is DevOps and that security is something we all need to keep discussing. I look forward to continuing that conversation at next year's DevOpsDays Chicago and other events, perhaps near you sooner!