DevOpsDays Birmingham AL 2024: Guardrails, Immutable Infrastructures, and Community

You might not think of Birmingham, Alabama, as a large city, but if you consider the whole metropolitan area, there are over 1 million residents. At one point in the early 1900s, it was home to the four tallest buildings in the state at an intersection dubbed the "Heaviest Corner on Earth." Most of the original wealth that built Birmingham was from the iron and steel industry, as is celebrated by the Vulcan statue, the largest cast iron statue in the world. Steel has given way to the healthcare industry and a burgeoning tech community these days. It made for a great backdrop as DevOps practitioners got together to celebrate the history of what got us here and talk about the challenges ahead at DevOpsDays Birmingham, AL 2024.

Around 100 community members came together for three days of workshops and presentations from 18 speakers. Two of the days featured Open Spaces, which are self-organized conversations on various topics brought up by the attendees. They are very informal but are some of the best learning opportunities at events, as the people who participate are very passionate about that particular topic.

Here are just a few highlights from this great event.

Vault can do more than you might think

On the first day of DevOpsDays Birmingham, Michael Kosi, Sr. Developer Advocate at HashiCorp, led us through a very informative hands-on workshop on zero trust and identity-based security using Vault by HashiCorp. It started with the most common use case for Vault: secrets management, which we love at GitGuardian. The second half of the workshop took us beyond the basics and gave us some experience working with Vault for Encryption as a Service.

Data is safest when encrypted at rest and in transit, just like our machine identity credentials. This prevents anyone without the decryption key from reading or using it. Developers are often asked to make sure they encrypt personal identifying information, like credit card or social security numbers, while also being told by the industry to never roll their own encryption. Vault users have a simple solution: use the same technology you trust for secrets to handle sensitive data.  

Vault lets you store data securely and return a referenceable public key, allowing for quick retrieval at runtime. This approach looks funky, though, as the key is a long and random-looking string. This can sometimes cause issues with other tools, especially those trying to normalize data. That is why they also offer a Transform Secrets Engine, which can preserve the data's original format. For example, when you use it to encrypt a real credit card number, what is returned is a fake credit card number that is merely a reference to the original data, which is completely useless to an attacker without Vault access.

Hardening edge computing security

In one of the more technical sessions, Mauro Morales, Open Source Developer at Spectro Cloud, shared "Creating Immutable Infrastructures with Kairos." Kairos is an open-source operating system (OS) lifecycle management project. While it is not an OS itself, it streamlines and secures the deployment and operation of Linux in edge, cloud, and bare metal computing.

Running an operating system as "immutable" means the system is read-only. Once the OS has been installed, the system files and directories cannot be modified. Any data or configuration changes made on the machine vanish on reboot. Similar to GitOps, the only way to make a permanent change is to create a new instance from an updated source.  

Mauro explained that at the heart of Kairos' security approach is a "trusted boot" process. This combines full disk encryption (FDE), secure booting, and measured booting. Secure booting means that the code was verified to be as expected when starting up. Measured booting means that the OS is reverified at multiple critical milestones in the startup process, checking to see if anything has changed from the first verification. Bring trusted Linux distributions, and Kairos will ensure they are deployed safely.

Guardrails help keep us all safe

In his session "A successful platform is an opinionated platform," Ben Goodman, Senior Site Reliability Engineer at Rokt, explained some lessons from the retail world and how they translate to better security practices. When you look at some of the most successful retailers, such as Costco, Apple, and Trader Joe's, one of the things they have in common is that they all carry far fewer items than their competitors. For example, an average Trader Joe's carries around 4,000 items, while Walmart Super Centers carry over 100,000. 

Ben argues that these companies are successful because of this "limited choice". Customers only need to choose between two versions of a product rather than navigating and selecting from dozens of competing options. This decision paralysis is the same dread that DevOps professionals feel when they are faced with too many options and platforms. They want to deploy applications to meet customer needs, not configure services endlessly, and always need to hope they picked the best route.

Ben advocated for embracing opinionated guardrails for dev and DevOps teams. Making the easiest path the safest path helps everyone and lets developers focus on what matters to customers: their new feature or application. It also stops the 'black market of copy and paste config,' where people borrow config from one running application for a new one, even if the security settings are outdated or misconfigured. He ended by telling us all to "Have opinions! People need them."

Compliance does not equal security

David Hawthorne, Director of Cloud Engineering at O3 Solutions, presented "Delivering Value through DevSecOps: A Post-Incident Review. " He began by telling us that "security is a kitchen table issue." It affects us as people. He said we have been building "Hello World" and trying to secure it only after it is in production. Instead, we need to build a "Hello Secure World" from the start.

While security compliance frameworks like Service Organization Control Type 2 (SOC2) were intended to help us ensure baseline security was implemented in a provable, repeatable way, they have been interpreted as a set of checkboxes that prove an organization is secure. We all want an 'easy button,' especially in a vast and rapidly evolving field such as cybersecurity. The temptation is to focus only on the metrics outlined in the framework and call security solved.  

David thinks we can get away from being seen as 'cops' and 'department of no' and start acting like lifeguards. Lifeguards teach you to swim and carry you back to shore when things go wrong. Compliance is much easier to achieve when you explain that the "auditors are going to ask for these data points" versus simply telling someone, "they did it wrong." He also stressed that internal communication was key to this approach. Any tools you are trying to adopt need wider buy-in before you buy the tooling, which is easier to get if you understand what other teams need.  

Security and community in the summertime

One of the best parts of any DevOpsDays event is the community networking and conversations after the official schedule ends. These are where some of the more philosophical and technical discussions take place. For example, your author had many discussions about how to approach Infrastructure as Code security, which was also the topic I spoke about at the event.

DevOps practitioners are, at the same time, both very serious folks keeping our applications safely deployed and a fun-loving bunch. The event itself took place right next door to Birmingham Barons stadium. The team had both the legend Willie Mays and professional basketball player Michael Jordon on the roster in different eras. For the few who could make it out, we had a fantastic time reflecting on all we learned while celebrating Birmingham and DevOps together.