How We Used to Handle Security

A few years ago, I was working on a completely new project for a Fortune 500 corporation, trying to bring a brand new cloud-based web service to life simultaneously in 4 different countries in the EMEA region, which would later serve millions of users.

It took me and my team two months to handle everything: cloud infrastructure as code, state-of-the-art CI/CD workflows, containerized microservices in multiple environments, frontend distributed to CDN, and tests passing in the staging environment. We were so prepared that we could go live immediately with just one extra click of a button. And we still had a whole month before the planned release date.

I know, things looked pretty good for us; until they didn't: because it was precisely at that moment a "security guy" stepped in out of nowhere and caused us two whole weeks.

Of course, the security guy. I knew vaguely that they were from the same organization but maybe a different operational unit. I also had no idea that they were involved in this project before they showed up. But I could make a good guess: writing security reports and conducting security reviews, of course. What else could it be?

After finishing those reports and reviews, I was optimistic: "We still have plenty of time," I told myself. But it wasn't long before my thought was rendered untrue by another unexpected development: an external QA team jumped in and started security tests. Oh, by the way, to make matters worse, the security tests were manual.

It was two crazy weeks of fixing and testing and rinsing and repeating. The launch was delayed, and even months after the big-bang release, the whole team was still miserable: busy on-call, fixing issues, etc.

Later, I would continue to see many other projects like this one, and I am sure you also have similar experiences. This is actually how we used to do security. Everything is smooth until it isn't because we traditionally tend to handle the security stuff at the end of the development lifecycle, which adds cost and time to fix those discovered security issues and causes delays.

Over the years, software development has evolved to agile and automatic, but how we handle security hasn't changed much: security isn't tackled until the last minute. I keep asking myself: what could've been done differently?

Understanding DevSecOps and Security as Code

DevSecOps: Shift Security to the Left of the SDLC

Based on the experience of the project above (and many other projects), we can easily conclude why the traditional way of handling security doesn't always work:

• Although security is an essential aspect of software, we put that aspect at the very end of the software development lifecycle (SDLC). When we only start handling the critical aspect at the very end, it's likely to cause delays because it might cause extra unexpected changes and even rework.
• Since we tend to do security only once (at least we hope so), in the end, we usually wouldn't bother automating our security tests.

To make security great again, we make two intuitive proposals for the above problems:

• Shift left: why does security work at the end of the project, risking delays and rework? To change this, we want to integrate security work into every stage of the SDLC: shifting from the end (right side) to the left (beginning of the SDLC, i.e., planning, developing, etc.), so that we can discover potential issues earlier when it's a lot easier and takes much less effort to fix or even rework.
• Automation: why do we do security work manually, which is time-consuming, error-prone, and hard to repeat? Automation comes to the rescue. Instead of manually defining policies and security test cases, we take a code-based approach that can be automated and repeated easily.

If we combine the "shift left" part and the "automation" part, bam, we get DevSecOps: a practice of integrating security at every stage, throughout the SDLC to accelerate delivery via automation, collaboration, fast feedback, and incremental, iterative improvements.

What is Security as Code (SaC)?

Shifting security to the left is more of a change in the mindset; what's more important is the automation, because it's the driving force and the key to achieving a better security model: without proper automation, it's difficult, if not impossible at all, to add security checks and tests at every stage of the SDLC without introducing unnecessary costs or delays. And this is the idea of Security as Code:

Security as Code (SaC) is the practice of building and integrating security into tools and workflows by identifying places where security checks, tests, and gates may be included.

For those tests and checks to run automatically on every code commit, we should define security policies, tests, and scans as code in the pipelines. Hence, security "as code".

From the definition, we can see that Security as Code is part of DevSecOps, or rather, it's how we can achieve DevSecOps. The key differences between Security as Code/DevSecOps and the traditional way of handling security are shifting left and automation: we try to define security in the early stages of SDLC and tackle it in every stage automatically.

The Importance and Benefits of Security as Code (SaC)

The biggest benefit of Security as Code, of course, is that it's accelerating the SDLC. How so? I'll talk about it from three different standpoints:

First of all, efficiency-boosting: security requirements are defined early at the beginning of a project when shifting left, which means there won't be major rework in the late stage of the project with clearly defined requirements in the first place, and there won't be a dedicated security stage before the release. With automated tests, developers can make sure every single incremental code commit is secure.

Secondly, codified security allows repeatability, reusability, and consistency. Development velocity is increased by shorter release cycles without manual security tests; security components can be reused at other places and even in other projects; changes to security requirements can be adopted comprehensively in a "change once, apply everywhere" manner without repeated and error-prone manual labor.

Last, but not least, it saves a lot of time, resources, and even money because, with automated checks, potential vulnerabilities in the development and deployment process will be caught early on in the SDLC when remediating the issues has a much smaller footprint, cost- and labor-wise.

To learn more about how adding security into DevSecOps accelerates the SDLC in every stage read this blog here and here.

Key Components of Security as Code

The components of Security as Code for application development are automated security tests, automated security scans, automated security policies, and IaC security.

1. Automated security tests: automate complex and time-consuming manual tests (and even penetration tests) via automation tools and custom scripts, making sure they can be reused across different environments and projects.
2. Automated security scans: we can integrate security scans CI/CD pipelines so that they can be triggered automatically and can be reused across different environments and projects. We can do all kinds of scans and analyses here, for example, static code scans, dynamic analyses, and vulnerability scans against known vulnerabilities.
3. Automated security policies: we can define different policies as code using a variety of tools, and integrate the policy checks with our pipelines. For example, we can define access control in RBAC policies for different tools; we can enforce policies in microservices, Kubernetes, and even CI/CD pipelines (for example, with the Open Policy Agent). To know more about Policy as Code and Open Policy Agent, read this blog here.
4. IaC security: in modern times, we often define our infrastructure (especially cloud-based) as code (IaC) and deploy it automatically. We can use IaC to ensure the same security configs and best practices are applied across all environments, and we can use Security as Code measures to make sure the infrastructure code itself is secure. To do so, we integrate security tests and checks within the IaC pipeline, as with the ggshield security scanner for your Terraform code.

Best Practices for Security as Code

With the critical components of Security as Code sorted out, let's move on to a few best practices to follow.

Security-First Mindset for Security as Code/DevSecOps

First of all, since Security as Code and DevSecOps are all about shifting left, which is not only a change of how and when we do things, but more importantly, a change of the mindset, the very first best practice for Security as Code and DewvSecOps is to build (or rather, transition into) a security-first mindset.

At Amazon, there is a famous saying, which is "Security is job zero". Why do we say that? Since it's important, if you only start dealing with it in the end, there will be consequences. Similar to writing tests, trying to fix issues or even rework components because of security issues found at the end of a project's development lifecycle can be orders of magnitude harder compared to when the code is still fresh, the risk just introduced, and no other components relying on it yet.

Because of its importance and close relationship with other moving parts, we want to shift security to the left, and the way to achieve that is by transitioning into a security-first mindset.

If you want to know more about DevSecOps and why "adding" security into your SDLC doesn't slow down, but rather speeds up things, I've got another blog here detailing exactly that.

Code Reviews + Automated Scanning

Security as Code is all about automation, so it makes sense to start writing those automated tests as early as possible, so that they can be used at the very beginning of the SDLC, acting as checks and gates, accelerating the development process. For example, we can automate SAST/DAST (static application security testing and dynamic application security testing) with well-known tools (for example, SonarQube, Synopsys, etc.) into our CI/CD pipelines so that everything runs automatically on new commits.

One thing worth pointing out is that SAST + DAST isn't enough: while static and dynamic application tests are the cornerstones of security, there are blind spots. For example, one hard-coded secret in the code is more than enough to compromise the entire system (to know more about this topic, read this blog here).

Two approaches are recommended as complements to the automated security tests. First of all, regular code reviews help. It's always nice to have another person's input on the code change because the four-eyes principle can be useful. For more tips on conducting secure code reviews, read this blog here.

However, code reviews can only be so helpful to a certain extent, because, first of all, humans still tend to miss mistakes, and second of all, during code reviews, we mainly focus on the diffs rather than what's already in the code base.

As a complement to code reviews, having security scanning and detection in place also helps. To know more about secrets in source code and how secrets sprawl works, see this blog here.

Continuous Monitoring, Feedback Loops, Knowledge Sharing

Having automated security policies as checks can only help so much if the automated security policies themselves aren't of high quality, or worse, the results can't reach the team.

Thus, creating a feedback loop to continuously deliver the results to the developers and monitor the checks is critical, too. It's even better if the monitoring can create logs automatically and display the result in a dashboard to make sure no security risk is found, no sensitive data or secret is shared, and developers can find breaches early so that they can remediate issues early.

Knowledge sharing and continuous learning can also be helpful, allowing developers to learn best practices during the coding process. Here is a cheat sheet for developers to do security in different stages of the SDLC in the cloud-native era.

Security as Code: What You Should Keep in Mind

Besides the best practices above, there are a few other considerations and challenges when putting Security as Code into action:

First of all, we need to balance speed and security when implementing Security as Code/DevSecOps. Yes, I know, earlier that we mentioned how security is job zero and how doing DevSecOps actually speeds things up, but still, implementing Security as Code in the early stages of the SDLC still costs some time upfront, and this is one of the balances we need to consider carefully, to which, unfortunately there is no one-size-fits-all answer. In general, for big and long-lasting projects, paying those upfront costs will most likely be very beneficial in the long run, but it could be worth a second thought if it's only a one-sprint or even one-day minor task with relatively clear changes and confined attack surface.

Secondly, to effectively adopt Security as Code, the skills and gaps in the team need to be identified, and continuous knowledge sharing and learning are required. This can be challenging in the DevOps/DevSecOps/Cloud world, where things evolve fast with new tools and even new methodologies emerging from time to time. Under this circumstance, it's critical to keep up with the pace, identify what could potentially push engineering productivity and security to another level, figure out what needs to be learned because we've only got limited time and can't learn everything, and learn those highly-prioritized skills quickly.

Last but not least, keep a close eye on newly discovered security threats and changes regarding security regulations, which are also evolving with time.

Conclusions and Next Steps

Security as Code isn't just a catchphrase; it requires continuous effort to make the best out of it: a change of mindset, continuous learning, new skills, new tooling, automation, and collaboration.

As a recap, here's a list of the key components:

• Automated security tests
• Automated security scans
• Automated security policies
• IaC security

And here's a list of the important aspects when adopting it proactively:

• Security-first, mindset change, shift left
• Automated testing/scanning with regular code reviews
• Continuous feedback and learning

At last, let's end this article with an FAQ list on Security as Code and DevSecOps.

F.A.Q.