In September, GitGuardian Developer Advocate Makenzie Jackson (aka Mack) hosted Step Security’s Varun Sharma and Anish Kurmi for a webinar titled “A Hands-on guide to Runtime Security for CI/CD Pipelines with StepSecurity.” Hundreds and hundreds of viewers joined Mack and his guests live and hundreds more have viewed the recording on YouTube.

Here’s the video if you’d like to watch.

Start (0:00) 

Mack introduces himself, the guest presenters, and welcomes the viewers from all over the world. They also provide a link to the repository they’ll be using for the lab:

https://github.com/step-security/github-actions-goat 

5:00 

Mack explains how GitGuardian’s HoneyTokens product is part of a complete software supply chain defense strategy, then hands off to Step Security. Ashish and Varun introduce themselves and provide a quick agenda.

8:42 

What could go wrong? Ashish shows a GitHub Actions workflow, pointing out some of the steps involved in building a container and pushing it to their AWS ECR account.

“CI/CD is a high-risk environment because [it’s] a high-privileged environment.” – Ashish

He points out that CI/CD environments contain cloud credentials, create production builds, and often run open-source code that may contain its own vulnerabilities.

11:20

Introduces the CodeCov breach. A popular tool for code coverage, it was breached and hackers added malicious code. When customers used it in CI/CD during that time, their keys were exfiltrated and used to hack them. He recommends Mack’s blog that walks through the CodeCov breach and its impact.

13:28

Next, on to the Solar Winds breach, where the build server was compromised. Massive fallout.

16:16

Ashish starts discussing the joint guidance from CISA and NSA on CI/CD and NIST’s 800-204D publication on securing microservice-based systems.

Recommendations highlighted include:

  • Segment the CI/CD network
  • Filter traffic going both in and out
  • Use Endpoint Detection and Response (EDR) tools
  • Tamper proof audit logs
  • Reduce unnecessary software

18:14

“Challenges in Securing CI/CD”

  • Tools lack CI/CD context
  • Runtime solutions focus more on anomaly detection instead of baseline deviation detection.
  • CI/CD can be hosted in so many ways, many tools can’t deal with all the possible combos.

21:21

Introduces Step Security’s “Harden Runner,” which is a tool for baseline deviation detection.

https://github.com/step-security/harden-runner   

25:00

Demo of Harden-Runner, showing how they integrate into a GitHub actions setup. Shows how a call to ruby-gems gets blocked because it’s not allow-listed.

28:33

Official start to the hands-on. Instructions on how to fork the GitHub repo for the exercise.

Network Monitoring with Harden Runner (30:45)

Varun introduces us to the first part. We’ll run a CI/CD process without their Harden Runner software. The point is to show not just what you see when you do this, but what you don’t see.

32:46

Now we run the same process with Harden Runner in place with it auditing the network traffic. He re-runs the workflow with the added Harden Runner step and shows where the workflow log provides a link to see an analysis of the monitoring and some suggestions.

35:10

Over to Step Security to see the processed output and suggestions. In the npm install step, he points out that there was a call to “attacker.com” (in case something else would be too subtle).

He then walks through tracking down where that call was being made and finds it in a file. It’s tied to `npm install` and he dives into the `package.json` file, which points him to a package already in the repo. Digging into the code of that package, he finds the call.

40:00

Next lab… network filtering.

A set of allowed network endpoints are provided as part of definition. In this case, the `npm init` process fails because it couldn’t reach attacker.com

43:10

File monitoring. We skip the demo of it without Harden Runner and go straight to the version with Harden Runner, then run the action.

Digging into the logs, we find that `index.js` has been changed in the process. This is not good. The source code has been altered on the build server. Traces it to a compile.js script running as a pre-install step that’s got malware in it.

50:34

Varun shows what this would look like in a self hosted environment. These are running in Kubernetes. In this case, you can set up a Harden Runner daemon.

Q&A (54:00)

  • A question about how the network filter works.
  • Are self-hosted CI/CD platforms more secure? 
  • When the call to attacker.com was blocked, did the process fail or did the build fail? The build failed, but you can construct failure tolerance.
  • How do you perform security assessments for network services like a load balancer? 

1:01:00

Thanks everyone and we’ll see you soon. We host a webinar a month if you'd like to attend visit our events page.