Over the past few years, we have seen a massive increase in software supply chain attacks. What is a supply chain attack? This is a type of cyber security attack where adversaries slip malicious code or components into a trusted piece of software or hardware. The goal of such an attack is the ability to infiltrate organizations that are down the chain of the affected component. The focus on supply chain attacks in recent history has been on compromised hardware, but with a number of high-profile incidents, we have seen the conversation shift towards software. This article will look at software-related supply chain attacks specifically and use recent examples to show what companies can do to protect themselves from such attacks.
The software supply chain
To understand this type of attack we need to be familiar with the concept that applications today are not monolithic pieces of software. They are built up of hundreds of building blocks, open-source libraries and packages, SaaS tools, deployment systems, cloud infrastructure and so on. This gets even more complicated though because each of these building blocks are also built up from a number of different building blocks. If you think about this in layers, a piece of software can be hundreds of layers deep with each component having dependencies until you get all the way down to the CPU instruction set. There is a phrase you will often hear developers use, “turtles all the way down” . This is an expression describing the problem of infinite regress, referring to the turtles standing on each other's back infinitely. Translated into software, this illustrates the perception of infinitely scoped issues related to software supply chain. Some of the issues we face in the software supply chain.
“The world is getting increasingly more interconnected every day, but in security, we don’t act like its interconnected” Eric Doerr Black Hat 2019
What makes a supply chain attack so concerning is that the organizations that are impacted by it, often are powerless to prevent it. You have control over your source code, some control over the components and tools you choose to use but almost no control over the components and tools they choose for you. When you acknowledge that being 100% secure or ‘unhackable’ is impossible, then even when you take incredible care with the elements within your software supply chain, you can never remove the risk of a supply chain attack. However, you can reduce risk and in this article, we will take a look at 6 steps you can take to limit both the risk and damage of such an attack.
Here is a an example of the multiple attack surfaces within a supply chain:
“We all rely on the supply chain being fully immunized but the reality is it's not there” - Jeff Moss Black Hat 2021
Recent examples of supply chain attacks
There are lots of different types of software Supply Chain attacks that target different components to achieve a foothold in an organization. We will look at three recent examples of different types of such attacks to get a better understanding.
SolarWinds - Attack through a trusted system with privileged access
Unfortunately, you cannot write an article on supply chain attacks without bringing up the SolarWinds attack. It has become the poster child for the dangers they represent. In the case of SolarWinds, the attack used IT infrastructure monitoring with elevated privilege. This attack is one of the worst-case scenarios where the compromised system not only was used with tens of thousands of high profile clients, such as within the military and other areas within the government but because the software had admin level access on the networks it was installed on. The target called SolarWinds Orion was compromised as early as September 2019. The first traces of code used by the attackers for testing the exploit were injected into the code in October 2019. It wasn’t until February 2020 when the malicious code, known as Sunburst, was injected into the Orion software and began to be deployed to its 18,000 users in March.
The Sunburst attack within the Orion code granted the attackers a back door into SolarWinds customers, some of the affected customers included Homeland Security, State Commerce and Treasury, FireEye, Microsoft, Intel, Cisco and Deloitte to name a few.
By targeting SolarWinds software and exploiting a security flaw (which is still debated), the attackers were able to gain access to some of the most protected organizations in the world. Many of these organizations were considered ‘unhackable’ (although any security professional worth their salt will tell you this is impossible), or at least had state-of-the-art security protections in place. But this was bypassed by exploiting part of their supply chain that had privileged access they were able to penetrate into these unhackable organizations and remain undetected for a significant amount of time.
Codecov - Attack through software deployment and testing tools
Next, we are going to look at an attack in software deployment tools, specifically the CI/CD environment. If you are unfamiliar with the CI/CD (Continuous Integration / Continuous Deployment) pipeline, it is a process by which software can be automatically tested before it is automatically deployed. This has become a fundamental part of the software supply chain and organizations leverage a number of different tools to test their software, this generally happens each time a new code is pushed to the production branch (among others) of a code repository.
One such tool that was again used by tens of thousands of customers was Codecov, a code coverage tool that lets users know how much of your application was being tested within your CI environment. Because this tool sat in the CI environment, it had access to the secrets (or credentials) the application used when it was being tested. For instance, it needed access to databases, third-party services and code repositories so the application could be tested.
In the example of Codecov, attackers were able to first gain access to Codecov’s private code repository and inject malicious code into their source code. This malicious code was only one line and did something very simple: It took the secrets in the CI environment (such as the git tokens) and sent them to the attacker’s remote server. Or in simpler terms, it took the credentials being used to build and test an application and gave them to the attacker. This allowed the attacker to access the private code repositories of many Codecov customers including Twilio Rapid7, HashiCorp and Monday.com to name a few. Private code repositories have become high-value targets for attackers because they often contain sensitive information including secrets. In some cases, with the right access attackers can even inject malicious code using these tokens.
Read a full break down of the Codecov attack here
EventStream - Attack through software dependencies
In the last example of a recent supply chain attack, we will look into open-source dependencies. Nearly all modern software applications use open-source dependencies. A study by Synopsys shows that 91% of modern applications are actually open-source code. Being able to use this open-source code is incredibly helpful as it means engineers don’t need to redesign components for each application they build and instead can focus on their unique value.
If an attacker can compromise a downstream dependency then they can achieve the critical step of initial access. A good example of this is the case of EventStream. This package was a toolkit to make streams easier to implement in Node.js, it was incredibly popular and was used by millions of applications. However, this EventStream itself had dependencies, including a module called Flatmap. In this case, an attacker posed as a maintainer of FlatMap and eventually took ownership over the open-source module. The attacker then injected malicious code into the module, turning it into a weapon to target downstream users. It also gave them a back door into EventStream and ultimately the EventStream users. So the attacker was able to gain access to targets by compromising a dependency of a dependency.
How to prevent a supply chain attack
Now that we understand some different ways a supply chain attack can take place, let's look at how we can harden our supply chain and protect ourselves against attacks. The difficulty with these types of attacks is that while you can do things to secure yourself, a lot of the time you are simply a passenger.
1. Only use trusted dependencies within your application
When choosing what dependencies and modules to use within your application you must make sure you are using software that:
Are well maintained with a continued track record of updates, this makes sure that any vulnerabilities discovered will be researched and patches updated. It also reduces the risk of a rogue maintainer injecting malicious code.
Is not misspelled! There is a type of attack called type-squatting where malicious actors will miss-spell a popular open-source dependency, usually, the module will perform the same job as the original, so it does not create suspicion, but with the added bonus of malicious code as well.
2. Scanning open-source software for known vulnerabilities
Open Source Scanning or OSS software like Snyk or WhiteSource are fundamental in hardening the supply chain. This software will search through the dependencies that your package uses and compare them to their extensive databases of vulnerable packages and versions to see if your application has any known vulnerabilities. If a dependency has a known vulnerability, especially one categorized as critical, then these packages can even automatically update this to the latest safe version in many cases. If however there is no update available, then it is extremely important that you use a different module or package as it is a sign that this software is no longer maintained.
3. Patching intelligently (regularly not instantly)
If you have ever read security articles, usually within that you will see a note telling you to update or patch your systems regularly. This of course makes sense, you want the latest version so that you can be sure any discovered vulnerabilities are no longer threats. While this is true, this does not mean you should update immediately. This may sound counterintuitive but if we take a look at all the examples we have in this article, the one similarity they have is that had the users not updated immediately, they wouldn’t have been affected. Now before you yell at your screen, I am not advocating that we don’t patch, or patch irregularly, but we need to apply thought to this.
Liran Tal, a developer advocate from Snyk said that it takes roughly 60 days for vulnerabilities to be discovered, reported and patched.
This is consistent with the examples given in this article. So with that in mind, you need to create an intelligent patching policy where you update to the latest version immediately if security flaws have been identified in dependent systems and otherwise follow an initial waiting period.
4. Segment your network
If an attacker can gain access to your organization through a supply chain attack, they will want to move quickly into different areas of your network. Network segmentation is an effective way to limit the blast radius in case of an attack. It refers to dividing a larger network into smaller sub-networks with limited inter-connectivity between them. By controlling traffic flows between various sub-networks and by restricting attacker lateral movement, network segmentation prevents unauthorized users from accessing the organization’s data. Some common ways to segment a network include
- VLAN network segmentation: This involves creating segments in networks with VLANs or subnets using IP addresses for partitioning.
- Firewall segmentation: Firewalls inside a network segment limit the attack surface and prevent threats from spreading. Since this method requires thousands of firewall rules, it introduces considerable complexity and cost to the system.
- Segmentation with Software-defined Networking: SDN-based network segmentation supports greater automation and programmability. However, it focuses less on security visibility, and more on network policy implementation.
- Micro-segmentation: This technique uses the host workload to enforce a segmented network, and whitelist models to block all traffic except what is permitted.
5. Implement advanced authentication and least privilege
There is a concept in security called Zero Trust. It means that just because someone has authentication credentials, we still don’t trust them. Examples of this include using multifactor authentication and limiting the IP addresses that can access systems. In a supply chain attack, we want to limit the amount of trust that the attacker ultimately has, implement strong authentication using principles of zero trust that can stop an attacker in their tracks.
It is also important to make sure we use the principle of least access, this means that users and also services only have the minimum amount of access to additional data and services. Because an attacker can launch an attack from a trusted system, often even using zero trust principles we cannot detect and block them, however, we can limit the amount of information that they have access to.
6. Make sure your repositories are free from secrets
It has become a classic playbook by attackers to target code repositories and backup servers through these types of attacks. This is exactly what happened in the case of Codecov where attackers were targeting git repositories. This is because these are the perfect initial access point to find credentials for the attacker to authenticate themselves and move deeper into an organization. We have to ensure that these systems are free from sensitive information that can be used by attackers. You can read more on why secrets in git are such a big problem here.
The simple truth is that even if we take all precautions and implement best practices we cannot always prevent a supply chain attack, so we must ensure we do not leave any gifts for attackers. Using secret scanning tools like GitGuardian is essential, they will conduct a complete historical scan on the repositories including their history and continue scanning in real-time to ensure you never have any secrets in these repositories.
Get a historical scan of your repositories todayWrap up
Supply chain attacks have been increasing dramatically and the simple truth is that no one is immune from them. The distributed architecture we use to build applications gives attackers the perfect opportunity to be able to target hundreds even thousands of organizations at the same time. This shift has completely changed the economics of the attackers, they can now assign significantly more resources to targeting the supply chain compared to targeting one individual company.
While it is impossible to be without the risk of being a victim in a supply chain attack, we can implement some best practices to reduce the risk of both being a victim and potential damage caused during an attack.
