Most organizations today are software development companies. It doesn’t matter much if you are building the latest in cloud computing services or manufacturing paint, you most likely have a team of software engineers building proprietary systems and at the very least you rely heavily on commercial software to keep your operation afloat. Therefore this article doesn’t just affect the start-ups with ambitions to change the IT landscape, it affects all organizations.
The reality is that in today's world if you can control an organization's source code, you can control and disrupt multiple elements of the operation.
This year at DefCon and BlackHat—the world's largest security and hacking conferences—many talks focused on how attackers are targeting these systems. In an escalation of a trend we have seen for many years, developers are increasingly being targeted by attackers as a way to poison the software supply chain and take control of an organization.
This article is going to compile lessons and tools from briefings within BlackHat and Defcon to outline some of the key trends of how adversaries are compromising developers and poisoning the source. Specifically, we are going to look at how they can gain access to source code management (SCM) systems like git and CI/CD pipelines. To achieve this we will look at the path attackers will take to achieve this:
- Compromise a developer account
- Compromising a source code manager (SCM)
- Compromising the CI/CD pipelines
Compromising the developer
“Assume that every developer is malicious or compromised” Iain Smart
The first step in compromising development tools is to compromise a developer account. While administration accounts may be highly protected, we have seen many examples of developer accounts being much easier targets. There are a few reasons for this, they do not have sufficient organizational security, the same accounts are used for personal and professional use (especially in the case of GitHub), git repositories are often misconfigured and most of all, developer accounts are not considered critical to security by organizations. While developer accounts might not by default have permissions to manipulate production code or deploy applications, the nature of the tools they interact with allows them to interact creatively with different systems and often bypass security controls. This is why they are such a big target for attackers, let's review some recent examples of developer accounts being compromised in the wild.
- Insider threat / Compromised user
At the beginning of 2022, the Lapsus$ group was able to gain access to a number of large organizations' private source code. This included Microsoft, Nvidia, Samsung, and others. How? One method was paying insiders to grant them access to internal systems such as code repositories. There are thousands of developers in large companies, would you trust all of them not to give an attacker access? - Misconfigured git repository.
Twitch was another company whose source code was leaked publically, this time it was a misconfigured git repository that allowed remote access. The attacker discovered that the entire code repository was remotely accessible on a private network by fuzzing different endpoints. - Poor password Hygiene
Uber had an incident where attackers gained access to private source code repositories because developers were reusing the same passwords across sites and attackers were able to use data from a breach concerning a different site to gain access. - Leaked git credentials
Developers often use connection tokens and SHH keys to authenticate with systems such as git. These keys often mistakenly end up in public repositories related to personal projects. In research by GitGuardian we discovered over 60,000 access tokens to private repositories Were leaked within PUBLIC git repositories. Adversaries often search public repositories trying to gain access to such accounts.
There are more examples but the idea here is that developer accounts are not all that secure and a motivated attacker will be able to find a weak leak to gain access to your code repositories where they can then continue an attack on the source code management systems or the CI/CD pipelines.
Compromising source code management system
Once an adversary gains access to a developer account with access to the internal source code system, such as a git server, they may only have access to limited repositories with insufficient access to be able to inject malicious code or affect production assets. So now we must use our developer access to gain additional control.
Luckily for our attacker, numerous methods and tools exist to help in this. Brett Hawkins is a security researcher at IBM X-Force red and is the creator of SCM-kit. Hawkins’s job at IBM centers on adversary simulation, demonstrating the potential impact of security holes and vulnerabilities to clients. In a fascinating talk, Hawkins presented research on how you can exploit source code management systems as an attacker or red teamer.
This centered around four attack scenarios that can be used on source code management (SCM) systems: Reconnaissance, User Impersonation, Admin Escalation, and Repository Takeovers. Below in the table, you can see the types of attacks that can be conducted under each stage.
An attacker with developer access will likely execute all of these activities trying to gain elevated privileges or access sensitive assets. Each one requires a significant amount of time generally and the longer an attacker spends in a system the more suspicious activity will appear within security logging and tools. Luckily for the attacker, all common SCM tools like GitHub, GitLab, BitBucket, etc, have REST APIs you can access and manipulate which means you can automate such tasks. None of these attack scenarios focuses on vulnerabilities in the systems, it instead looks toward creative applications of the features developers have access to (and need access to).
“It’s all about abusing the features that are available within the system,” Brett Hawkins
You don’t even need to create your own automation tools as one exists already. SCM kit can be used to run through multiple attack scenarios (far beyond what is listed above) automatically from developer access. I will briefly explain some of the scenarios below.
Reconnaissance
During an early stage of the attack, the attacker is discovering any and all vulnerabilities they can exploit. This might include quick wins like scanning for hardcoded secrets or the discovery of logic errors in an application they have access to. But it is also going to involve the attacker trying to discover the structure of the SCM and all the different resources they have, and do not have, access to. This is important because with this information they can start to try and pick apart any configuration vulnerabilities in private resources where they can elevate their privileges. SCM-kit will connect with any developer account and can be used to discover all kinds of information within a system.
User Impersonation and maintaining access
Most SCM tools have the ability for admins to impersonate users, this can involve generating an impersonation token (an access token for another user). Another way to maintain access is to generate other authentication methods such as an SSH key to allow the attacker direct access. This is very important in an attack because it makes it much harder to detect if an attacker is in the system. As they control their own authentication now, even if a compromised user changes their access credentials the attacker can maintain access.
Elevating privileges to admin access
Every attacker will be trying to gain access to an admin account. This may be achieved after accessing regular SSH or user tokens at the reconnaissance stage of the exploit. To maintain presence an attacker will want to promote or create new accounts with admin access to avoid being blocked. Using the SCM kit an attacker could very quickly find admin access and promote a new user all with API access via the SCM kit tool so now even if original admin access is revoked the attacker will maintain authentication.
Repository takeovers
Restricting access to repositories for users is a way to persist an attack and reduce a company's ability to detect and prevent an attacker. With admin access, malicious users can take over and block access to these repositories directly from the SCM kit via the API. If an attacker has sole access to a repository, they easily deploy malicious applications into production environments without interruption.
Compromising the CI/CD pipeline
In addition to compromising source code management systems, another area that has become a huge target is CI/CD pipelines. CI/CD pipelines are tools used by developers to automate tasks triggered by events such as merging code. If an attacker can control the CI/CD pipeline, then they can take control of the testing and deployment integrity of an application.
In a briefing from speakers Viktor Gazdag and Iain Smart from NCC, they ran through attack scenarios they have done to compromise CI/CD pipelines from limited developer accounts.
They were able to show multiple real-life examples where they were conducting attacks from a developer account by elevating their privileges and taking advantage of common misconfigurations to perform admin-level tasks and ultimately take over the CI/CD environment. Smart described one engagement where the client granted NCC Group researchers a set of developer credentials, which lacked admin privileges, to see if they could break out of the pipeline. While they had access to just a single Bitbucket repository, they were also able to modify which dependencies were required for the pipeline to build an application.
Smart's injected a custom malicious package so when the CI/CD pipeline connected to it they were given access to a Jenkins runner. While the runner had a limited environment, they were able to move further into the system thanks to a private SSH key that was leaked inside the environment. With the key, the researchers were able to gain access to all the organization's Jenkins control nodes and obtain all secrets including credentials and API keys across multiple projects.
While this turned out to be a fairly involved method of gaining elevated privileges, there are a lot of examples that Smart gave where the result was much simpler. In one case, developers were not meant to have access to the production credentials. But Smart wrote some simple code to print out on screen the credentials used by the CI/CD pipeline, and by changing the text DEV to PROD, all of a sudden they had access to all production credentials.
Concluding notes
You may be thinking that all these examples need one thing, for the attacker to have access to developer accounts. Yes, that is true but what we can see via these DefCon and BlackHat briefings, combined with all the breaches we have witnessed this year, is that developer accounts are one of the weakest links in the supply chain for an attacker to gain access. The rapid pace of development of tools to assist this makes the life of attackers way easier to be able to start with a developer account and move their way through multiple different systems to take over systems, turn applications malicious and disrupt run times.
Preventing this is not quite as simple as stating we need MFA on developer accounts (although we do), but it requires a change in mindset that puts adversaries in the developers' shoes. Organizations need to run attack scenarios where an attacker has developer access so that you can see where too much trust is being applied.
