How We Built a Supply Chain Security Watchtower: Meet SaaS-Sentinel

TL;DR: we have built SaaS-sentinel, the first (as far as we know!) SaaS watchtower. The platform allows you to be notified when your favorite tool may be under attack, helping you be on the lookout for ongoing supply chain attacks. Scroll down to the third part to understand the objective and how it works. Or read below to learn the full story!

The Story

In July 2022, GitGuardian announced ggcanary, an open-source project designed to help organizations detect compromised developer and DevOps environments. The goal was to democratize the fields of intrusion detection and deception techniques, making it easier for organizations to detect potential threats with high precision and low entrance cost.

The ggcanary project uses honeytokens, which are secrets left in the infrastructure to tempt attackers. When an attacker uses a honeytoken, it sends an alert, enabling organizations to detect potential breaches before they cause significant damage.

🍯

What are honeytokens? Honeytokens are secrets (like AWS API keys or other credentials) that are left in your infrastructure to tempt attackers to try and exploit them. Once an attacker uses a honeytoken, it sends an alert, and this lets you know you have an intruder in your systems. You can find a much more detailed description in this blog post.

The concept behind this mechanism is that when a system is breached, hackers typically search for easy targets to move laterally, escalate privileges, or steal data. In this context, AWS secrets are an ideal target for scanning as they have a recognizable pattern that starts with "AKIA..." or "ASIA..." and often contain useful information for the attacker. Therefore, they are a prime target for attackers to search for and exploit during a breach.

Here is a video tutorial if you want to get your feet wet with ggcanary:

GitGuardian has been working in the secrets detection space for years and has noticed two recurring facts about data breaches:

secrets are pervasive in codebases, making them easily accessible when source code is leaked (see how we found 6,695 secrets hard-coded inside Samsung’s source code after it was leaked, for example).
intruders always use the discovery of secrets in their reconnaissance phase (take, for example, the attack on Uber in September 2022)

Therefore, the idea of decoy credentials looked very attractive as a way for defenders to turn the tables on attackers: by spreading out up to 5,000 AWS secrets in various locations, such as source code management, CI/CD pipelines, developer workstations, artifact registries, or other places, defenders can be alerted when a perimeter is breached in near real-time.

💡

We have now fully integrated the honeytoken technology into the GitGuardian platform! You can now benefit from zero-configuration, easy-to-roll-out, and scalable honeytokens to protect your assets, all from a centralized dashboard. Learn more about the new Honeytoken module here.

At this point, we didn't know that this idea would prove to be even more relevant to what was to come!

What We Didn’t See Coming: LastPass and CircleCI Fall for Supply Chain Attacks

Two incidents in early 2023 sparked the idea for SaaS-Sentinel.

First, we learned that LastPass had been targeted by a threat actor which exploited a third-party vulnerability to compromise a software engineer’s workstation and access a cloud environment where encryption keys, along with other data such as API keys and customer data, were stored.

💡

Did you know that, on average, it takes 327 days to identify a data breach?

Then, we learned that CircleCI had also, unfortunately, fallen victim to a supply chain attack: in a similar scenario, an attacker leveraged malware deployed to a CircleCI engineer’s laptop to steal a valid, 2FA-backed SSO session and, from there, could escalate privileges to the company production infrastructure. It was later confirmed that the actor was able to exfiltrate customers’ environment variables, tokens, and keys.

In both cases, the customers of these two companies experienced a nightmare scenario: they had to work on rotating potentially sensitive data stored in the two SaaS services while investigating the damage and mitigating the consequences of the incident.

In the last story, something interesting happened. A security engineer, Daniel Hückmann, tweeted about investigating the unauthorized access to his company’s CircleCI account thanks to a canary token he had planted in the platform. The decoy AWS key had been triggered by the attacker, and Daniel had been notified before anyone else, including CircleCI itself.

The Birth of SaaS-Sentinel

SaaS-Sentinel was born from this idea: what if we planted honeytokens in the most used SaaS providers to alert people when a tripwire gets triggered, suggesting a possible breach?

In a world where third-party compromises and supply chain threats are becoming more and more common, leveraging a high-fidelity signal to win a few hours over threat actors sounded like a no-brainer.
With ggcanary, we already had the open-source technology, we just had to build a unique monitoring service!

How does it work?

SaaS Sentinel is based on the honeytoken technology: GitGuardian engineers have planted multiple honeytokens in a number of SaaS tools, such as source code management platforms (GitHub, GitLab, Bitbucket), continuous integration services (GitLab CI/CD, GitHub Actions, etc.), password managers (LastPass, Bitwarden), and of course, the GitGuardian platform itself!

💡

By the way, we are looking to extend our surveillance to many more SaaS tools. You can vote for the next service provider you would like to see monitored in the future and even suggest a service we haven’t thought of.

When one of the honeytokens is triggered, subscribers to the (free) alerting service will receive a notification. However, to minimize the occurrence of false alerts and enhance the reliability of the signal, we have implemented a series of validation steps before sending out alerts to the public. The alerting flow can be better understood through the following diagram:

There are two crucial points to note:

If a honeytoken is triggered, the alert flow will first contact the affected service provider. We will then wait a minimum of 4 hours for a response before changing the service status indicator on the front page and sending an email notification to subscribers (manual process).
Despite the fact that most service providers have implemented vulnerability disclosure programs, our flow intentionally circumvents this mechanism. This is because our primary objective in the event of a breach is to promptly notify downstream users and minimize the potential impact.

💡

We plan to provide information about the service providers’ vulnerability disclosure programs directly on the platform in a future release.

Summary

The idea for SaaS-Sentinel grew from GitGuardian's ggcanary project, which uses honeytokens to detect intrusions. The recent supply chain attacks on LastPass and CircleCI revealed the value of providing a monitoring platform to alert users as quickly as possible in the event of a compromise.

SaaS-Sentinel is a free monitoring platform that aims at helping organizations stay on top of their supply chain risks and reduce the Mean-Time-to-Detect as much as possible. It is GitGuardian’s contribution to democratizing the honeytoken technology as a way to turn the tables on attackers.