The Secret's Out: How Stolen Okta Auth Tokens Led to Cloudflare Breach
What Happened?
Cloudflare’s internal Atlassian systems were breached using tokens and service accounts compromised from a previous Okta breach. The attackers gained access to the Confluence wiki, Jira database, and Bitbucket source code system. The incident illustrates the damaging domino effect of secrets sprawl and the importance of maintaining rigorous secrets security across the supply chain.
Cloudflare Security Breach Recap
1. On November 14th, Cloudflare’s self-hosted Atlassian server was breached by suspected nation-state hackers.
2. The hackers utilized one access token and three service account credentials stolen from Okta’s previous compromise. Cloudflare had failed to revoke these credentials after the Okta breach.
3. The breached systems included Cloudflare’s Confluence, Jira, and Bitbucket systems.
4. Attempts by the threat actors to hack into Cloudflare’s São Paulo data center – not yet in operation – were unsuccessful.
5. Threat actors established persistent access and attempted to gain deeper access to Cloudflare's global network.
6. Cloudflare detected the breach on November 23rd and cut off the attackers' access the following morning.
7. Remediation efforts included rotating all 5,000 production credentials, segmenting test and staging systems, extensive forensic triage, and rebooting of all the company's systems.
Cloudflare Hacked: The Long Story
On November 14th, reputedly a 'nation-state attacker' infiltrated Cloudflare's Atlassian server, compromising its Confluence wiki, Jira database, and Bitbucket source code management system. From the breach's onset, the hackers established relentless access to Cloudflare's Atlassian server using the ScriptRunner tool for Jira.Interestingly, the attackers used one access token and three service account credentials obtained during Okta's earlier breach in October 2023.
Cloudflare's failure to rotate these compromised details was a significant oversight and a primary enabling factor of the attack.
While attempts by threat actors to infiltrate Cloudflare’s data center in São Paulo were unsuccessful, extensive damages occurred.
A thorough examination of accessed wiki pages, bug database issues, and source code repositories suggested the hackers were seeking comprehensive data on the architecture, security, and management of Cloudflare's global network.
Although Cloudflare concluded that customer data and global network systems were unaffected, the implications are severe.
"Even though we understand the operational impact of the incident to be extremely limited, we took this incident very seriously because a threat actor had used stolen credentials to get access to our Atlassian server and accessed some documentation and a limited amount of source code,"
Prince, Graham-Cumming, and Bourzikas.
This breach underscores the formidable security challenges businesses face, including the urgent need to manage secrets sprawl and detect intrusions as soon as possible to prevent similar impacts.
Lesson Learned
The Cloudflare breach vividly illustrates the actual threat of secrets sprawl - the spreading of sensitive data across various platforms in an uncontrolled manner. It shows that even businesses reputed for high security, such as Cloudflare, are not impervious to attacks if secrets are not adequately secured.
In the case of Cloudflare, threat actors navigated from one leaked secret to another, reflecting how secrets serve as pivot points in hackers' attack chains. From gaining an initial foothold into a third-party auth provider (Okta) to infiltrating a downstream target (Cloudflare), the hackers collected more secrets during reconnaissance stages.
Knowing What Secrets You Have
If you are not aware you have credentials that need rotating, they will never get remediated. GitGuardian Secret Detection Platform is built to help enterprises solve this exact issue. With over 400 enterprise-tuned detectors, you can quickly identify where any secrets reside in your code or in your Slack and (soon) Jira instances. Once you know a secret has been exposed, then you can rotate it, hopefully before a breach occurs. The platform also helps you track the remediation efforts, giving you a clear timeline of when the credential was added to your codebase or first mentioned in a Slack channel and track who is working on the fix. The platform also gives you a way to quickly recheck if a token is valid and if it has been exposed publicly on GitHub yet, allowing you to assign the proper urgency to your queue. After the incident is resolved, you are left with a record of all the steps taken and can rest assured that the threat has been mitigated.
Early Warning is Key
We commend the Cloudflare team for being transparent here and for working quickly to protect their customers. While a nine-day dwell time is shorter than the average, we would always like to know sooner than later. The best defense here, we believe, is to mislead attackers into giving themselves away in a matter of minutes after establishing a foothold through GitGuardian Honeytokens. Honeytokens are decoy credentials that can be deployed at scale throughout your environments. There is no legitimate use for them, but when someone tries to use one, alarms go off, and you get the IP address, user agent, and actions the user, or more likely the scripted tool, was attempting to perform. In this case, as the threat actors moved through various systems, there is a very high likelihood they would have attempted to exploit any secrets found, making themselves known back in step 3 instead of step 6, using the stages listed in the summary.
Be Safe Out There
This incident has underlined the severe dangers of underestimating the threats secrets sprawl can pose.
It serves as a stark reminder that cybersecurity is a shared responsibility stretching across the supply chain. In order to truly secure our digital assets, there needs to be stringent control and management of secrets and prompt updates or rotations whenever a breach occurs.