TLDR

Sumo Logic reported a security breach on November 3, 2023, due to a compromised credential that allowed unauthorized AWS account access. They quickly secured their infrastructure and advised customers to rotate their API access keys as a precaution, hinting at potential risks. The incident underscores the persistent challenge of safeguarding credentials, with factors such as human error, system complexity, inadequate security training, and insufficient secret management tools contributing to such leaks. While the breach's specifics weren't disclosed, leaked credentials were found in public GitHub repositories, although their connection to the breach remains unclear. Services, such as GitGuardian, offer methods to determine whether credentials have been compromised, emphasizing the importance of proactive measures in secret management.

What happened with Sumo Logic

Sumo Logic has reported a security breach detected on November 3rd, 2023, involving a compromised credential that allowed unauthorized access to one of their AWS accounts. Despite this, they claim to have not found any impact on their networks or systems, and customer data continues to be encrypted.

In response to the incident, Sumo Logic stated that they “immediately secured the exposed infrastructure and rotated all potentially compromised credentials".

Despite this, Sumo Logic still advises all customers to rotate all Sumo Logic API access keys immediately, which indicates that there is definitely some potential for the attackers to move from the infrastructure of Sumo Logic into their customers' dashboards and supply chain, potentially gaining more information and credentials. They also offered some additional steps people should take to be cautious on November 7th but were later downgraded on November 8th only to include 

“Third-party credentials that have been stored with Sumo as part of webhook connection configuration.” Again, this indicated that an attacker could have moved into the dashboard of its customers. 

This breach, while not overly significant, shows a continued trend we have seen in the industry over the past few years of secrets (credentials like the keys in question here) being targeted by attackers. 

Why are leaked secrets such a persistent problem? 

In the interconnected realm of modern software development, secrets such as passwords, API keys, and tokens are the linchpins of security. Their leakage poses severe risks, often with far-reaching consequences. Not only do these secrets allow an attacker into a system, but they also provide a valid method of communication that often bypasses any threat intelligence tools as they operate within the expected conditions. Additionally, secrets often grant access to more secrets, the example with Sumo Logic is perfect, based on the company's own recommendation you should rotate your access API keys and any third-party keys stored with Sumo Logic, as an attacker might be able to move laterally through different systems. 

We are all aware secrets are sensitive, so why do leaks still happen? Like everything in security, it is often a combination of factors.

Human Error: The Achilles' Heel

The most common culprit behind the leaking of API keys is human error. Developers might inadvertently include keys in code repositories or share them in internal communications that become exposed. 

Complexity of Modern Systems

As systems become more complex, the sheer number of keys and secrets multiplies. Each service integrated into an application demands its own key, escalating the challenge of securely managing a growing inventory of sensitive information.

Insufficient Security Training

Developers and IT professionals are under constant pressure to deliver rapidly. In the race against time, security best practices may take a back seat. Without regular and rigorous security training, the proper handling of API keys can be overlooked.

Lack of Robust Tools

Organizations sometimes fail to provide robust tools for secret management. Developers resort to makeshift methods like storing keys in configuration files or environment variables, which are not secure and can easily be leaked through version control systems. That, coupled with a lack of quality detection tools, can lead to undetected leaks.

Overlooking Key Rotation

The practice of rotating API keys regularly is a critical security measure. However, it's often neglected due to the operational overhead it creates. Static keys that remain unchanged are at a higher risk of being leaked and exploited.

The Challenge of Tracking

In large-scale operations, it's challenging to track which keys are active and who has access to them. When developers leave a team or service accounts decommissioned, keys that should be invalidated continue to linger unnoticed until they are possibly leaked.

In summary, the leakage of API keys is often a blend of human error, complex systems, inadequate tools, and overlooked security practices. By addressing these factors, organizations can better protect these valuable digital assets and maintain the integrity of their systems.

How Attackers Find Leaked Secrets

In the case of Sumo Logic, no information has been provided about how the leak happened. After some digging, we were able to find examples of when AWS credentials appear to have been leaked through public GitHub repositories belonging to Sumo Logic. However, we must note these are no longer valid, and we cannot be sure if they were valid at the time they were leaked.

While there is no indication GitHub is the source of the leak, it does provide an indication that perhaps keys were being mishandled by developers at Sumo Logic. 

On a more general note, attackers are becoming increasingly sophisticated in how they find credentials by specifically targeting developers who often have access to source code, which frequently contains secrets. We are also seeing increases in supply chain attacks, like what we saw this year at Circle CI or several years ago at CodeCov. 

Find out if your secrets have leaked

One of the most difficult aspects of leaked secrets is that leaks often happen outside of the visibility of organizations. The most common example is on the personal GitHub account of a developer working for the organization. But now you can discover if a credential has leaked on GitHub at any point during the last 6 years. The Has My Secret Leaked service by GitGuardian takes a secure hash of your secret and compares it to a database of over 20 million leaked secrets to discover if your secrets ever made their way to the public. Note: GitGuardian doesn’t use or store your secret, only a secure hash of the secret. 

You can also use secret scanners like GitGuardian or GGShield to discover if you have secrets inside your code repositories or within your infrastructure. Revoking secrets that have leaked internally will significantly reduce the risk of them being used in a breach, like with Sumo Logic.