Attackers' abuse of hardcoded secrets is not a far-fetched idea or a sample scenario for threat modeling–it's real and happening before our eyes. Well-documented attacks against the software supply chain show a clear pattern of attackers leveraging open-source secret scanners to gain initial access or move laterally from one system to another, venturing as deep as possible to achieve their objectives.
For the past five years, we've been helping development and security teams in organizations of all sizes control the chaos of secrets sprawl in their software delivery pipeline. And we've also been contributing to industry efforts to grasp the problem of hardcoded secrets fully and its magnitude with our unique research on all public and open-source repositories hosted on GitHub.com–analyzing over 1 billion commits yearly and publishing our results in The State of Secrets Sprawl report. For reference, in 2022, GitHub notified its partners of more than 1.7 million potential secrets exposed in public repositories. During the same period, we found approximately 3 million unique secrets 😉
Now, the word "hardcoded" and its generalized use in our field may lead us to think that secrets are only exposed in source code, and while it does get the lion's share of exposed secrets, these can be found elsewhere.
A good rule of thumb to identify areas where secrets might get exposed in your organization is to shift your gaze from the secrets themselves to the developers that handle them. On any given day, developers move up and down the software development lifecycle and go deep and wide on it, interacting with many stakeholders and juggling multiple tools, potentially contributing to amplifying the effects of secrets sprawl.
In the rest of this article, we'll look at the many places where secrets can get exposed, leaving your development and cloud environments at risk.
Version Control Systems issues and comments
While developer platforms like GitHub and GitLab serve as code repositories, they also facilitate collaboration among developers. This collaboration often involves reviewing and commenting on each other's code, creating issues to report bugs, or suggesting improvements. However, it's important to note that issues and comments can sometimes unintentionally expose hardcoded secrets.
For example, consider a scenario where a developer includes a code snippet containing API keys or passwords to illustrate his point in an issue or comment on the platform; those secrets become visible to anyone with access to the project.
CI/CD systems
Maturing DevOps practices have turned CI/CD pipelines into the central nervous system of almost every modern cloud-native application development environment. These pipelines automate applications' building, testing, and deployment, ensuring a swift and reliable delivery cycle.
Unfortunately, misconfigurations in CI/CD pipelines can inadvertently leak secrets and expose them in plain-text format in the job logs. For instance, a developer might inadvertently include a secret in a pipeline configuration file or fail to encrypt or mask sensitive information properly. As a result, the secrets are logged and potentially accessible to unauthorized individuals who gain access to those logs.
To illustrate this, consider the following example of a misconfigured CI/CD pipeline using a popular tool like Jenkins:
pipeline {
stages {
stage('Build') {
steps {
sh 'echo "Secret API Key: $SECRET_KEY"'
}
}
}
}In this example, the pipeline prints the value of the `$SECRET_KEY` environment variable directly to the console output. If the variable contains a sensitive API key, it will be visible in the job logs, exposing it to potential attackers.
Thomas Segura
Observability systems
Observability platforms, such as Datadog, New Relic, Grafana, and many others, provide valuable insights into the performance and behavior of applications. However, these platforms may inadvertently expose sensitive data, including secrets and credentials, in various ways.
For instance, consider an application that generates logs containing interactions with external APIs. If a developer inadvertently includes an API key in the logging statements, it becomes visible in the logs. If an application utilizes Real User Monitoring or Session Replay events, which capture user interactions, sensitive information like usernames and passwords might be logged, compromising security.
Container images
Docker images provide a convenient way to package and deploy applications. However, they can also unintentionally expose secrets if developers are not cautious.
One common source of exposed secrets in Docker images is including sensitive information in the source code itself. For example, a developer might mistakenly commit source code that contains hardcoded credentials, such as API keys or passwords. These secrets become embedded within the Docker image, making them accessible to anyone who has access to the image.
Furthermore, using environment variables within Dockerfiles can introduce risks if not handled carefully. Developers might inadvertently pass sensitive information through environment variables used in the Dockerfile, like in the following example:
FROM python:3.9
# Add API key as an environment variable
ENV API_KEY=mysecretapikey
# Install dependencies
RUN pip install ...
# Copy source code
COPY . /app
# Set working directory
WORKDIR /app
# Start the application
CMD ["python", "app.py"]In this example, the API key is set as an environment variable within the Dockerfile itself. However, this exposes the secret directly in the file, making it visible to anyone with access to the Docker image or its configuration.
Another risk with Docker images is the layered structure they employ. Each layer in a Docker image represents a change or modification to the image. This structure allows for greater efficiency in image delivery and deployment. However, it also means that secrets included in a previous layer may not be readily visible in the final state of the image, potentially leading to their unintended exposure.
Guardians
Learn how better to manage your secrets in Docker with this hands-on guide:
Guest Expert
Runtime environments
Runtime environments, including servers, virtual machines (VMs), applications, data, and appliances, are potential targets for secret exposure. In a notable example from last year, security firm Wiz uncovered a supply chain vulnerability in IBM Cloud Databases for PostgreSQL, dubbed "Hell's Keychain." The researchers exploited this weakness by accessing the Kubernetes pod where the PostgreSQL instance ran.
By exploiting the Kubernetes API token present in the `/var/run/secrets/kubernetes.io/serviceaccount/token` file, they accessed the Kubernetes API and utilized the `kubectl` utility to inspect privileges and available resources. This allowed them to advance their attack further, illustrating the criticality of securing runtime environments.
Messaging and communication tools
Platforms like Slack and Microsoft Teams have become integral communication platforms for distributed development teams. Teams leverage these tools to communicate in real-time, share code snippets, discuss issues, and collaborate on tasks. However, developers must be cautious when using these tools as they may inadvertently share secrets in private messages or channels. For example, a developer might privately share an API key or password to verify its functionality or troubleshoot an issue. It's vital to be mindful of the sensitivity of the information transmitted and ensure that secrets are only exchanged through secure and encrypted channels.
Please note this integration is only available for Business and Business trial accounts on the SaaS version. To learn more, please read our public documentation.
Static storage
Services, like S3 in AWS, are commonly used to store files and assets. While these buckets offer features that allow files to be accessible to the public without authentication, they pose a significant risk if sensitive information, such as developer secrets, is stored within them. While this feature is beneficial for storing content intended for public consumption, like compiled or uncompiled assets like HTML, CSS, and JavaScript for a website, it becomes a double-edged sword when protecting secrets. It's crucial to ensure that secrets are appropriately secured and not displayed in plain text in these assets.
Project management & productivity tools
Product and engineering teams rely on project management and productivity tools like Jira, Linear, or Monday to collaborate, plan software development, and track progress. These tools are repositories for technical specifications, tickets, and tasks engineers pick up for their work. However, these platforms can unwittingly expose secrets if developers include sensitive information in ticket descriptions, comments, or attachments. Developers and teams need to be mindful of what information they have in these tools and ensure that secrets are not accidentally revealed.
Guest Expert
Developer documentation
Developers often document their projects and applications using various tools like Confluence, Notion, Google Docs, or even Git repositories using frameworks like Docusaurus or Markdoc. While documentation is essential for knowledge sharing and reference, it can also be a potential source of exposed secrets. If developers include sensitive information like API keys, database credentials, or encryption keys within the documentation, it puts their applications' security at risk. It's crucial to carefully review and scan for secrets before including any content in documentation to prevent unintentional exposure.
If you create and maintain your documentation "as code," you can configure ggshield (the GitGuardian CLI) in pre-commit hooks to scan your markdown content for secrets before it's pushed to your remotes.
Conclusion
The exposure of hardcoded secrets goes beyond source code repositories. Throughout the software development lifecycle, areas like issues and comments, CI/CD systems, observability platforms, container images, runtime environments, static storage systems, project management tools, developer documentation, and chat platforms can all become potential sources of secret exposure.
As developers and organizations, it's essential to be aware of these risks and take proactive measures to prevent the exposure of sensitive information. Regular security audits, strong access controls, proper encryption, secure coding practices, and ongoing training and awareness are crucial in mitigating the risks of exposed secrets. By adopting a comprehensive approach to application security and staying vigilant, we can significantly reduce the likelihood of secrets being compromised and protect our development and cloud environments.
