Storing and managing secrets like API keys and other credentials can be challenging, even the most careful policies can sometimes be circumvented in exchange for convenience. We have compiled a list of some of the best practices to help keep secrets and credentials safe. Secrets management doesn’t have a one-size-fits-all  approach so this list considers multiple perspectives so you can be informed in deciding to, or not to, implement strategies.



Never store unencrypted secrets in .git repositories

It is common to wrongly assume that private repositories are secure vaults that are safe places to store secrets. Private repositories are not appropriate places to store secrets.

Private repositories are high value targets for bad actors because it is common practice to store secrets within them.

In addition, .git is designed to sprawl. Repositories get cloned onto new machines, forked into new projects and new developers regularly enter and exit a project with access to complete history. Any secrets that exist within a private repository's history will exist in all new repositories born from that source.

If a secret enters a repository, private or public, then it should be considered compromised.

A secret in a private repo is like a password written on a $20 bill, you might trust the person you gave it to, but that bill can end up in hundreds of peoples hands as a part of multiple transactions and within multiple cash registers.

Avoid git add * commands on git

Using wildcard commands like git add *or git add . can easily capture files that should not enter a git repository, this includes generated files, config files and temporary source code.

Add each file by name when making a commit and use git status to list tracked and untracked files.

According to git-scm:

“Remember that each file in your working directory can be in one of two states: tracked or untracked.

Tracked files are files that were in the last snapshot; they can be unmodified, modified, or staged. In short, tracked files are files that Git knows about.
Untracked files are everything else.”

Advantages

  • Complete control and visibility over what files are committed
  • Reduces the risk of unwanted files entering source control
  • Requires thought and consideration when adding files

Disadvantages

  • Takes additional time when making a commit
  • Can mistakenly miss files when committing
TIP: Committing early and committing often will not only help navigate file history and break up otherwise large tasks, in addition it will reduce the temptation to use wildcard commands.

Add sensitive files in .gitignore

To prevent sensitive files ending up within git repositories a comprehensive .gitignore file should be included with all repositories and include:

  • Files with environment variables like .env or configuration files like .zshrc or .config
  • Files generated by another process (such as application logs or checkpoints, unit tests / coverage reports)
  • Files containing “real” data (other than test data) like database extracts.

GitHub published a collection of useful .gitignore templates. See these examples for Python or Go.


Don’t rely on code reviews to discover secrets

It is extremely important to understand that code reviews will not always detect secrets, especially if they are hidden in previous versions of code. The reason code reviews are not adequate protection is because reviewers are only concerned with the difference between current and proposed states of the code, they do not consider the entire history of the project.

If secrets are committed into a development branch and later removed, these secrets won’t be visible or of importance to the reviewer. The nature of git means that if a secret gets overlooked in history it is compromised forever as anyone with access to the repository can find this secret in previous revisions of the codebase.

TIP: As a rule, automation should be implemented wherever predefined rules can be established, like secrets detection. Human reviews should be left to check code for errors that cannot be easily predefined, such as logic.

Use automated secrets scanning on repositories

Even when all best practices are followed, mistakes are common. When dealing with highly sensitive data, no chances should be taken. GitGuardian offers a free secrets scanning solution for developers which should be installed on both private and public repositories.

Visibility is the key to great secret management, if you don’t know you have a problem, you cannot take action to fix it. Secrets scanning provides essential visibility over your internal systems.

It is important to also consider that even the best secrets management systems and policies do not prevent newly generated secrets entering the code base or old secrets being extracted and included again.

Advantages

  • Difficult to circumvent and ignore compared to tools that need to be manually run
  • Much faster and more accurate than relying on human checking
  • Can detect secrets buried within logs and history that manual reviews and searches will not uncover
  • Live scanning ensures all active data leaks are captured

Don’t share your secrets unencrypted in messaging systems like slack

A common secret sprawl enabler is sending secrets in plain text over messaging services. While these systems are intended to keep messages secure, they are not intended to hold sensitive information such as secrets.

These systems are high value targets for attackers, it only takes one compromised email or Slack account to uncover a trove of sensitive information. If secrets are being sent over internal systems it also makes it possible for bad actors to move laterally between services by ‘using secrets to find secrets’.


Store secrets safely

There is no silver bullet solution for secrets management, different factors such as project size, team geography and project scope must be considered. Multiple solutions may need to coexist. Carefully consider each option, not just to meet your current needs, but also considering how each solution will scale with the growth of your project.


Use encryption to store secrets within .git repositories

Encrypting your secrets using common tools such as git secret and storing them within a git repository can be beneficial when working in teams as it keeps secrets synced. This does however introduce a new issue to consider because you now need to handle additional security keys to encrypt and decrypt secrets. These keys also need to be stored and securely shared which might make it seem like a never ending problem!

Advantages

  • Your secrets are synced

Disadvantages

  • You have to deal with your encryption keys securely
  • No audit logs (who accessed which secret and when)
  • No role based access control (RBAC)
  • Hard to rotate access. Rotating access implies to revoke the key and redistribute it. The distribution part is not easy to handle with git repositories when dealing with multiple developers

Use local environment variables, when feasible

An environment variable is a dynamic object whose value is set outside of the application. This makes them easier to rotate without having to make changes within the application itself. It also removes the need to have these written within source code, making them more appropriate to handle sensitive data.

Advantages

  • They are easy to change between deployed versions without changing any code
  • They are less likely to be checked into the repository
  • Simple and clean

Disadvantages

  • This approach may not be feasible at scale when working in teams because there is no way to easily keep developers, applications and/or infrastructure in sync

Use "Secrets as a service" solutions

Secrets management systems such as Hashicorp Vault or AWS Key Management Service are encrypted systems that can safely store your secrets and tightly control access. Vaults and other managed secrets solutions are not appropriate in all cases because they are complicated to set up and need to be well maintained. Both take a considerable investment of resources.

Advantages

  • It prevents secrets from sprawling
  • It provides audit logs

Disadvantages

  • As they introduce a single point of failure, they must be hosted on a highly-available and secure infrastructure
  • All the codebase must be changed to integrate with them
  • Keys giving access to the system must be carefully protected

Restrict API access and permissions

It can be difficult to detect when an attacker is using secrets like API keys maliciously, because often, they are using them within their scope. By restricting access and permissions of the API key you not only limit damage and restrict lateral movement, you also provide greater visibility over when a API key is being used outside of its scope.


Default to minimal permission scope for APIs

When using external services, make sure the permissions of that API matches the task it is fulfilling. This includes making sure you have separate APIs for read only and read/write permissions as needed. Many APIs also allow you to have increased control over what data can be accessed, for example the Slack API has a large range of scopes, using these scopes to meet the minimal requirements of the task is important to prevent an attacker accessing sensitive data or moving laterally through systems. It  is common for inexperienced developers to use master APIs allowing them to use one key throughout all your projects. But this increases the potential damage of a data breach.


Whitelist IP addresses where appropriate

IP whitelisting provides an additional layer of security against bad actors attempting to use APIs nefariously. By providing a whitelist of IP addresses from your private network, your external services will only accept requests from those trusted sources. It is common to include a range of acceptable IP addresses or a network IP address.

In an example from GitHub, you can use IP whitelisting to prevent any untrusted sources accessing your GitHub repositories. “The allow list for IP addresses will block access via the web, API, and Git from any IP addresses that are not on the allow list.”

Advantages

  • Limited requests to select trusted sources and prevents attacks from external sources even with secret keys

Disadvantages

  • Not always feasible depending on the traffic the source is expecting
  • Can prevent legitimate information requests
  • Needs to be maintained constantly

Use short-lived secrets

It is common for APIs to typically provide long-lasting access tokens. These tokens could last indefinitely. While this is convenient for developers this means that a secret poses the same security risks for its entire life and increases the chances of them being used in an attack. Short lived secrets prevent historic data leaks that were not detected from still being a threat and means that if an attacker discovers a secret his access will still be cut off if the breach is not discovered.

It is also good practice to make sure you revoke and redistribute all APIs often, particularly if it is not possible to introduce validity period on APIs.

Imagine you own a company with hundreds of employees that all have keys to your office, keys will inevitably get lost, employees will leave the company, new keys will get cut and you will soon lose visibility over where each key is. It would be widely considered good practice to change the locks from time to time.

Advantages

  • Enforces good secret hygiene
  • Reduces the risk of long term threats

Disadvantages

  • Requires an active secrets management strategy

Conclusion

Managing secrets and storing secrets is a challenge that requires vigilance from even the most experienced developer, who needs to carefully consider how they are using, storing, sharing and distributing secrets. Unfortunately there is no perfect checklist that a developer can follow and policies, tools and strategies will differ from projects but it is crucial for developers to understand consequences of policies so that secrets management can be an informed, active strategy throughout the entire development process.