Secrets at the Command Line [cheat sheet included]
If developers all share one thing in common, it is their use of the command line to get their jobs done. Many development tools don’t come with a graphical user interface (GUI) and rely on a command line interface (CLI). There are a lot of upsides to a CLI first or only approach. Once you master the command line, you can work more efficiently than a GUI might allow and gain the awesome superpower of scripting, allowing all of your tools to start working in concert. Scripting is the bedrock for building and managing software delivery pipelines and CI/CD workflows.
However, there is nothing more intimidating, especially to a newer dev, than a blank terminal window with a blinking cursor awaiting your commands. There is no helpful UI to guide you towards your goal; you just have to know what to enter. All the burden of getting it _just right_ falls on the developer's shoulders, and there is a lot to learn, especially when you factor in security.
One area that often gets neglected while mastering the command line is local security around credentials, or what we like to call secrets. While it might feel like secrets management is an area reserved for code repositories, run time environments, and the CI/CD pipelines that drive modern application delivery, good security practices start at home. ‘Home’ for devs means the terminal.
What Are Local Credentials
The first step toward securing secrets in the command line is taking inventory of what secrets might exist. Secrets are any sensitive data that we want to keep private that would grant access to systems or data, what you will see referred to as digital authentication credentials. Secrets fall into a few broad categories: Passwords, Keys, and Certificates
Passwords
Since the beginning of computing, passwords have played a large part in security, and are synonymous with how a user can log into a system. The term “user name” is almost always paired with the word “password” when either is used. Passwords are something you know to prove who you are.
Keeping your passwords secure is as important as having them in the first place. No one would think it is a good idea to write passwords on a post-it note and stick it on their monitor, but a lot of developers are guilty of storing their passwords in plaintext in local files. If someone gets access to your local machine, a quick search of the contents can reveal any such document and start pretending to be you.
In an ideal world, you would simply keep all your passwords securely inside your head. But in reality, we all have way too many passwords for that to be a viable strategy. Fortunately, there are many approaches for securing passwords locally, and we will dig into those later in this article; for now, let’s keep going with making our inventory of local secrets.
Keys
Keys serve the same basic function as passwords, granting access to systems and data, but differ in several key ways. Passwords are generally generated by humans, for human access to a system or data. They are generally shorter, and ideally, you can memorize them.
Keys are typically generated by an algorithm and are generally much longer and more complex than passwords. Overall, they are not meant to be manually entered, nor are they intended for human access to a system; keys are meant to grant machines and processes access to another system or unlock encrypted data. Another way to say that would be keys are used to lock and unlock cryptographic functions such as encryption or authentication.
One type of key you are likely very familiar with as a developer is your SSH key. This is a prime example of a paired public/private key system at work. SSH public keys are intended to be shared with any remote system that applications your local machine will need to access, such as GitHub or AWS. A corresponding private key is stored on your local file system and should never, ever be shared with anyone. When used together, these keys ensure that access is granted only to trusted systems. This is a very secure approach.
Just like with passwords, the security benefits of keys come with the overhead of needing to secure them locally. While it is unlikely you would hand type a plaintext key into a system, or write it down on paper, as we will see in the next section, there are multiple ways keys can become exposed if we are not careful.
Certificates
Certificates are a way to store, transport, and use keys. Certificates contain keys, as well as important metadata, such as the issuer, what the certificate is for, and a signature intended to verify the authenticity of the certificate.
While SSL or TLS certificates might spring to mind as a primary use case for security certificates, an increasing number of applications and platforms, leverage certificate-based authentication. Identity management services, like Active Directory, offer integrations that make it easier to leverage certificates to better control access rights for users, while freeing the users from needing to manually manage passwords.
Where and how you store certificates locally might not be obvious, especially on machines provisioned by central IT departments. It is still the developer's responsibility to be aware of what certificates are on your local machine and to ensure the right safeguards are in place to prevent them from being inappropriately shared. It is also important to not expose the keys these certificates hold, as that is a potential threat as well.
How Credentials Can Be Exposed
Now that we know what kinds of secrets we are looking for, the next logical question is how might they be exposed to a bad actor or malicious code? Secrets can be exposed from your laptop can be compromised, and any plaintext files with passwords or stored keys can be stolen. But there is also the possibility that any server you connect to via SSH might also be unexpectedly accessed. The SSH credentials themselves might mean someone can cause problems while pretending to be you!
Keeping command line secrets safe in any situation where you are using a shell will help you stay safe. Let’s take a look at some of the ways you might be exposing secrets.
Credentials Files
Credential files are a way to store secrets safely away from any directories that get version controlled. You can set permissions for these files easily with chmod and can programmatically access contents. You can even manage a separate file for each credential, making it harder for any intruder to gather them and limiting the scope of their attacks. These do carry the risk of having credentials in plaintext, but we can address that, and any file, with good encryption.
Bash History
Entering passwords or keys into a CLI prompt is necessary from time to time. The danger here comes from the fact that anything entered into the terminal in plaintext means it is stored in plaintext in your terminal’s history. All shells store your history, but to help put this in perspective, if you are running Bash or Zsh, your entire shell history is stored in a file called either .bash_history or .zhs_history. If you go examine that file, you might be surprised at the number of lines it contains. Anyone who gains access to your machine or a shared terminal environment would quickly be able to find any credentials entered directly into a shell by you as a user.
Fortunately, most applications have ways to safely pass credentials without entering them as plaintext. If you encounter an application that needs plaintext credentials, you should consider one of the approaches we outline in the next section. Most of the time, you can still work safely and never expose a credential. If you work with a tool that in no way allows you to pass credentials in a safe way though, it might be time to have a conversation with your security team about the tool in question.
Logs
Just like with your Bash history, logs can expose any and all secrets that are stored in plaintext or loaded in an insecure way. Arguably, log files are less secure than your Bash history, given that logs are publicly visible in /var/log and might be accessed by unexpected actors.
/dev/stdin
Piping credentials between locations is overall a very secure way to handle secrets assuming you are not calling the special /dev/stdin file along the way. Shells like Bash that use stdin (Standard Input) automatically store any input into a file that is accessible by any other process on your machine.
If you go to the terminal right now and just try to print that file with cat, a curious thing happens. Anything that you type is immediately printed to the screen upon hitting enter. Why? Your shell is concatenating (cat) the contents of the file at /dev/stdin to the standard output ( stdout) of the terminal. If there is malware or spyware on your computer, or if someone has injected code into your scripts, it is possible that they can intercept the plaintext contents of this file even if you securely loaded a password or key directly into stdin from a secure source.
ps
Process Status is a utility in Bash and derivative shells that you invoke with the command ps. This utility provides information about processes running in memory and is very important for understanding what your system is doing. For UNIX-like OSes, any value, including the contents of private key files, can be seen via ps when these commands are running and it stores them locally in a file, /proc/<pid>/cmdline, which is globally readable for any process ID, (pid). This can become an especially dangerous situation on machines with shared access, such as remote VMs or servers.
Approaches To More Secure Command Line Credentials
While local credentials management might feel daunting, there are a number of approaches and tools that can help you work more safely and with more confidence in your day-to-day duties. While we are going to cover some of them here, we recognize there are likely more tools and tips that can address this issue; we invite your thoughts on this by letting us know on social media or our contact form.
Password Managers
You might already be familiar with password managers through your internet browsers. Just as providers like LastPass, 1Password, or DashLane, have made managing logging into web interfaces a lot simpler, there are plenty of tools out there that can help us store and manage our passwords safely for use on the command line.
One of the best examples of such a solution is Hashicorp Vault. They have great documentation on how to leverage Vault for programmatically calling any needed credential without exposing it in plaintext. If a bad actor gets their hands on your code, they will see calls to Vault rather than the keys themselves, making it much harder to cause you any harm.
There are plenty of alternatives to Hashicorp Vault as well, such as KeePass, Azure Key Vault, Keeper Password Manager, and Akeyless Vault Platform, just to name a few. They all offer their own idiosyncrasies, but as long as they keep your passwords secure and out of plaintext, then we encourage you to adopt one as soon as you can. Your IT and Security teams likely already have some approved password managers you can start using right away.
Encryption
While we often think of encryption while transporting data, it is equally, or perhaps even more important at times, to address encryption of data at rest. Any sensitive data that we do not want to expose should be encrypted when we are not actively working with it. Admittedly this does carry the overhead of having yet another encryption key to manage. However, combined with a good password manager, you will be surprised how easy it can be to keep credentials unusable by bad actors.
While there are a lot of possible ways to employ encryption, here are three types of tools we think are useful.
Local Filesystem Encryption
If a bad actor accesses your computer, you will want to make it as hard as possible for them to actually do anything with your data. That is the core idea behind local filesystem encryption; when you are logged out, the data on your machine gets encrypted and becomes unusable to anyone else.
There are a lot of different options out there for Linux systems derived from LUKS, Linux Unified Key Setup. In fact, when installing most distributions of Linux, you will be prompted to enable this by default.
Windows users can leverage tools like Microsoft’s BitLocker, Folder Lock, or free open source tools like VeraCrypt,
If you are using a newer Macbook with the T2 security chip integrated, good news, you already have a very sophisticated encryption tool ready to use; FileVault.
SOPS
SOPS name derives from the term, “Secrets OPerationS”. Unlike local file encryption schemes, SOPS is an encrypted files editor, created by the team at Mozilla. The idea is to remove the manual steps of decrypting highly sensitive files, editing, and then re-encrypting them. Instead, SOPS offers an editing experience that keeps encryption in place throughout the editing process. When opening SOPS encoded files with other text editing tools, the structure of the file is preserved, but any sensitive data is protected.
SOPS is highly customizable and allows you to choose from multiple encryption mechanisms like GPG or Hashicorp Vault, making it easy to fit your workflow. It is a free and open-source tool. There is even a VS Code extension available.
Shellclear
Shellclear is a cross-platform shell plugin that promises “a simple and fast way to secure your shell commands history”. It works by
- Showing sensitive command summary when opening a new terminal
- Clearing sensitive commands from shell history
- Stash your history command before presentations OR screen sharing
It is free and open source and very customizable. While this is a newer project, we think it is an elegant solution to finding what secrets are in your Bash history and helping you clean them out.
These are just a few options for securing data at rest on your machine. There are a lot of other tools and methods available. When evaluating any encryption tool, make sure they use a proven, known encryption algorithm. This is definitely one area where you do not want to write your own encryption scheme. Talk to your security folks about other options and tools that they might already have approved.
Environmental Variables
In simplest terms, environmental variables are the settings the terminal uses to set behaviors such as time formatting or local UTF encoding. By default, these are only accessible to you as the user and the system at runtime.
Environment variables can also be used to store credentials locally, especially for systems called programmatically. Overall this is a safe approach as they are stored safely in the system and can easily be invoked in scripts. However, there are a few things to be aware of:
- They should be correctly scoped, i.e. not set at a global level
- A process spawning a child subprocess will copy its env var to it! i.e a script with access to sensitive envs will pass these upon launching another process
If you have never run the command env in a terminal, the number of variables already there might seem a bit overwhelming. We invite you to pop open a terminal and run it now. If there is anything in there that looks like an API key or bearer token, it is a good idea to ask if it is still needed and clear out environmental variables that are not in use.
Pipes
While we mentioned using pipes as a potential security threat surface if used incorrectly earlier in this article, we do actually love pipes in general!
Pipes in Bash take the output from one process or application and pass it into another process or tool. Since there are only two ends to a pipe and they exist in memory only for the duration of the communication, they are extremely secure in and of themselves.
The issue around using pipes comes from the temporary storage of items from standard input, stdin. This issue can be solved by moving the input to the pipe close to the source, aka the application that is feeding into stdin.
A good example that illustrates this point can be seen with cURL. When passing data to a request, you first might try to just pass it from the output with -d "$(< /dev/stdin)". Instead, cURL allows you to grab the data directly from the source without writing to stdin at all with -d @-. Thanks to Carl Tashian for writing a very good summary of this pattern on his blog.
Keep Your Credentials A Secret
Working on the command line means working faster and more efficiently with a wider range of tools at your disposal. However, as we have spelled out here in this article, it also brings a certain overhead of vulnerability when it comes to credentials management.
The good news is you are not alone in this fight to keep your secrets a secret. There are plenty of tools out there that can help keep your credentials safe and secure. To sum up, our advice is to:
- Identify any and all places where credentials are stored on your computer that are accessed by the command line. These take the form of Passwords, Keys, and Certificates.
- Stay alert and be aware of how your local credentials can become compromised. Storing anything sensitive in plaintext, or even calling the standard input ‘/dev/stdin’, is most likely the wrong approach.
- Use tools to keep files encrypted when at rest. Also, leverage tools to clean up after yourself in your logs and Bash history.
We encourage you to have a conversation with your teams about security and see what tools they have already vetted for your organization. They will likely be able to help you identify ways you can work more securely every day while staying productive on the command line.