1. Introduction to SOPS
1.1 What is SOPS?
First, let's cut to the chase and get straight to the elephant in the room: what's SOPS?
Because an image is worth a thousand words, here is SOPS illustrated (kudos to @sec_r0 for the artwork!)
SOPS, short for Secrets OPerationS, is an open-source encrypted files editor that encrypts/decrypts files automagically with AWS KMS, GCP KMS, Azure Key Vault, PGP, etc.
Emphasis on file editor, encryption, and automation.
Typically, when you want to encrypt a text file, this is what you do:
- Use your favorite editor for writing, editing, and manipulating the text data, and save it as a file.
- Use an encryption/decryption tool to encrypt the whole file.
When you need to read the encrypted file:
- First, you need to decrypt the file with an encryption/decryption tool.
- Open the decrypted file (now it's a regular text file) with a text editor of your choice.
The drawback of this "normal" process is obvious: you need two tools (an editor and an encryption/decryption tool) for one job.
Enter SOPS.
Initially launched in 2015 as a Mozilla project, it has been donated and accepted to CNCF on May 17, 2023. Check out the official website here for more information.
Today, let's have a look at how it works and how to use it with various key management services such as AWS KMS and HashiCorp Vault.
1.2 Install SOPS
For macOS users, the easiest way to install SOPS is via brew, the package manager for macOS:
brew install sops
For other OS users, binaries and packages of the latest stable release are available at the official GitHub repo of SOPS.
💡
Note: you can use other text editors with SOPS. For VSCode, you even have an extension called
vscode-sops
that will enable you to auto-decrypt/encrypt files within VSCode with a simple project-based configuration. Please bear in mind this extension is still in early development and may contain bugs.If you don't want to install the extension, you can still use VSCode to edit the files by setting your
EDITOR
environment variable tocode --wait
in your~/.bashrc
or~/.zshrc
, such as:export EDITOR="code --wait"
, then when you runsops
command to edit a file, VSCode will be used automatically.
1.3 How Does SOPS Work?
In short, SOPS offers encryption/decryption and editing in one package, and the workflow is fully automated. This is where it shines:
- When you write a file, SOPS first uses the specified encryption method of your choice to encrypt the file before saving it to the disk. This is done automatically, and no human intervention is required whatsoever. Of course, if you open the file with another text editor other than SOPS, you can't read the content since it's encrypted.
- When you read the encrypted file with SOPS, it first decrypts, opens it, and shows it to you. Although this sounds like a two-step job, SOPS does it automatically, so it looks like only one step to the end users.
1.4 Highlights: Flexibility
SOPS supports various encryption methods, like:
- GPG (if you don't know what it is, read on)
- AWS KMS
- GCP KMS
- Azure Key Vault, age, and PGP
- HashiCorp Vault
- and so on.
This makes managing and editing sensitive files simple and flexible.
If you want to edit a text file using a local GPG key, no problem; if you want to edit an ENV file and want it to be encrypted by HashiCorp Vault, SOPS also has you covered. And, you don't have to use the same encryption method for all your files because SOPS is highly customizable.
OK, enough said; let's give it a go to have a hands-on feeling about it.
2. SOPS with PGP Keys
2.1 PGP vs GPG
PGP, GPG, confusing... I know, right?
In short, GPG is a CLI tool that implements PGP:
- PGP (Pretty Good Privacy) is an encryption program that provides cryptographic privacy and authentication for data communication. It can be used for signing, encrypting, and decrypting texts, e-mails, files, directories, and whole disk partitions and to increase the security of e-mail communications.
- GPG, on the other hand, stands for GnuPG, which is a free implementation (a command line tool) of the OpenPGP standard, RFC4880. GPG encrypts and signs your data and communications; it features a versatile key management system and access modules for all public key directories.
Now that we've got the confusing names out of the way, let's create a PGP key using the GPG CLI tool; then, we can test how SOPS works.
2.2 Creating PGP Key with GPG
For macOS users, the easiest way to install GnuPG is via brew:
brew install gnupg
If you are using another OS, for example, Linux, you can use the corresponding package manager. Most likely, this would work:
sudo apt-get install gnupg
With GnuPG, Creating a key is as simple as the following (remember to put your name as the value of KEY_NAME
):
export KEY_NAME="Tiexin Guo"
export KEY_COMMENT="test key for sops"
gpg --batch --full-generate-key <<EOF
%no-protection
Key-Type: 1
Key-Length: 4096
Subkey-Type: 1
Subkey-Length: 4096
Expire-Date: 0
Name-Comment: ${KEY_COMMENT}
Name-Real: ${KEY_NAME}
EOF
Now, if we run gpg --list-keys
, we will see the generated key:
$ gpg --list-keys
gpg: checking the trustdb
gpg: marginals needed: 3 completes needed: 1 trust model: pgp
gpg: depth: 0 valid: 1 signed: 0 trust: 0-, 0q, 0n, 0m, 0f, 1u
/Users/tiexin/.gnupg/pubring.kbx
--------------------------------
pub rsa4096 2022-09-15 [SCEA]
A2B73FB4DA0891B38EECD35B47991CD146C9C4BC
uid [ultimate] Tiexin Guo (test key for sops)
sub rsa4096 2022-09-15 [SEA]
In the "pub" part of the output, we can get the GPG key fingerprint. Alternatively, we can run gpg --list-secret-keys "${KEY_NAME}"
to get it.
Store the key fingerprint for future reference.
2.3 SOPS Configuration
We need a simple configuration to let SOPS know we will use the previously generated PGP key for encryption/decryption. To do so, create a file named .sops.yaml
under our $HOME directory with the following content:
creation_rules:
- pgp: >-
A2B73FB4DA0891B38EECD35B47991CD146C9C4BC
Remember to replace the key fingerprint you generated in the previous step.
OK, now we are all set.
2.4 SOPS with Text Files
Run the following command to create a new file named a-text-file.txt
:
sops a-text-file.txt
Start editing the content (for example, I put "hello world"), then save.
Now, if we try to cat the file, we won't see the content, because it's encrypted:
$ cat a-text-file.txt
{
"data": "ENC[AES256_GCM,data:U1j0eOC7URJZsNb+Iqs=,iv:4F3Bw1YOIE2xxdag+PbrMLGcqAjG48qVDeiu+xTfnro=,tag:2tX0jUssBYeI0IF7Lxmy+A==,type:str]",
"sops": {
"kms": null,
"gcp_kms": null,
"azure_kv": null,
"hc_vault": null,
"age": null,
"lastmodified": "2022-09-15T04:38:19Z",
"mac": "ENC[AES256_GCM,data:w1HJ/A7t0DxD1KObLeLQVzSWVrobIAF/lUg/8DRH9khba8/cDGVjhrkheki09uWDfHo4vb+odrPsBBW6vtRemqd4Y+HTAH37l8e9IltbH1eUzgoKyOh0uNULLnxbpDTnxykTOCychgiHmPS0ggpetagHLxi0jOLGFe4FxOfbCdY=,iv:qMOCu7AtT7HnnQn22L3VsJ7JY/Wb5C0RLEfdl8g5OG4=,tag:PAb7sKHQZB65TBPts71s2g==,type:str]",
"pgp": [
{
"created_at": "2022-09-15T04:38:09Z",
"enc": "-----BEGIN PGP MESSAGE-----\n\nhQIMAxuQA5jWgsG6AQ/9HE0Xk8zM/oLJOqx4SaxKFKNmncYvRWSdADlXC341qYUG\n353VeMIbfl1u8Mg3iZxoUvD4SrkAEl63DOunhATWTtY2phtjoVxpoFhxOnu170zw\nKawewc9qzzjcAG5Oq/EqZeo0ip81VdZdfk9h05rSvH7y1vJ8YLgt6/3t6ZYlLdKA\nOF5jkjEqSyY9iaSZiFxVarT2PLygaIeG3zp9+mWcUjIaR9dzGILXDfwhNooFYlMN\no8GoEoMZUQVLLb4oFmWvnP/dG92ksimxp9ba4L0YWcd+HUJNGIxx8a1w5njGduqI\nDI67rVH8t94CwD30WGJq6d4yeo99JkN/gx88GdgLzuNzdclrMVbtR2f+ifqW0+KX\nza0e/AMCaCFHOUVZ8OGPJPZbBAB11hS/cRb9+JA/+OVolB9eGu6YkGhhAPS2xYVD\n4WC6vaNBgRwbu0YUxEiMWng2KM2QVeZlnFDJzUN/WNDwnmP9vY3keOqLJAAkkNtp\nyGCzA2PG2LvrMEPxoFj1uEtR8vxVcs56vp9Q6JdtA1wh+w61Mt1mnFbY6/6fsFQK\nC40L0UpukNNGsl1rl6jvlJEUoyjqxG9eWPMX3VKyXIrwEFYcEQ2pvEnsTHLx9pZa\nfxN4oiEW7leMPYzS5bEbx2pUxSrp94Om6T1fVRtBhWBGb2QaWPZCYR2p2cQRW1zU\naAEJAhAV9CcPDhhrHpYn06QbJf4kvM1SNXT5vRkeeC/fxS7HkqM3ukNzHkgrUijz\nj5k9vB7prn+2fGkpNxYSLOOW0Zgn0INpEkaBoRpdcfvFeE+kVv+rmUL9jsPWJpOA\nygzUZnejMwOn\n=v3OP\n-----END PGP MESSAGE-----\n",
"fp": "A2B73FB4DA0891B38EECD35B47991CD146C9C4BC"
}
],
"unencrypted_suffix": "_unencrypted",
"version": "3.7.3"
}
}
According to the output, we can deduce:
- The whole file is encrypted (see the
data
section from the output above). - The encryption method (GPG, key fingerprint) is stored in the same file, as part of the file.
To decrypt the file and read the encrypted content in the file, run:
sops a-text-file.txt
And we get our original content back!
Note that when reading the encrypted file, we do not need to specify which encryption/decryption method to use since when we created the encrypted file in the first place, the encryption/decryption method was already stored as part of the file by SOPS.
2.5 SOPS with Structured Data
Encrypting the full content of a text file automatically is a significant advantage of SOPS, but if the file contains structured data, for example:
- YAML
- JSON
- *.ini
- ENV
SOPS can do even better!
Let's give it a try:
sops a-yaml-file.yaml
And let's put some key-value content:
username: tiexin
password: test
If we open the file with a regular text editor, say, we cat
it:
$ cat a-yaml-file.yaml
username: ENC[AES256_GCM,data:h/BVd2tf,iv:IjzYAQQErVeWhwIIuMMfq/pjFr9YJVDNSl6ceRPv+6Q=,tag:2+xJOR89rsOMIQMWHNSEqw==,type:str]
password: ENC[AES256_GCM,data:zaz1Jw==,iv:VZ81YN4FRQf14g4olKwb6A8W00BIFT/OgWSEqkrO29s=,tag:tRJHVU7ANvGSw0tOjqGKiQ==,type:str]
...(omitted)
We can see that that only the values part are encrypted, not the keys!
This is a huge advantage because if we open this file with a regular text editor, we can still get the structure of the data without showing the encrypted content.
I think you know why I'm excited about this feature because I just thought of a great application for it: encrypting Kubernetes secrets YAML files! After encryption, we can still clearly see the structure and keys, but not the values, making it safe to store the file somewhere and share it with others. And I'm not the first who thought about it, because for example, in the GitOps tool Flux CD, it's possible to manage Kubernetes secrets with SOPS, read more about it here.
It's worth mentioning that for YAML files, SOPS also encrypts comments, which is another great feature, because chances are, sensitive value fields might have some sensitive descriptions in the comment sections, too.
OK, so far, we've demonstrated the basic features of SOPS with both standard text files and structured files. But we've done everything with PGP keys, which probably isn't widely used in a production environment. Next, let's try SOPS with HashiCorp Vault and AWS KMS, two of the most famous secrets/key managers.
3. SOPS with HashiCorp Vault
3.1 A Short Introduction to Vault
If you don't know HashiCorp Vault, at its core it is a secrets manager, but it can do much more, including encryption management. Typical use cases include secrets management, encryption services, key management, etc. It's an open-source project you can run in your infrastructure; HashiCorp also provides a cloud-based service.
3.2 Set up a Local Vault for Testing
For the SOPS tutorial, we will set up a local Vault service for testing, and the quickest way to do so is by using Docker:
docker run -d -p8200:8200 vault:1.2.0 server -dev -dev-root-token-id=toor
Please note that this is only to set up a local development Vault server; do not do this in a production environment. For production usage, check out the official docs.
After running Vault locally in a Docker container, run the following commands to validate:
$ export VAULT_ADDR=http://127.0.0.1:8200
$ export VAULT_TOKEN=toor
$ vault status
Key Value
--- -----
Seal Type shamir
Initialized true
Sealed false
Total Shares 1
Threshold 1
Version 1.2.0
Cluster Name vault-cluster-618cc902
Cluster ID e532e461-e8f0-1352-8a41-fc7c11096908
HA Enabled false
Once Vault is up and running, let's create a key to be used by SOPS:
$ # enable a transit engine if not already done
$ # It is suggested to create a transit engine specifically for SOPS
$ vault secrets enable -path=sops transit
Success! Enabled the transit secrets engine at: sops/
$ # create a key
$ vault write sops/keys/firstkey type=rsa-4096
Success! Data written to: sops/keys/firstkey
3.3 Using a Key from Vault with SOPS
Next, let's edit a file using the key we created in Vault:
sops --hc-vault-transit $VAULT_ADDR/v1/sops/keys/firstkey another-yaml-file.yaml
Write some key/value pairs in YAML format, and save.
If we open the file with another text editor (vim, for example), we can see the Vault-related information is also saved as part of the file. So, when we need to read it, we just run:
sops another-yaml-file.yaml
And that's it: the process is precisely the same as working with a PGP key.
SOPS supports fine-scoped rules in its config file, so it knows what encryption method and which key to use for each file based on these user-defined rules. Read more about this feature in the following sections.
4. SOPS with AWS KMS
4.1 A Short Introduction to KMS
AWS KMS (Key Management Service) is a cloud-based service that helps create and control cryptographic keys. Similarly to HashiCorp Vault, it can create, store and use encryption keys; the difference is that HashiCorp Vault is also a secrets manager, while AWS has a separate service (AWS Secrets Manager) for rotating, managing, and serving secrets.
4.2 Create an AWS KMS Key
For this part, we will use the AWS CLI.
If you haven't installed it yet, please follow the official doc here. After the installation, we need to configure it properly before using it.
Here we assume that your AWS CLI can already work correctly and has the required permission to access AWS KMS.
Creating a key in AWS KMS using the AWS CLI is surprisingly easy:
aws kms create-key \
--tags TagKey=Purpose,TagValue=Test \
--description "Test key"
From the output, copy the ARN, and we can save it as an ENV var (replace the ARN with your value):
export ARN=arn:aws:kms:ap-southeast-1:858104165444:key/620abad4-8043-4166-8712-2b4684378d3a
4.3 Using a Key from AWS KMS with SOPS
Also simple and straightforward:
sops --kms $ARN aws-kms-encryption-example.yaml
Similarly, the encryption method information is saved as part of the file, and reading the file with SOPS doesn't require specifying the decryption method, just like using it with GPG or Vault.
4.4 Using SOPS with AWS KMS Encryption Context
AWS KMS encryption context is a set of non-secret key–value pairs containing, as the name suggests, additional contextual information about the data, and it can be used in KMS key policies to define roles that can only access a given context.
When creating a new file with SOPS, we can set the --encryption-context
flag by comma-separated list of key-value pairs. For example:
$ sops edit --encryption-context Environment:production,AppName:ExampleApp test.dev.yaml
Similar to the fact that the key used for encryption is stored in the file as well so that when decrypting it you don't have to specify the key, the encryption context are also stored in the file (as metadata), and does not need to be provided at decryption.
In KMS key policies, we can use the encryption contexts to define roles. For example, the following key policy statement allows the RoleForExampleApp
role to use the KMS key in decryptions. It uses the encryption context condition key to allow this permission only when the encryption context in the request includes AppName:ExampleApp
:
{
"Effect": "Allow",
"Principal": {
"AWS": "arn:aws:iam::111122223333:role/RoleForExampleApp"
},
"Action": "kms:Decrypt",
"Resource": "*",
"Condition": {
"StringEquals": {
"kms:EncryptionContext:AppName": "ExampleApp"
}
}
}
Note: If you are not an AWS user, don't worry: SOPS supports more encryption methods, like GCP KMS, Azure Key Vault, etc. Discover more from SOPS's repo!
5. Other Useful Features of SOPS
5.1 Using .sops.yaml
Config to Select Different KMS Services for New Files
As we mentinoed in previous sections, SOPS supports fine-scoped rules in its config file, so that it knows what encryption method and which key to use for each file automatically without having to specify parameters when creating new files.
For example, we want to encrypt *.dev.yaml
using one set of KMS, but for *.prod.yaml
we want to use a different set of KMS, then finally we want to have a default PGP key for everything else, the following config should do the trick:
# evaluated sequentially
creation_rules:
- path_regex: \.dev\.yaml$
kms: "arn:aws:kms:us-east-1:111111111111:key/xxx-xxx-xxx"
- path_regex: \.prod\.yaml$
kms: "arn:aws:kms:us-east-1:111111111111:key/yyy-yyy-yyy"
# No path_regex, match everything as a default "catchall" clause.
- pgp: "A2B73FB4DA0891B38EECD35B47991CD146C9C4BC"
5.2 Key Groups
When you encrypt your data, your data is protected, but you have to protect your encryption key. One strategy is to encrypt it, this is called "envelope encryption" where we encrypt the "data key" with another "master key":
For more information, read the envelope encryption section in AWS KMS, and this blog about AWS KMS and HashiCorp Vault.
By default, you can define multiple master keys (for example a good practice is to have two keys in two different regions). When SOPS encrypts the data key for a file with each of the master keys, so that if any one of the master keys is available, the file can be decrypted.
However, it is sometimes desirable to require access to multiple master keys in order to decrypt files. This can be achieved with the "key groups" feature, so that the data key is split into fragments, each part encrypted with a different master key, and multiple master keys from multiple groups are required to decrypt a file.
Fun fact: This is achieved with Shamir's secret sharing algorithm, if you are interested, you can give it a thorough read.
For example, if we define key groups in the .sops.yaml
config file as following:
creation_rules:
- path_regex: .*keygroups.*
key_groups:
# First key group
- pgp:
- fingerprint1
- fingerprint2
kms:
- arn: arn1
role: role1
context:
foo: bar
- arn: arn2
aws_profile: myprofile
# Second key group
- pgp:
- fingerprint3
- fingerprint4
kms:
- arn: arn3
- arn: arn4
# Third key group
- pgp:
- fingerprint5
With this config, we can create a new encrypted file like we normally do, and optionally provide the --shamir-secret-sharing-threshold
command line flag if we want to override the default threshold. SOPS will then split the data key into three parts (from the number of key groups) and encrypt each fragment with the master keys found in each group.
5.3 Auditing
At times, you might want to know who accessed which encyprted files in your setup. With SOPS, you can create audit logs that track actions. Once activated, SOPS will log details like the time, the user running SOPS, and the decrypted file into a PostgreSQL database.
For example, to enable auditing to a PostgreSQL database named sops running on localhost, using the user sops and the password sops, /etc/sops/audit.yaml should have the following contents:
backends:
postgres:
- connection_string: "postgres://sops:sops@localhost/sops?sslmode=verify-full"
Of course, you must first create the database and credentialsRead more on the detailed steps of enabling auditing in the official doc here.
5.4 Using the "Publish" Feature
Although SOPS has already saved us tons of manual operations and time, we still need to save the file somewhere if we want to share it, for example, we might want to store it in an AWS S3 bucket to share with the team or maybe we want to store it in HashiCorp Vault. Wouldn't it be nice if the file created by SOPS can be uploaded automatically?
And luckily there is this feature: Simply run the command sops publish $file
, and the file will be published to a pre-configured destination (configured in .sops.yaml
, of course). For example, with the following configuration:
destination_rules:
- s3_bucket: "sops-secrets"
path_regex: s3/*
recreation_rule:
pgp: A2B73FB4DA0891B38EECD35B47991CD146C9C4BC
- vault_path: "sops/"
vault_kv_mount_name: "secret/"
vault_kv_version: 2
path_regex: vault/*
omit_extensions: true
It will put all files under s3/*
into the S3 bucket sops-secrets
, and the contents of all files under vault/*`` into Vault's KV store under the path
secrets/sops/``. For the files that will be published to S3, it will decrypt them and re-encrypt them using the PGP key defind in the recreation_rule
section, which works just like the creation_rules
mentioned in previous sections.
6. Summary
6.1 Why You Need SOPS
After following this whole tutorial and executing many commands yourself, you must be a bit tired. Now it's story time to get relaxed:
Years ago, I was working on a large-scale fin-tech-related project, where the center of the whole infrastructure was an OpenShift cluster (think of it as a customized Kubernetes).
In the cluster, we used namespaces to separate different environments; we used the OpenShift cluster as the single source of truth for secrets management.
For example: if we needed to update the value of a secret, we'd do it in the OpenShift console. If we needed to use a secret in Jenkins, we'd add a secret in OpenShift and it would be automatically synchronized as a Jenkins secret. All seemed fine.
If, however, we wanted to share a secret with another team member or migrate the secret into another cluster/environment, it got tricky: we'd have to export the secrets as K8s secret YAML file and then share the file.
This was a security and productivity problem: what if the YAML got shared with somebody who wasn't supposed to have access to that secret? What if the secret in the OpenShift cluster changed, but we didn't know since our YAML files at hand couldn't possibly sync with the cluster automatically?
In general, using K8s native secrets as the single source of truth isn't a best practice, but consuming secrets from outside a K8s environment isn't simple. Even if the secrets are only used within the cluster, you'd still have to create hardcoded YAML copies: where do you save those YAML files, and how do you share them safely?
This is where SOPS could help: it encrypts your sensitive files with minimum to no overhead, making it easy to store and share secret files.
6.2 Alternatives for Certain Use Cases
There are two similar tools less powerful than SOPS, but, depending on your specific use case, they can be equally valuable:
git-crypt
enables automated encryption and decryption of files in a git repo. Files you choose to protect are encrypted when committed and decrypted when checked out. git-crypt lets you freely share a repository containing a mix of public and private content. It's a great tool if you want to share secrets within your team, but it only supports PGP key encryption; you'd have to generate and manage those PGP keys yourself. And, if you only want to do some encryption/decryption automatically without using Git, it doesn't make sense.- Ansible Vault encrypts variables and files to protect sensitive content such as passwords or keys rather than leaving it visible as plaintext in Ansible playbooks or roles. It works similarly to SOPS, but apparently, it can only be used within the Ansible context. Of course, you can also use SOPS with Ansible.
SOPS is still king for generic usage, especially if you want different encryption methods for different files.
6.3 What SOPS Is Not
To be clear: SOPS is not a secrets manager. Yes, you can use it as a quick-and-cheap solution to store sensitive data in encrypted files. But, as your use cases expand, you are likely to hit a wall. Having secrets only in file format has its limitations. For example, you'll likely need API access to your secrets at some point. SOPS is not made for that, and building a custom solution on top of it would be clunky at best.
This is exactly what secrets managers are made for: flexibility in terms of formats, policies, and access methods for all kinds of your secrets (as files, K8s native YAML, from apps, by actual users, etc.). Modern secret managers have many integrations with CLIs, CI/CD systems, orchestration platforms, etc., so chances are, all your needs could be satisfied without encrypting files.
Recently, I discovered and tested a new kid on the block: Doppler, which can sync secrets to a K8s cluster automatically, eliminating the need to store K8s secrets in YAML files. It can even restart a pod when the dependent secret gets updated. Read more about it here if interested.
FAQ about SOPS
Is SOPS Secure?
When it comes to the security of the data encrypted with SOPS, think of it like this: it's as tough as the weakest cryptographic mechanism. The data gets wrapped up in AES256_GCM, which is like the Hulk of encryption methods right now. And to double down on that, our data keys are shielded either in KMS (yep, more AES256_GCM goodness) or in PGP, where we bring in RSA or ECDSA keys to keep things under lock and key.
What File Formats Can SOPS Be Used for?
YAML, JSON, ENV, INI and BINARY formats. Yes, you read it right, binary files are supported, although SOPS's primary use case is encrypting YAML and JSON configuration files.
What Key Management Services Does SOPS Support?
PGP Keys, age (a simple, modern, and secure tool for encrypting files), AWS KMS, GCP KMS, Azure Key Vault, Hashicorp Vault.
If you liked this tutorial, please follow me and stay tuned. See you in the next article!