How to Handle Secrets in Helm

Kubernetes (K8s), an open-source container orchestration system, has become the de-facto standard for running containerized workloads thanks to its scalability and resilience.

Although K8s has the capabilities to streamline deployment processes, the actual deployment of applications can be cumbersome, since deploying an app to a K8s cluster typically involves managing multiple K8s manifests (like Deployment, Service, ConfigMap, Secret, Ingress, etc.) in YAML format. This isn't ideal because it introduces additional operational overhead due to the increased number of files for one app. Moreover, it often leads to duplicated, copy-pasted sections of the same app across different environments, making it more susceptible to human errors.

Helm, a popular package manager for Kubernetes, is designed to solve these deployment issues and help us manage K8s apps.

1. Helm Charts and Helm Secrets

Helm provides a straightforward way to define, install, and upgrade apps on K8s clusters. It is based on templates called Helm charts, which encapsulate all the necessary K8s manifests and dependencies into a single package, making it much easier to consistently version-control, publish, share, and deploy apps.

Since most apps need configurations, Helm charts often rely on ConfigMaps and Secrets (for sensitive information) to pass values to the apps as environment variables, following the Twelve-Factor App methodology. However, handling secrets in Helm can be challenging due to security concerns and collaboration & access control reasons:

• Security concerns: Managing secrets securely is crucial to protect sensitive information from unauthorized access. Helm must ensure that secrets are properly encrypted, stored, and used.
• Collaboration & access control: Helm makes team collaboration easier because we can share charts and configurations. However, if secrets are included in these shared charts, controlling access becomes challenging, if possible at all. Ensuring only authorized individuals can access and modify secrets is crucial for security and compliance.

In this blog, we aim to provide comprehensive solutions to solve these challenges once and for all. Here's what you can expect:

• Hands-on Tutorials: We will walk you through practical guides that demonstrate using tools such as the helm-secrets plugin and the External Secrets Operator. These tutorials equip you with the knowledge and skills to effectively manage secrets in Helm deployments.
• Integration with CI/CD: Deploying manually using helm install with secrets stored in a local values file is easy, what's more challenging is to integrate Helm deployment with CI/CD pipelines while handling secrets securely. We will explore possibile solutions, pros and cons of each option, making sure Helm secrets and security are handled in automated workflows.
• Secrets Rotation: A mandatory security practice. We will introduce a tool that simplifies the redeployment of apps when secrets get rotated.
• Tools Comparison and FAQs: To assist you in selecting the right tool for your specific need, we will list the pros and cons and a flowchart to help you decide. Additionally, we will address some frequently asked questions to further clarify any doubts you might have.

Without further ado, let's dive in and explore the solutions to revolutionize your Helm chart secrets management.

2. Tutorial: The helm-secrets Plugin

As aforementioned: Managing secrets for Helm charts can be challenging because Helm chart Secrets contain sensitive information, and it's difficult to control access between team members.

With that in mind, from the perspective of whether to store sensitive information as part of the Helm charts, there are two (and maybe only two) types of approaches to managing secrets in Helm charts: either we store sensitive information in Helm charts encrypted, or we don't store them in the charts at all:

• Encrypt the Secrets values in the values file of the Helm charts. This way, we ensure the chart is safe to be shared and checked into version control systems without worrying about leaking sensitive information. For this method, we need to figure out a mechanism for encryption/decryption and sharing access among team members.
• Do not store the Secrets values in the Helm charts at all. This way, the sensitive information isn't in the Helm charts at all, so it's completely secure. Then, we need to figure out some methods to pass in those sensitive information, either to helm install commands, or as Secrets directly into K8s clusters.

2.1 A Quick Introduction to the helm-secrets Plugin

TL;DR: the helm-secrets plugin can work in both of the two above-mentioned approaches.

• Encrypting sensitive information in the Helm charts and decrypting the values on the fly.
• Or, storing secrets elsewhere (like in a secret manager) and injecting them into the Helm charts when Helm deployments happen.

To learn more about how to use major cloud providers' secret managers, read my blogs:

• Handling Secrets with AWS Secrets Manager
• How to Handle Secrets with Google Cloud Secret Manager
• How to Handle Secrets with Azure Key Vault

2.2 How helm-secrets Works

OK, now the long version.

helm-secrets is a Helm plugin that manages secrets.

What's a Helm plugin, then? Good question.

Helm plugins are extensions adding additional capabilities to Helm. Plugins can be used to perform various tasks, such as managing secrets, encrypting/decrypting, validating charts, etc.

To use a Helm plugin, we typically install it using the helm plugin install command, and then we can invoke the plugin's commands just like any other native Helm command.

With the helm-secrets plugin, Helm chart values can be stored encrypted. However, the plugin does not handle the encryption/decryption operations itself; it offloads the cryptographic work to another tool: SOPS.

SOPS, short for Secrets OPerationS, is an open-source file editor by Mozilla (donated to CNCF in 2023) that encrypts/decrypts files automatically. With SOPS, when we write a file, it automatically encrypts the file before saving it to the disk, using encryption keys of our choice: PGP key, AWS KMS key, or many others. To learn more about SOPS and its integrations with different encryption key service providers, read my blog here:

https://blog.gitguardian.com/a-comprehensive-guide-to-sops/

It's worth mentioning that besides with SOPS, helm-secrets can also work in a "cloud" mode, where secrets are not stored encrypted in the Helm chart, but in a cloud secret manager. We then simply refer to the path of the secret in the cloud, and the secret is automatically injected upon invoking helm install.

2.3 Tutorial: Helm-secrets with SOPS

In this tutorial, we will store encrypted secrets (with PGP keys) in the Helm charts. First, we need to install SOPS. For macOS users, the easiest way to install SOPS is via brew:

brew install sops

For other OS users, refer to the releases page of the official SOPS GitHub repo.

Then, let's configure SOPS to use PGP keys for encryption:

brew install gnupg

If you are using another OS, for example, Linux, we can use the corresponding package manager. Most likely, this would work:

sudo apt-get install gnupg

With GnuPG, Creating a key is as simple as the following (remember to put your name as the value of KEY_NAME):

export KEY_NAME="Tiexin Guo"
export KEY_COMMENT="test key for sops"

gpg --batch --full-generate-key <<EOF
%no-protection
Key-Type: 1
Key-Length: 4096
Subkey-Type: 1
Subkey-Length: 4096
Expire-Date: 0
Name-Comment: ${KEY_COMMENT}
Name-Real: ${KEY_NAME}
EOF

Get the GPG key fingerprint:

$ gpg --list-secret-keys "${KEY_NAME}"
gpg: checking the trustdb
gpg: marginals needed: 3  completes needed: 1  trust model: pgp
gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u
sec   rsa4096 2023-10-17 [SCEAR]
      BE574406FE117762E9F4C8B01CB98A820DCBA0FC
uid           [ultimate] Tiexin Guo (test key for sops)
ssb   rsa4096 2023-10-17 [SEAR]

In the "pub" part of the output, you can get the GPG key fingerprint (in my case, it's "BE574406FE117762E9F4C8B01CB98A820DCBA0FC").

Then we need to configure SOPS to use this PGP key for encryption/decryption. To do so, create a file named .sops.yaml under our $HOME directory with the following content:

creation_rules:
    - pgp: >-
        BE574406FE117762E9F4C8B01CB98A820DCBA0FC

Remember to replace the key fingerprint generated in the previous step.

Refer to my blog on SOPS for more details on installing and configuring SOPS (with PGP keys or other key management services).

Finally, we can install the helm-secrets plugin. Click here to get the latest version (at the time of writing, the latest version is v4.6.1). Then, run the following command to install:

helm plugin install https://github.com/jkroepke/helm-secrets --version v4.6.1

We can run helm plugin list to validate the installation, and you should get similar output to：

$ helm plugin list
NAME   	VERSION	DESCRIPTION
secrets	4.6.1  	This plugin provides secrets values encryption for Helm charts secure storing

Let's create a secret file and name it as credentials.yaml.dec with the following content:

password: test

To encrypt this file using helm-secrets, run the following command:

helm secrets encrypt credentials.yaml.dec > credentials.yaml

(If you open the generated credentials.yaml file, you will see that its content is encrypted by SOPS.)

Next, we can refer to the encrypted value in the Helm chart's Secrets. Suppose we have a file named your-chart/templates/secrets.yaml with the following content:

apiVersion: v1
kind: Secret
metadata:
  name: helloworld
  labels:
    app: helloworld
    chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
    release: "{{ .Release.Name }}"
    heritage: "{{ .Release.Service }}"
type: Opaque
data:
  password: {{ .Values.password | b64enc | quote }}

This template will use the value from the helm-secrets encrypted credentials.yaml. When installing the Helm chart, instead of using helm install, use helm secrets install:

helm secrets install release-name -f values.yaml -f credentials.yaml your-chart

Remember not to submit the credentials.yaml.dec file to git repositories as it contains clear text passwords. The SOPS encrypted result credentials.yaml, on the other hand, can be submitted as part of the Helm chart.

To share the encrypted values file with our teammates, we can add their public key fingerprints to the SOPS configuration. This way, access control is managed by SOPS rather than helm-secrets.

Using PGP keys for encryption isn't ideal in the real world because it adds overhead of managing those keys. As mentioned, SOPS supports various encryption methods, such as using AWS KMS keys for encryption and sharing access through KMS key policies. For more encryption methods supported by SOPS and advanced usage with AWS KMS and so on, refer to my blog on SOPS.

If you need more information on helm-secrets, refer to the official doc here.

2.4 Tutorial: helm-secrets with Cloud Secrets Managers

In the previous tutorial, we discussed how to store sensitive information in an encrypted form within the Helm values file. However, helm-secrets offers another mode that integrates with popular cloud secret managers like AWS Secrets Manager or Hashicorp Vault. To enable the integration with cloud secrets managers, we need to set the environment variable HELM_SECRETS_BACKEND=vals before running Helm. This will activate the vals integration in helm-secrets:

export HELM_SECRETS_BACKEND=vals

vals is a tool specifically designed for managing secret values in Helm. It requires cloud provider credentials to fetch secrets from the secret services; make sure you have the necessary credentials in place.

Let's assume we have a file named secrets.yaml located at your-chart/templates/secrets.yaml. Here's an example of its content:

apiVersion: v1
kind: Secret
metadata:
  name: helloworld
  labels:
    app: helloworld
    chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
    release: "{{ .Release.Name }}"
    heritage: "{{ .Release.Service }}"
type: Opaque
data:
  password: '{{ .Values.password | b64enc }}'

In our values.yaml file, we can include the following snippet:

password: ref+awssecrets://path/to/my/secret/value

This config above makes sure that when deploying the chart with the helm-secrets plugin, it injects the secret values from AWS Secrets Manager (as specified in the values file: ref+awssecrets), located at path/to/my/secret/value, into the variable password.

Then we can deploy everything together using the following command:

helm secrets install release-name -f values.yaml your-chart

2.5 Continuous Deployment with helm-secrets

If you're already using SOPS, then helm-secrets is a great choice for seamless integration. If you prefer not to store secrets data (encrypted) in Helm values files, helm-secrets also offers cloud integrations, as shown in the previous section.

However, while helm-secrets can be integrated with popular continuous deployment tools like Argo CD, there is some operational overhead involved because both SOPS and the helm-secrets plugin are required in the CD environment for the integration to work.

For instance, to integrate Argo CD with helm-secrets, we need to ensure that the Argo CD server has both SOPS and helm-secrets installed, which can be achieved by either building a customized Docker image or installing SOPS (or vals) and helm-secrets through an init container, which requires changing the initContainers args. I know this sound complicated, because it is; both options have their drawbacks: Customizing the Docker image means maintaining an additional image, and customizing initContainers commands results in a more complex values file for Argo CD itself.

At this point, one might ask: Is there an alternative, more importantly, easier way to manage Helm secrets?

Let's continue exploring.

3. External Secrets Operator

3.1 A Quick Introduction to External Secrets Operator

The External Secrets Operator is a K8s operator that integrates external secret managers with K8s. Simply put, this operator automatically retrieves secrets from secret managers like AWS Secrets Manager, HashiCorp Vault, Google Secrets Manager, Azure Key Vault, etc., using external APIs, and injects them into Kubernetes as Secrets.

Unlike helm-secrets (which either stores encrypted data or refers to secrets stored in cloud secret managers in the values file), External Secrets Operator doesn't include secrets.yaml in the Helm templates at all; it uses another custom resource: ExternalSecret, which references cloud secret managers.

3.2 How External Secrets Operator Works

Here's an overview of how the External Secrets Operator works under the hood:

• SecretStore: We first configure a SecretStore that specifies the details of the secret manager to be integrated.
• ExternalSecret: Next, we create an ExternalSecret resource that maps external secrets to Kubernetes Secrets.
• Syncing: External Secrets Operator continuously monitors the ExternalSecret resources.
• Synchronization: The External Secrets Operator periodically synchronizes the secrets based on a defined refresh interval.

3.3 External Secrets Helm Chart Tutorial

In this tutorial, let learn to use External Secrets Operator in Helm deployments.

The idea behind this is simple: Do not include K8s Secrets as part of the Helm chart templates, but rather, use ExternalSecret, which contains no sensitive information, only mappings.

First, let's install the External Secret Operator:

helm repo add external-secrets https://charts.external-secrets.io
helm repo update
helm install external-secrets \
  external-secrets/external-secrets \
  -n external-secrets \
  --create-namespace

We need to make sure that the access to AWS Secrets Manager is granted to the external secret operator. For a quick test with AWS Secrets Manager, we can create a secret containing our AWS credentials (DO NOT do this in production!) with access to Secrets Manager. Execute the following commands:

echo -n 'KEYID' > ./access-key
echo -n 'SECRETKEY' > ./secret-access-key
kubectl create secret generic awssm-secret --from-file=./access-key --from-file=./secret-access-key

Make sure the access key has permission to access AWS Secrets Manager. For more information, check out AWS IAM policies.

This approach (access key as a K8s Secret) is only suitable for tutorials. For a production environment, it is recommended to use IAM roles for service accounts. Refer to the AWS official documentation and the External Secret Operator official documentation for more details.

Then, let's create a SecretStore pointing to AWS Secrets Manager in a specific account and region. Create a file named secretstore.yaml with the following content:

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: secretstore-sample
spec:
  provider:
    aws:
      service: SecretsManager
      region: ap-southeast-1
      auth:
        secretRef:
          accessKeyIDSecretRef:
            name: awssm-secret
            key: access-key
          secretAccessKeySecretRef:
            name: awssm-secret
            key: secret-access-key

Apply the configuration by running the command:

kubectl apply -f secretstore.yaml

Make sure to update the region with the appropriate AWS Secrets Manager region where your secrets are stored.

Then we can create an ExternalSecret as part of a Helm chart template that synchronizes a secret from AWS Secrets Manager as a Kubernetes Secret. Create a file named your-chart/templates/externalsecret.yaml with the following content:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: example
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: secretstore-sample
    kind: SecretStore
  target:
    name: helloworld
    creationPolicy: Owner
  data:
  - secretKey: password
    remoteRef:
      key: MyTestSecret1
      property: password

The above ExternalSecret will create a K8s Secret named "helloworld" (specified in the "target" section) with one key "password", whose value is retrieved from MyTestSecret1.password in the AWS Secrets Manager.

Without changing anything else, we can simply install the chart by running:

helm install release-name -f values.yaml your-chart

After installation, we can verify the Secret is already created:

kubectl describe secret helloworld

In the output, you should see the details of the created K8s Secret, which is automatically synchronized by the External Secrets Operator, values fetched from AWS Secrets Manager.

At this point, we can now utilize envFrom and secretRef in the Helm chart's Deployment to pass these secret values as environment variables, just as if the secrets.yaml were included as part of the Helm chart.

3.4 Seamless Integration of External Secrets Operator with Continuous Deployment Tools

Unlike helm-secrets (which requires the installation of plugins to Helm and additional command-line tools like SOPS), the External Secrets Operator offers a more streamlined, hassle-free approach: It does not require any modifications to Helm or local binaries; instead, it is solely installed on the K8s cluster side (well, because it's an operator pattern).

Due to this inherent simplicity, integrating the external secret operator with continuous deployment tools such as Argo CD is effortless. No changes need to be made on the continuous deployment tool side, with the only modification on the Helm chart side: use ExternalSecret instead of Secrets in Helm chart template.

4. HashiCorp Vault Secrets Operator for Kubernetes Secrets and Helm Secrets

There is another major choice regarding managing secrets for Helm charts: the Vault Secrets Operator.

The Vault Secrets Operator works more or less in a similar way to the External Secrets Operator, but since it's made by HashiCorp, it only works with HashiCorp Vault. It watches for changes in the Vault and synchronizes from the vault to K8s secrets. The operator writes secret data from the source directly to K8s Secret, ensuring that any changes made to the source are replicated to the destination over its lifetime.

It's worth noting that the External Secrets Operator also works with HashiCorp Vault. Still, if you are already using HashiCorp Vault, maybe the Vault Secrets Operator can be a better choice since it's specifically designed for HashiCorp Vault and provides additional features tailored to Vault's capabilities, like dynamic secrets.

5. How to Automatically Restart Pods When Secrets Are Updated

Pod automatic restart upon secret change is a common and important requirement for most teams and companies, let's explore different options.

If you are using a standard Helm chart without helm-secrets or the External Secrets Operator, you can use a hack to ensure that a Deployment's annotation section is updated when the secrets.yaml file changes. This can be done by using the sha256sum function. Here's an example:

kind: Deployment
spec:
  template:
    metadata:
      annotations:
        checksum/config: {{ include (print $.Template.BasePath "/secrets.yaml") . | sha256sum }}
[...]

This would work, and based on my experience, a lot of teams and companies use this in production, but in my eyes, it's not ideal. I believe in the "Occam's razor" philosophy where the best solution is usually the simplest, but the above code is a long way from "the simplest".

If you are using the helm-secrets plugin, if the values are updated, we must do another helm chart upgrade, since the secrets values are not part of the Helm chart and are injected as fixed values in deploy time. This could be a little bit unnecessary, because what if neither the app nor the chart is updated, and only a secret value is updated? A full-on Helm upgrade seems not the simplest, again, violating the Occam's razor philosophy.

Things are not better with External Secrets Operator: there is no secrets.yaml in the charm at all, and the ExternalSecret only references paths in secret managers, meaning the externalsecret.yaml doesn't change even if the secrets values change, rendering the checksum method useless.

So, apparently, no solution so far is perfect.

To address these challenges, we recommend using Reloader. Reloader can watch for changes in Secrets and ConfigMaps, and perform rolling upgrades on Pods associated with DeploymentConfigs, Deployments, Daemonsets, Statefulsets, and Rollouts. Here's how to use it.

To install Reloader, simply run:

helm repo add stakater https://stakater.github.io/stakater-charts
helm repo update
helm install stakater/reloader # For helm3 add --generate-name flag or set the release name

Once Reloader is installed, we can configure it to automatically discover Deployments where specific ConfigMaps or Secrets are used. To do this, add the reloader.stakater.com/auto annotation to the main metadata of our Deployment as part of the Helm chart template:

kind: Deployment
metadata:
  annotations:
    reloader.stakater.com/auto: "true"
spec:
  template:
    metadata:
[...]

This will discover Deployments automatically where a ConfigMap or a Secret is used, and it will perform rolling upgrades on related pods when either is updated. Combine it with External Secrets Operator, everything is solved, without untidy hacks like the checksum annotation!

For more detailed usage of Reloader, check out the official doc here.

6. Summary

Learning to use helm-secrets can be challenging due to its integration with SOPS and the various encryption options SOPS offers. Plus, integrating helm-secrets with CD tools can result in increased overhead. However, if you only need to store sensitive data securely in Helm values file and do not plan on using a cloud secret manager, helm-secrets is a suitable choice.

The Vault Secrets Operator and External Secrets Operator share their similarities, but the Vault Secrets Operator, designed specifically for HashiCorp Vault, offers additional features such as dynamic secret generation. For most users, the External Secrets Operator is likely the better choice of these two, as it is compatible with major public cloud providers' secret managers and integrates well with continuous deployment tools (and the Reloader, for that matter). This conclusion is supported by the higher number of GitHub stars for the External Secrets Operator (3k stars) compared to the Vault Secrets Operator (0.3k stars) and helm-secrets (1k stars).

Of course, depending on your preferences and needs, your mileage might differ. To make things easier, we created a workflow helping you choose the best for your need:

F.A.Q.

When it comes to handling secrets, Helm treats them as part of its templates, just like other Kubernetes resources such as Deployments and ConfigMaps. These templates can reference values defined in the values.yaml file. However, since secrets contain sensitive information, it is bad practice to store clear-text credentials directly in the Helm values file. Instead, the values file should only contain sample default values.

There are two options for using Secrets with Helm Charts. The first is to encrypt the sensitive information in the Helm values file. The second option is to use an alternative mechanism to handle Kubernetes Secrets, rather than including them as part of the Helm chart.

helm-secrets provides a solution for encrypting sensitive values files using SOPS. This ensures that the encrypted files can be safely committed to git repositories and shared among team members. SOPS also offers access control capabilities and supports different encryption methods.

Additionally, helm-secrets provides the option to store sensitive data in cloud secrets managers instead of the values file. These values can then be injected during helm installs.

For those looking for an alternative to helm-secrets, the External Secrets Operator is a reliable choice. HashiCorp Vault users may also consider the Vault Secrets Operator.