Kubernetes (K8s), an open-source container orchestration system, has become the de-facto standard for running containerized workloads thanks to its scalability and resilience.

Although K8s has the capabilities to streamline deployment processes, the actual deployment of applications can be cumbersome, since deploying an app to a K8s cluster typically involves managing multiple K8s manifests (like Deployment, Service, ConfigMap, Secret, Ingress, etc.) in YAML format. This isn't ideal because it introduces additional operational overhead due to the increased number of files for one app. Moreover, it often leads to duplicated, copy-pasted sections of the same app across different environments, making it more susceptible to human errors.

Helm, a popular package manager for Kubernetes, is designed to solve these deployment issues and help us manage K8s apps.

Helm Charts and Helm Secrets

Helm provides a straightforward way to define, install, and upgrade apps on K8s clusters. It is based on reusable templates called Helm charts, which encapsulate all the necessary K8s manifests, configurations, and dependencies into one single package, making it a whole lot easier to consistently version-control, publish/share, and deploy apps.

Since most apps rely on configurations, Helm charts often rely on ConfigMaps and Secrets (for sensitive information) to pass values to the apps as environment variables, following the Twelve-Factor App methodology. However, handling secrets in Helm can be challenging due to security concerns and collaboration/access control reasons:

• Security concerns: Secrets contain sensitive information such as passwords, API keys, and database credentials. Managing secrets securely is crucial to protect sensitive data from unauthorized access. Helm must ensure that secrets are properly encrypted, stored, and transmitted.
• Collaboration/access control: Helm promotes collaboration among teams by sharing charts and configurations. However, if secrets are included in these shared charts, controlling access becomes challenging. Ensuring that only authorized individuals can access and modify secrets is crucial for maintaining security and compliance.

In this blog, we aim to provide comprehensive solutions to solve these challenges once and for all. Here's what you can expect:

• Hands-on Tutorials: We will guide you through practical tutorials that demonstrate the usage of specialized tools such as the helm-secrets plugin and the External Secrets Operator. These tutorials will equip you with the knowledge and skills to effectively manage secrets in Helm deployments.
• Integration with CI/CD: Helm secrets are also a critical aspect of CI/CD pipelines. We will explore the possibilities, pros, and cons of CI/CD integrations, making sure Helm secrets and security are handled not only in manual deployment but also in automated workflows.
• Secrets Rotation: Secrets rotation is a necessary security practice. We will introduce a tool that simplifies the redeployment of apps when secrets rotation occurs.
• Tool Comparison and FAQs: To assist you in selecting the right tool for your specific needs, we will list the pros and cons and a flowchart to help you decide. Additionally, we will address some frequently asked questions to further clarify any doubts you may have.

Without further ado, let's dive in and explore the solutions to revolutionize your secrets management within Helm charts.

Tutorial: The helm-secrets Plugin

As mentioned in the previous section, managing secrets for Helm charts can be challenging because Helm chart Secrets contain sensitive information, and it's difficult to control access between team members.

With that in mind, there are two (and maybe only two) approaches to managing secrets in Helm charts: either we store sensitive information in Helm charts encrypted, or we don't store them in the charts at all:

• Encrypt the Secrets values in the values file of the Helm charts. This way, we ensure the chart is safe to share and check into version control without worrying about leaking sensitive information. We still need to figure out a control mechanism to share access (decrypt the values) among team members.
• Do not store the Secrets values in the Helm charts at all. This way, the sensitive information isn't in the Helm charts at all, so it's completely secure. Then we need to figure out a way to actually deploy Secrets into K8s clusters.

A Quick Introduction to the helm-secrets Plugin

TL;DR: the helm-secrets plugin can work in both of the two above-mentioned methods:

• encrypting sensitive information in the Helm charts and decrypting the values on the fly.
• storing secrets elsewhere (like in a secrets manager) and injecting them into the Helm charts when Helm deployments happen.

To learn more about how to use major cloud providers' secrets manager services, read my blogs:

• Handling Secrets with AWS Secrets Manager
• How to Handle Secrets with Google Cloud Secret Manager
• How to Handle Secrets with Azure Key Vault

How helm-secrets works

OK, now the long version.

Helm-secrets is a Helm plugin that manages secrets.

What's a Helm plugin, then? Good question.

Helm plugins are extensions that add additional functionalities and capabilities to Helm. These can be new commands, features, or integrations. Plugins can be used to perform various tasks, such as managing secrets, encrypting values, validating charts, etc. To use a Helm plugin, we typically install it using the helm plugin install command, and then we can invoke the plugin's commands just like any other native Helm command.

With helm-secrets, the values can be stored encrypted in Helm charts. However, the plugin does not handle the encryption/decryption operations itself; it offloads and delegates the cryptographic work to another tool: SOPS

SOPS, short for Secrets OPerationS, is an open-source text file editor by Mozilla that encrypts/decrypts files automatically. With SOPS, when we write a file, SOPS automatically encrypts the file before saving it to the disk. For that, it uses the encryption key of our choice: this can be a PGP key, an AWS KMS key, or many others. To learn more about SOPS and its possible integrations with different encryption key service providers, read my other blog here:

A Comprehensive Guide to SOPS: Managing Your Secrets Like A Visionary, Not a Functionary

Have you heard about SOPS? If you have already been in a situation where you needed to share sensitive information with your teammates, this is for you.

GitGuardian Blog - Automated Secrets DetectionGuest Expert

helm-secrets can also work in a "cloud" mode, where secrets are not stored in the Helm chart, but in a cloud secret manager. We then simply refer to the path of the secret in the cloud in the file, and the secret is automatically injected upon invoking helm install.

Example 1: Helm-secrets with SOPS

In this example, we will store encrypted secrets in the Helm charts. Since this relies on SOPS, we first need to install it. The easiest way to install SOPS is via brew:

brew install sops

For other OS users, refer to the official GitHub repo of SOPS.

Then, let's configure SOPS to use PGP keys for encryption:

brew install gnupg

If you are using another OS, for example, Linux, you can use the corresponding package manager. Most likely, this would work:

sudo apt-get install gnupg

With GnuPG, Creating a key is as simple as the following (remember to put your name as the value of KEY_NAME):

export KEY_NAME="Tiexin Guo"
export KEY_COMMENT="test key for sops"

gpg --batch --full-generate-key <<EOF
%no-protection
Key-Type: 1
Key-Length: 4096
Subkey-Type: 1
Subkey-Length: 4096
Expire-Date: 0
Name-Comment: ${KEY_COMMENT}
Name-Real: ${KEY_NAME}
EOF

Get the GPG key fingerprint:

$ gpg --list-secret-keys "${KEY_NAME}"
gpg: checking the trustdb
gpg: marginals needed: 3  completes needed: 1  trust model: pgp
gpg: depth: 0  valid:   1  signed:   0  trust: 0-, 0q, 0n, 0m, 0f, 1u
sec   rsa4096 2023-10-17 [SCEAR]
      BE574406FE117762E9F4C8B01CB98A820DCBA0FC
uid           [ultimate] Tiexin Guo (test key for sops)
ssb   rsa4096 2023-10-17 [SEAR]

In the "pub" part of the output, you can get the GPG key fingerprint (in my case, it's "BE574406FE117762E9F4C8B01CB98A820DCBA0FC").

Then we need to configure SOPS to use this PGP key for encryption/decryption. To do so, create a file named .sops.yaml under your $HOME directory with the following content:

creation_rules:
    - pgp: >-
        BE574406FE117762E9F4C8B01CB98A820DCBA0FC

Remember to replace the key fingerprint generated in the previous step.

Installing and configuring SOPS with PGP keys is not simple; refer to my blog on SOPS for more details.

Finally, we can install helm-secrets. Click here to get the latest version (at the time of writing, the latest version is v4.5.1). Then, run the following command to install:

helm plugin install https://github.com/jkroepke/helm-secrets --version v4.5.1

Let's create a secret file and name it as credentials.yaml.dec with the following content:

password: test

To encrypt this file using helm-secrets, run the following command:

helm secrets encrypt credentials.yaml.dec > credentials.yaml

If you open the generated credentials.yaml file, you will see that its content is encrypted by SOPS.

Next, we can refer to the encrypted value in the Helm chart's Secrets. Suppose we have a file named your-chart/templates/secrets.yaml with the following content:

apiVersion: v1
kind: Secret
metadata:
  name: helloworld
  labels:
    app: helloworld
    chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
    release: "{{ .Release.Name }}"
    heritage: "{{ .Release.Service }}"
type: Opaque
data:
  password: {{ .Values.password | b64enc | quote }}

This template will use the value from the helm-secrets encrypted credentials.yaml. When installing the Helm chart, instead of using helm install, use helm secrets install:

helm secrets install release-name -f values.yaml -f credentials.yaml your-chart

Remember not to submit the credentials.yaml.dec file to git repositories as it contains clear text passwords. However, the SOPS encrypted result credentials.yaml can be submitted as part of the Helm chart (although it is generally recommended not to include secret values in Helm charts). If you choose to submit encrypted values, make sure to use a private chart repository for internal use only.

In this case, we can store sensitive information in an encrypted values file. To share the encrypted values file with your team members, you can add their public key fingerprints to the SOPS configuration. This way, access control is managed by SOPS rather than helm-secrets.

SOPS supports various encryption methods, such as using AWS KMS keys for encryption and sharing access through AWS IAM policies. For more encryption methods supported by SOPS, refer to my other blog.

For more details on using helm-secrets with SOPS, refer to the official doc here.

Example 2: helm-secrets with Cloud Secrets Managers

In the previous example, we discussed how to store sensitive information in an encrypted form within the values file. However, helm-secrets offers another mode that integrates with popular cloud secrets managers like Hashicorp Vault or AWS Secrets Manager.

To enable the integration with cloud secrets managers, you need to set the environment variable HELM_SECRETS_BACKEND=vals before running Helm. This will activate the vals integration in helm-secrets:

export HELM_SECRETS_BACKEND=vals

Vals is a tool specifically designed for managing secret values in Helm. It requires cloud provider credentials to fetch secrets from the secret services. Make sure you have the necessary credentials in place before attempting to use them.

Let's assume you have a file called secrets.yaml located at your-chart/templates/secrets.yaml. Here's an example of its content:

apiVersion: v1
kind: Secret
metadata:
  name: helloworld
  labels:
    app: helloworld
    chart: "{{ .Chart.Name }}-{{ .Chart.Version }}"
    release: "{{ .Release.Name }}"
    heritage: "{{ .Release.Service }}"
type: Opaque
data:
  password: '{{ .Values.password | b64enc }}'

In your values.yaml file, you can include the following snippet:

password: ref+awssecrets://path/to/my/secret/value

Finally, you can install everything together using the following command:

helm secrets install release-name -f values.yaml your-chart

This command will inject the secret value from AWS Secrets Manager, located at "path/to/my/secret/value", into the variable "password" defined in the values file.

Simplifying Continuous Deployment Integration with helm-secrets

If you're already using SOPS, then helm-secrets is a great choice for seamless integration. It also offers cloud integrations if you prefer not to store encrypted data in values files.

While helm-secrets can be integrated with major CD tools like Argo CD, there is some operational overhead involved. This is because both SOPS and the helm-secrets plugin are required by the CD tool, as shown in the previous examples.

For instance, to integrate Argo CD with helm-secrets, you need to ensure that the Argo CD server container has both SOPS and helm-secrets. This can be achieved by building a customized Docker image: more details can be found here.

Another option is to install SOPS or vals and helm-secret through an init container on the argocd-repo-server Deployment. This requires changing the initContainers args. More details can be found here.

However, both options have their drawbacks. Customizing the Docker image means maintaining an additional image while customizing initContainers commands results in a more complex values file for Argo CD, which can be challenging to maintain.

Is there a better or alternative way to manage Helm secrets? Let's continue exploring.

External Secrets Operator

A Quick Introduction to External Secrets Operator

The External Secrets Operator is a K8s operator that facilitates the integration of external secret management systems, such as AWS Secrets Manager, HashiCorp Vault, Google Secrets Manager, and Azure Key Vault, with K8s.

Simply put, this operator automatically retrieves secrets from these secrets managers using external APIs and injects them into Kubernetes Secrets.

Unlike helm-secrets which either stores encrypted data in the values file using another tool (SOPS) or refers to secrets stored in cloud secrets managers in the values file, the External Secrets Operator does not require including secrets.yaml as part of the Helm templates. It uses another custom resource ExternalSecret, which contains the reference to cloud secrets managers.

What does the custom resource do? Let's dive deeper to take a look under the hood of External Secrets Operator.

How External Secrets Operator Works

Here's an overview of how the External Secrets Operator works:

• SecretStore configuration: First, we define a SecretStore resource that specifies the connection details and authentication credentials for the external secret management system with which we want to integrate.
• ExternalSecret Configuration: Next, we create an ExternalSecret resource that defines the mapping between the external secrets and Kubernetes Secrets.
• Syncing secrets: The External Secrets Operator continuously monitors the ExternalSecret resources. When a new or updated ExternalSecret is detected, the operator retrieves the specified secrets from the external secret management system using the configured SecretStore.
• Automatic Synchronization: The External Secrets Operator periodically synchronizes the secrets based on a defined refresh interval.

External-Secrets Helm Chart Example

Let's see an example of using external secrets to manage Helm secrets.

The idea is simple: we do not include K8s Secrets as part of the Helm chart templates, but rather, we use ExternalSecret, which contains no sensitive information at all.

First, let's install the External Secret Operator itself:

helm repo add external-secrets https://charts.external-secrets.io
helm repo update
helm install external-secrets \
  external-secrets/external-secrets \
  -n external-secrets \
  --create-namespace

We need to make sure access to AWS Secrets Manager is granted to the external secret operator. For a quick test with AWS Secrets Manager, we can create a secret containing our AWS credentials (do not do this in production) with access to Secrets Manager. Execute the following commands:

echo -n 'KEYID' > ./access-key
echo -n 'SECRETKEY' > ./secret-access-key
kubectl create secret generic awssm-secret --from-file=./access-key --from-file=./secret-access-key

Make sure the access key has permission to access AWS Secrets Manager. For more information, check out AWS IAM policies.

This approach (access key as a K8s Secret) is only suitable for tutorials. For a production environment, it is recommended to use IAM roles for service accounts. Refer to the AWS official documentation and the External Secret Operator official documentation for more details.

Then, let's create a SecretStore pointing to AWS Secrets Manager in a specific account and region. Create a file named secretstore.yaml with the following content:

apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: secretstore-sample
spec:
  provider:
    aws:
      service: SecretsManager
      region: ap-southeast-1
      auth:
        secretRef:
          accessKeyIDSecretRef:
            name: awssm-secret
            key: access-key
          secretAccessKeySecretRef:
            name: awssm-secret
            key: secret-access-key

Apply the configuration by running the command:

kubectl apply -f secretstore.yaml

Make sure to update the region with the appropriate AWS Secrets Manager region where your secrets are stored.

Then we can create an ExternalSecret as part of a Helm chart template that synchronizes a secret from AWS Secrets Manager as a Kubernetes Secret. Create a file named your-chart/templates/externalsecret.yaml with the following content:

apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: example
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: secretstore-sample
    kind: SecretStore
  target:
    name: helloworld
    creationPolicy: Owner
  data:
  - secretKey: password
    remoteRef:
      key: MyTestSecret1
      property: password

The above ExternalSecret will create a K8s Secret named "helloworld" (specified in the "target" section) with one key "password", whose value is retrieved from MyTestSecret1.password in the AWS Secrets Manager.

Without changing anything else, we can simply install the chart by running:

helm install release-name -f values.yaml your-chart

After installation, we can verify the Secret is already created:

kubectl describe secret helloworld

In the output, you should see the details of the created K8s Secret, which is automatically synchronized by the External Secrets Operator, fetching values from AWS Secrets Manager.

We can now utilize envFrom and secretRef in the Helm chart's Deployment to pass these secret values as environment variables.

Seamless Integration of External Secrets Operator with Continuous Deployment Tools

Unlike helm-secrets, which necessitates the installation of plugins to Helm and additional command-line tools like SOPS, the External Secrets Operator offers a more streamlined approach. It does not require any modifications to Helm or local binaries. Instead, it is solely installed on the K8s cluster side, utilizing an operator that needs to be installed and a SecretStore custom resource that must be defined.

Due to this inherent simplicity, integrating the external secret operator with continuous deployment tools such as Argo CD is effortless and trouble-free. No changes need to be made on the continuous deployment tool side. The only modification required is to the Helm chart itself: Secrets should not be included in the Helm chart template; instead, ExternalSecret should be used in the templates.

Vault Secrets Operator for Kubernetes Secrets and Helm Secrets

There is another major choice regarding managing secrets for Helm charts: the Vault Secrets Operator.

The Vault Secrets Operator works more or less similarly to the External Secrets Operator, but since it's made by HashiCorp, it only works with HashiCorp Vault. It works by watching for changes in the vault and synchronizing from the vault to a K8s Secret. The operator writes the source secret data directly to the destination Kubernetes Secret, ensuring that any changes made to the source are replicated to the destination over its lifetime.

It's worth noting that the External Secrets Operator also works with HashiCorp Vault. Still, if you are already using HashiCorp Vault, maybe the vault secrets operator can be a better choice since it's specifically designed for HashiCorp Vault and provides additional features tailored to Vault's capabilities, like dynamic secrets.

How to Automatically Restart Pods When Secrets Are Updated

In this section, we will discuss how to automatically restart pods when secrets are updated. This is an important requirement for most teams and companies. We will explore different approaches to achieve this.

If you are using a standard Helm chart without helm-secrets or the External Secrets Operator, you can use a hack to ensure that a Deployment's annotation section is updated when the secrets.yaml file changes. This can be done by using the sha256sum function. Here's an example:

kind: Deployment
spec:
  template:
    metadata:
      annotations:
        checksum/config: {{ include (print $.Template.BasePath "/secrets.yaml") . | sha256sum }}
[...]

This would work, and based on my experience, a lot of teams and companies use this in production, but in my eyes, this isn't ideal. For starters, it looks a bit messy in the annotations section.

In the case of helm-secrets, if the values are updated, we must do another helm secrets upgrade, since the values are either part of the Helm chart or injected in run time. This could be a little bit redundant, because what if neither the app nor the chart is updated, and only a secret value is updated? A full-on Helm upgrade seems a bit much.

Using the External Secrets Operator with Helm is even trickier: there is no secrets.yaml anymore, and the ExternalSecret only refers to secrets managers in the cloud, meaning the externalsecret.yaml doesn't change even if the values are updated in secrets managers, so we can't even use the checksum function.

To address these challenges, we recommend using Reloader. Reloader can watch for changes in secrets and ConfigMaps, and perform rolling upgrades on pods associated with DeploymentConfigs, Deployments, Daemonsets, Statefulsets, and Rollouts.

To install Reloader, simply run:

helm repo add stakater https://stakater.github.io/stakater-charts
helm repo update
helm install stakater/reloader # For helm3 add --generate-name flag or set the release name

Once Reloader is installed, you can configure it to automatically discover Deployments where specific config maps or secrets are used. To do this, add the reloader.stakater.com/auto annotation to the main metadata of your Deployment as part of the Helm chart template:

kind: Deployment
metadata:
  annotations:
    reloader.stakater.com/auto: "true"
spec:
  template:
    metadata:
[...]

This will discover Deployments automatically where a configmap or a secret is being used, and it will perform rolling upgrades on related pods when either is updated. Combined with External Secrets Operator, everything is solved without untidy hacks like the checksum annotation!

For more detailed usage of Reloader, check out the official doc here.

Summary

Learning to use helm-secrets can be challenging due to its integration with SOPS and the various encryption options it offers. Additionally, integrating helm-secrets with CD tools can result in increased operational overhead. However, if you only need to securely store sensitive data in the values file and do not plan on using a cloud secrets manager, helm-secrets is a suitable choice.

The Vault Secrets Operator and External Secrets Operator function similarly, but the Vault Secrets Operator, designed specifically for HashiCorp Vault, offers additional features such as dynamic secret generation.

For most users, the External Secrets Operator is likely the better choice as it is compatible with major public cloud providers' secrets managers and seamlessly integrates with CD tools and the Reloader tool. This conclusion is supported by the higher number of GitHub stars for the External Secrets Operator (3k stars) compared to the Vault Secrets Operator (0.3k stars) and helm-secrets (1k stars).

Of course, your own experience may differ, depending on your preferences. To make things easier, we created a workflow helping you choose among these solutions:

F.A.Q.