How to Handle Secrets in Kubernetes

Kubernetes has become the de facto standard for container orchestration, enabling organizations to build, deploy, and scale modern applications with efficiency and agility. As more organizations adopt Kubernetes, the need for proper security and management of sensitive data within these environments becomes paramount. One crucial aspect of ensuring a secure Kubernetes infrastructure is the effective management of secrets, such as API keys, passwords, and tokens.

In this blog post, we will delve into the world of secret management in Kubernetes, exploring various approaches and best practices to keep your sensitive data safe and accessible only to authorized components.

We will discuss how to create, store, and use secrets in Kubernetes, along with the importance of encryption, role-based access control, and auditing. By understanding and implementing these practices, you can significantly enhance the security and reliability of your Kubernetes cluster, safeguarding your applications and data from unauthorized access and potential breaches.

What are Secrets in Kubernetes (K8s)?

In the context of K8s, secrets are objects that store sensitive information, such as passwords, API keys, OAuth tokens, and Secure Shell (SSH) keys. These secrets are crucial for the proper functioning of applications and services within a Kubernetes cluster, as they often grant access to essential resources and authentication mechanisms.

Understanding and managing secrets is vital for maintaining the security and integrity of your Kubernetes environment. Inadequate management of secrets can lead to unauthorized access, data breaches, and compromised infrastructure. Therefore, it is essential to ensure that secrets are stored, transmitted, and utilized securely within your cluster.

Different Ways to Manage Secrets in K8s

A variety of methods are available to manage secrets in Kubernetes. In this section, we will focus on Kubernetes Secrets, a native solution for storing and managing sensitive data within your cluster.

Kubernetes Secrets

Kubernetes Secrets are built-in objects that enable you to store and manage sensitive information securely. They provide a convenient way to distribute confidential data to containers and maintain the separation of concerns between the application and its credentials.

Creating Kubernetes Secrets

There are several ways to create Kubernetes Secrets:

Using kubectl

You can create a secret using the kubectl command-line tool by providing the required data as key-value pairs. For example:

kubectl create secret generic my-secret --from-literal=username=myuser --from-literal=password=mypassword

This command creates a secret named my-secret containing two key-value pairs for the username and password.

Using a configuration file

Another way to create a Kubernetes Secret is through a YAML configuration file. First, encode your sensitive data in base64 format, then create a YAML file with the following structure:

apiVersion: v1
kind: Secret
metadata:
  name: my-secret
data:
  username: bXl1c2Vy
  password: bXlwYXNzd29yZA==

Apply the YAML file using kubectl apply -f my-secret.yaml

Using the Kustomize tool

Kustomize is a standalone tool for customizing Kubernetes objects through a kustomization.yaml file. You can create a secret generator with it like this:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

generators:
- secret.yaml

secretGenerator:
- name: my-secret
  literals:
  - username=admin
  - password=secret

Then to generate the secret run: kubectl apply -k .

Ways to employ secrets within Pods

Secrets can be used within Pods in the following ways:

• Mounting secrets as files in a volume for containers: You can mount a secret as a volume inside a container. Each key-value pair in secret will be presented as a separate file within the mounted volume.
• Setting secrets as container environment variables: Secrets can be exposed as environment variables in containers. This approach allows your application to read the secret value without changing the code, making it more secure and manageable.
• Leveraging kubelet for image pulling with secrets: Kubelet can use secrets to pull container images from private registries. By specifying the secret in the Pod's imagePullSecrets field, Kubernetes will authenticate to the registry using the provided credentials.

Also read: Kubernetes Hardening Tutorial Part 1: Pods

Accessing secrets in containers

Once a secret is mounted as a file or exposed as an environment variable, the application running in the container can access the secret data. If an attacker gains access to the container, they may be able to view the secret data and potentially use it to compromise other parts of the system. Therefore, it is important to ensure that only authorized containers and users can access these secrets.

This can be achieved in several ways, such as using RBAC (Role-Based Access Control) to restrict access to the secrets or encrypting the secrets at rest and in transit.

Also read: Kubernetes Hardening Tutorial Part 3: Authn, Authz, Logging & Auditing

Updating and rotating secrets

Regularly updating and rotating secrets is a crucial security practice. While you can update a secret using kubectl edit, this approach is not recommended because it can be error-prone and can potentially lead to unintended consequences.

When you edit a secret with kubectl edit, you are modifying the existing secret in place. This means that any existing references to the secret (such as environment variables or volume mounts) will continue to use the old secret data until the application is restarted or the pod is deleted and recreated.

If you need to rotate a secret, you would have to update any references to the old secret to use the new secret instead!

That's why past a certain size, it becomes more interesting to either implement app-specific mechanisms that reload configuration at runtime or deploy a sidecar container that monitors for changes and restarts the main container when necessary.

Limitations of Kubernetes Secrets

While Kubernetes Secrets provide a convenient way to manage sensitive data, they have some limitations:

• Limited encryption: by default, secrets are stored unencrypted in etcd. K8s does support encryption, but the encrypting key needs separate management.
• Limited rotation: K8s secrets are designed to be immutable, which means that they cannot be modified or versioned once they are created. This, as we have seen, makes it difficult to rotate secrets. They can't be audited either.
• Limited access control: While Kubernetes provides RBAC (Role-Based Access Control) to control access to secrets, it is still possible for unauthorized users to gain access to secrets if they are able to compromise the cluster or the underlying infrastructure.
• They may not be suitable for large-scale or highly regulated environments, where more advanced secret management solutions might be necessary.

Despite these limitations, Kubernetes Secrets remain a handy tool for managing secrets when you don't need to scale immediately your cluster.

Kubernetes External Secrets

Kubernetes External Secrets offer an alternative approach to managing secrets in Kubernetes by integrating with external secret management solutions. This allows you to maintain sensitive data outside of your Kubernetes cluster while still providing seamless access to applications running within the cluster.

How does it work?

Kubernetes External Secrets are custom resources that act as a bridge between your Kubernetes cluster and external secret management systems.

Instead of storing secrets directly in Kubernetes, External Secrets fetch and synchronize secrets from external systems, making them available as native Kubernetes Secrets. This ensures that your applications can access sensitive data without any code changes while benefiting from the security features provided by external secret management solutions.

Integrating with external secret management solutions

Kubernetes External Secrets can integrate with a variety of external secret management solutions, such as HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, and Google Cloud Secret Manager.

To integrate External Secrets with your chosen secret management system, you need to deploy the corresponding External Secrets controller in your cluster and configure it to communicate with the external system.

For example, to integrate with HashiCorp Vault, you would deploy the Kubernetes External Secrets controller for Vault and configure it with the necessary Vault authentication and connection details.

Creating and using External Secrets in Kubernetes

To create an External Secret, you need to define a custom resource in a YAML file, specifying the reference to the secret stored in the external system:

apiVersion: external-secrets.io/v1alpha1
kind: ExternalSecret
metadata:
  name: my-external-secret
spec:
  backendType: vault
  data:
    - secretKey: username
      remoteRef:
        key: secret/data/my-secret
        property: username
    - secretKey: password
      remoteRef:
        key: secret/data/my-secret
        property: password

Apply the YAML file using kubectl apply -f my-external-secret.yaml

The External Secrets controller will fetch the secret data from the external system and create a native Kubernetes Secret with the same name. This generated secret can be used by your applications in the same way as regular Kubernetes Secrets.

Advantages of Kubernetes External Secrets

Using Kubernetes External Secrets offers several benefits:

• Enhanced security by leveraging the features of external secret management solutions, such as encryption, access control, and auditing.
• Reduced risk of exposing sensitive data within the Kubernetes cluster.
• Simplified secret management for organizations already using external secret management systems.
• Centralized secret management across multiple Kubernetes clusters and other platforms.

By integrating Kubernetes External Secrets with your chosen secret management solution, you can achieve a higher level of security and control over your sensitive data while maintaining compatibility with your existing Kubernetes applications.

Best practices for managing secrets in Kubernetes

To ensure the security and integrity of your sensitive data, it is crucial to follow best practices for secret management in Kubernetes. In this section, we will discuss some of the most important practices to keep your secrets secure and maintain a robust Kubernetes environment.

Role-based access control (RBAC)

RBAC is essential for managing secrets securely, as it enables you to control which users and components can create, read, update, or delete secrets. By implementing fine-grained access control, you can minimize the risk of unauthorized access and potential data breaches.

To implement RBAC for secrets management, you should create roles and role bindings that define the allowed actions on secrets for each user or group. For example, you can create a role that allows read-only access to secrets within a specific namespace and bind it to a specific user or group:

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: my-namespace
  name: secret-reader
rules:
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get", "watch", "list"]

Kubernetes secrets encryption

Encrypting secrets is crucial for protecting sensitive data from unauthorized access, both when stored in etcd (at rest) and when transmitted within the cluster (in transit).

Kubernetes provides native encryption options, such as enabling etcd encryption to protect secrets at rest and using TLS for securing communications within the cluster. Ensure these options are configured and enabled to maintain the confidentiality of your secrets.

In addition to Kubernetes native encryption options, you can also integrate third-party encryption solutions, such as HashiCorp Vault or cloud-based key management services, to further enhance the security of your secrets.

Secret rotation and expiration

Regularly rotating secrets is an essential security practice that minimizes the risk of unauthorized access and potential data breaches.

Strategies for secret rotation include manual updates using kubectl or automated rotation using custom controllers or third-party secret management solutions.

Automating secret rotation can be achieved using Kubernetes operators, external secret management systems, or custom scripts that periodically update secrets based on a predefined schedule or events.

Auditing and monitoring

Auditing and monitoring are crucial for maintaining the security and integrity of your secrets, as they enable you to track and analyze secret access, usage, and modifications and detect potential security incidents.

Several tools can be used for auditing and monitoring secrets, such as Kubernetes audit logs, Prometheus, Grafana, and complimentary solutions such as GitGuardian that provide detection against hard-coded secrets.

Configure alerts and notifications to proactively notify administrators of potential security incidents or irregular secret access patterns, enabling timely investigation and response to potential threats.

Conclusion

In this blog post, we have discussed the importance of secrets management in Kubernetes and explored various methods and best practices for securely managing sensitive data in your applications. Kubernetes Secrets, Kubernetes External Secrets, and GitOps workflows provide powerful tools and methodologies for managing secrets effectively, ensuring the security and integrity of your sensitive information.

We encourage you to follow the best practices outlined in this post, such as implementing RBAC, encrypting secrets at rest and in transit, rotating secrets regularly, and auditing and monitoring your secrets. Continuously exploring additional resources and tools can further enhance your secret management processes.

One such tool is GitGuardian, which helps you protect your sensitive data by monitoring your Git repositories for leaked secrets and credentials. GitGuardian can be integrated with your GitOps workflows to ensure that secrets are not accidentally exposed in your repositories, providing an additional layer of security for your Kubernetes environment.

In conclusion, managing secrets securely in Kubernetes is critical for maintaining the confidentiality and integrity of your sensitive data. By following best practices and utilizing the right tools and methodologies, you can create a robust and secure Kubernetes environment, safeguarding your applications and data from potential threats.

Read more articles from our "How to Handle Secrets" series: