👉
TL;DR: Learn how to securely manage Kubernetes secrets with best practices for creation, storage, encryption (at rest and in transit), RBAC, and auditing. Understand the risks of common anti-patterns, such as hardcoding secrets or overprivileged access. Explore the benefits of Kubernetes External Secrets and external vault integrations to enhance security, streamline secret rotation, and maintain compliance in complex, regulated environments.

This blog post covers creating, storing, and using secrets in Kubernetes, encryption, RBAC, and auditing. It introduces Kubernetes External Secrets and best practices to enhance security. Let's dive in!

portrait

Keshav Malik

Keshav is a full-time Security Engineer who loves to build and break stuff.
He is constantly looking for new and exciting technologies
and enjoys working with diverse technologies in his spare time.
He loves music and plays badminton whenever the opportunity presents itself.

Kubernetes has become the de facto standard for container orchestration, enabling organizations to build, deploy, and scale modern applications with efficiency and agility. As more organizations adopt Kubernetes, the need for proper security and management of sensitive data within these environments becomes paramount. One crucial aspect of ensuring a secure Kubernetes infrastructure is the effective management of secrets, such as API keys, passwords, and tokens.

In this blog post, we will delve into the world of secret management in Kubernetes, exploring various approaches and best practices to keep your sensitive data safe and accessible only to authorized components.

We will discuss how to create, store, and use secrets in Kubernetes, along with the importance of encryption, role-based access control, and auditing. By understanding and implementing these practices, you can significantly enhance the security and reliability of your Kubernetes cluster, safeguarding your applications and data from unauthorized access and potential breaches.

What are Secrets in Kubernetes (K8s)?

In the context of K8s, secrets are objects that store sensitive information, such as passwords, API keys, OAuth tokens, and Secure Shell (SSH) keys. These secrets are crucial for the proper functioning of applications and services within a Kubernetes cluster, as they often grant access to essential resources and authentication mechanisms.

Understanding and managing secrets is vital for maintaining the security and integrity of your Kubernetes environment. Inadequate management of secrets can lead to unauthorized access, data breaches, and compromised infrastructure. Therefore, it is essential to ensure that secrets are stored, transmitted, and utilized securely within your cluster.

Kubernetes Secret Types and Their Security Implications

Understanding the different types of Kubernetes secrets is crucial for implementing proper security controls. Kubernetes provides several built-in secret types, each designed for specific use cases with varying security implications. The most common types include Opaque secrets for arbitrary user-defined data, kubernetes.io/tls for TLS certificates and keys, kubernetes.io/dockerconfigjson for container registry authentication, and kubernetes.io/service-account-token for ServiceAccount credentials.

Each secret type has specific validation requirements and security considerations. For instance, TLS secrets must contain valid certificate and key pairs, while ServiceAccount token secrets should be avoided in favor of short-lived tokens obtained through the TokenRequest API. When selecting a secret type, consider the principle of least privilege—use the most restrictive type that meets your requirements. Custom secret types can be defined using domain-prefixed naming conventions, but they require careful security evaluation to ensure they don't introduce vulnerabilities into your cluster's secret management strategy.

Different Ways to Manage Secrets in K8s

A variety of methods are available to manage secrets in Kubernetes. In this section, we will focus on Kubernetes Secrets, a native solution for storing and managing sensitive data within your cluster.

Kubernetes Secrets

Kubernetes Secrets are built-in objects that enable you to store and manage sensitive information securely. They provide a convenient way to distribute confidential data to containers and maintain the separation of concerns between the application and its credentials.

Creating Kubernetes Secrets

There are several ways to create Kubernetes Secrets:

Using kubectl

You can create a secret using the kubectl command-line tool by providing the required data as key-value pairs. For example:

kubectl create secret generic my-secret --from-literal=username=myuser --from-literal=password=mypassword

This command creates a secret named my-secret containing two key-value pairs for the username and password.

Using a configuration file

Another way to create a Kubernetes Secret is through a YAML configuration file. First, encode your sensitive data in base64 format, then create a YAML file with the following structure:

apiVersion: v1
kind: Secret
metadata:
  name: my-secret
data:
  username: bXl1c2Vy
  password: bXlwYXNzd29yZA==

Apply the YAML file using kubectl apply -f my-secret.yaml

Using the Kustomize tool

Kustomize is a standalone tool for customizing Kubernetes objects through a kustomization.yaml file. You can create a secret generator with it like this:

apiVersion: kustomize.config.k8s.io/v1beta1
kind: Kustomization

generators:
- secret.yaml

secretGenerator:
- name: my-secret
  literals:
  - username=admin
  - password=secret

Then to generate the secret run: kubectl apply -k .

Ways to employ secrets within Pods

Secrets can be used within Pods in the following ways:

  • Mounting secrets as files in a volume for containers: You can mount a secret as a volume inside a container. Each key-value pair in secret will be presented as a separate file within the mounted volume.
  • Setting secrets as container environment variables: Secrets can be exposed as environment variables in containers. This approach allows your application to read the secret value without changing the code, making it more secure and manageable.
  • Leveraging kubelet for image pulling with secrets: Kubelet can use secrets to pull container images from private registries. By specifying the secret in the Pod's imagePullSecrets field, Kubernetes will authenticate to the registry using the provided credentials.


Also read: Kubernetes Hardening Tutorial Part 1: Pods

Accessing secrets in containers

Once a secret is mounted as a file or exposed as an environment variable, the application running in the container can access the secret data. If an attacker gains access to the container, they may be able to view the secret data and potentially use it to compromise other parts of the system. Therefore, it is important to ensure that only authorized containers and users can access these secrets.

This can be achieved in several ways, such as using RBAC (Role-Based Access Control) to restrict access to the secrets or encrypting the secrets at rest and in transit.

Also read: Kubernetes Hardening Tutorial Part 3: Authn, Authz, Logging & Auditing

Updating and rotating secrets

Regularly updating and rotating secrets is a crucial security practice. While you can update a secret using kubectl edit, this approach is not recommended because it can be error-prone and can potentially lead to unintended consequences.

When you edit a secret with kubectl edit, you are modifying the existing secret in place. This means that any existing references to the secret (such as environment variables or volume mounts) will continue to use the old secret data until the application is restarted or the pod is deleted and recreated.

If you need to rotate a secret, you would have to update any references to the old secret to use the new secret instead!

That's why past a certain size, it becomes more interesting to either implement app-specific mechanisms that reload configuration at runtime or deploy a sidecar container that monitors for changes and restarts the main container when necessary.

Limitations of Kubernetes Secrets

While Kubernetes Secrets provide a convenient way to manage sensitive data, they have some limitations:

  • Limited encryption: by default, secrets are stored unencrypted in etcd. K8s does support encryption, but the encrypting key needs separate management.
  • Limited rotation: K8s secrets are designed to be immutable, which means that they cannot be modified or versioned once they are created. This, as we have seen, makes it difficult to rotate secrets. They can't be audited either.
  • Limited access control: While Kubernetes provides RBAC (Role-Based Access Control) to control access to secrets, it is still possible for unauthorized users to gain access to secrets if they are able to compromise the cluster or the underlying infrastructure.
  • They may not be suitable for large-scale or highly regulated environments, where more advanced secret management solutions might be necessary.

Despite these limitations, Kubernetes Secrets remain a handy tool for managing secrets when you don't need to scale immediately your cluster.

Kubernetes Secrets Encryption at Rest and in Transit

Securing kubernetes secrets requires implementing encryption both at rest and in transit to protect sensitive data from unauthorized access. By default, Kubernetes stores secrets unencrypted in etcd, making them vulnerable to anyone with API server or etcd access. Enabling encryption at rest through Kubernetes' native encryption providers or external key management services like GitGuardian or HashiCorp Vault significantly enhances security posture.

For encryption in transit, Kubernetes uses TLS to secure communications between cluster components, but additional measures may be necessary for highly sensitive environments. Consider implementing envelope encryption, where data encryption keys are themselves encrypted by master keys stored in external key management systems. This approach provides defense-in-depth and enables centralized key rotation policies. When configuring encryption, ensure that encryption keys are properly managed, rotated regularly, and stored separately from the encrypted data. External secret management solutions can provide advanced encryption capabilities, including hardware security modules (HSMs) for cryptographic operations and automated key lifecycle management.

Kubernetes External Secrets

Kubernetes External Secrets offer an alternative approach to managing secrets in Kubernetes by integrating with external secret management solutions. This allows you to maintain sensitive data outside of your Kubernetes cluster while still providing seamless access to applications running within the cluster.

How does it work?

Kubernetes External Secrets are custom resources that act as a bridge between your Kubernetes cluster and external secret management systems.

Instead of storing secrets directly in Kubernetes, External Secrets fetch and synchronize secrets from external systems, making them available as native Kubernetes Secrets. This ensures that your applications can access sensitive data without any code changes while benefiting from the security features provided by external secret management solutions.

Integrating with external secret management solutions

Kubernetes External Secrets can integrate with a variety of external secret management solutions, such as HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, and Google Cloud Secret Manager.

To integrate External Secrets with your chosen secret management system, you need to deploy the corresponding External Secrets controller in your cluster and configure it to communicate with the external system.

For example, to integrate with HashiCorp Vault, you would deploy the Kubernetes External Secrets controller for Vault and configure it with the necessary Vault authentication and connection details.

Creating and using External Secrets in Kubernetes

To create an External Secret, you need to define a custom resource in a YAML file, specifying the reference to the secret stored in the external system:

apiVersion: external-secrets.io/v1alpha1
kind: ExternalSecret
metadata:
  name: my-external-secret
spec:
  backendType: vault
  data:
    - secretKey: username
      remoteRef:
        key: secret/data/my-secret
        property: username
    - secretKey: password
      remoteRef:
        key: secret/data/my-secret
        property: password

Apply the YAML file using kubectl apply -f my-external-secret.yaml

The External Secrets controller will fetch the secret data from the external system and create a native Kubernetes Secret with the same name. This generated secret can be used by your applications in the same way as regular Kubernetes Secrets.

Advantages of Kubernetes External Secrets

Using Kubernetes External Secrets offers several benefits:

  • Enhanced security by leveraging the features of external secret management solutions, such as encryption, access control, and auditing.
  • Reduced risk of exposing sensitive data within the Kubernetes cluster.
  • Simplified secret management for organizations already using external secret management systems.
  • Centralized secret management across multiple Kubernetes clusters and other platforms.

By integrating Kubernetes External Secrets with your chosen secret management solution, you can achieve a higher level of security and control over your sensitive data while maintaining compatibility with your existing Kubernetes applications.

Kubernetes Secret Management Anti-Patterns and Security Risks

Several common anti-patterns in Kubernetes secret management can expose organizations to significant security risks, especially when shortcuts undermine built-in protections. Key issues include:

  • Storing secrets directly in container images or configuration files, which makes them immutable, hard to rotate, and easy to leak through registries or shared artifacts.
  • Using environment variables for highly sensitive data, which can reveal secrets via process lists, debugging output, or container inspection tools.
  • Allowing overprivileged RBAC configurations that grant broad secret access across namespaces, violating least-privilege principles and increasing lateral-movement risk after compromise.
  • Hardcoding secrets in Helm charts or deployment manifests, which often end up in version control and create persistent, widely replicated exposures.

To reduce these risks, teams should add secret scanning to CI/CD pipelines, rely on external secret managers for high-sensitivity data, define clear secret lifecycle policies (creation, access, rotation, revocation), and conduct regular security audits to identify and remediate these patterns early.

Best practices for managing secrets in Kubernetes

To ensure the security and integrity of your sensitive data, it is crucial to follow best practices for secret management in Kubernetes. In this section, we will discuss some of the most important practices to keep your secrets secure and maintain a robust Kubernetes environment.

Role-based access control (RBAC)

RBAC is essential for managing secrets securely, as it enables you to control which users and components can create, read, update, or delete secrets. By implementing fine-grained access control, you can minimize the risk of unauthorized access and potential data breaches.

To implement RBAC for secrets management, you should create roles and role bindings that define the allowed actions on secrets for each user or group. For example, you can create a role that allows read-only access to secrets within a specific namespace and bind it to a specific user or group:

kind: Role
apiVersion: rbac.authorization.k8s.io/v1
metadata:
  namespace: my-namespace
  name: secret-reader
rules:
- apiGroups: [""]
  resources: ["secrets"]
  verbs: ["get", "watch", "list"]


Kubernetes secrets encryption

Encrypting secrets is crucial for protecting sensitive data from unauthorized access, both when stored in etcd (at rest) and when transmitted within the cluster (in transit).

Kubernetes provides native encryption options, such as enabling etcd encryption to protect secrets at rest and using TLS for securing communications within the cluster. Ensure these options are configured and enabled to maintain the confidentiality of your secrets.

In addition to Kubernetes native encryption options, you can also integrate third-party encryption solutions, such as HashiCorp Vault or cloud-based key management services, to further enhance the security of your secrets.

Secret rotation and expiration

Regularly rotating secrets is an essential security practice that minimizes the risk of unauthorized access and potential data breaches.

Strategies for secret rotation include manual updates using kubectl or automated rotation using custom controllers or third-party secret management solutions.

Automating secret rotation can be achieved using Kubernetes operators, external secret management systems, or custom scripts that periodically update secrets based on a predefined schedule or events.

Auditing and monitoring

Auditing and monitoring are crucial for maintaining the security and integrity of your secrets, as they enable you to track and analyze secret access, usage, and modifications and detect potential security incidents.

Several tools can be used for auditing and monitoring secrets, such as Kubernetes audit logs, Prometheus, Grafana, and complimentary solutions such as GitGuardian that provide detection against hard-coded secrets.

Configure alerts and notifications to proactively notify administrators of potential security incidents or irregular secret access patterns, enabling timely investigation and response to potential threats.

Conclusion

In this blog post, we have discussed the importance of secrets management in Kubernetes and explored various methods and best practices for securely managing sensitive data in your applications. Kubernetes Secrets, Kubernetes External Secrets, and GitOps workflows provide powerful tools and methodologies for managing secrets effectively, ensuring the security and integrity of your sensitive information.

We encourage you to follow the best practices outlined in this post, such as implementing RBAC, encrypting secrets at rest and in transit, rotating secrets regularly, and auditing and monitoring your secrets. Continuously exploring additional resources and tools can further enhance your secret management processes.

One such tool is GitGuardian, which helps you protect your sensitive data by monitoring your Git repositories for leaked secrets and credentials. GitGuardian can be integrated with your GitOps workflows to ensure that secrets are not accidentally exposed in your repositories, providing an additional layer of security for your Kubernetes environment.

In conclusion, managing secrets securely in Kubernetes is critical for maintaining the confidentiality and integrity of your sensitive data. By following best practices and utilizing the right tools and methodologies, you can create a robust and secure Kubernetes environment, safeguarding your applications and data from potential threats.

Read more articles from our "How to Handle Secrets" series:

How to Handle Secrets in Helm
Learn step-by-step techniques and best practices to handle secrets in Helm charts safely and effectively. Level up your Helm deployments today!
GitGuardian Blog - Code Security for the DevOps generationGuest Expert
4 Ways to Store & Manage Secrets in Docker
DevOps engineers must handle secrets with care. In this series, we summarize best practices for leveraging secrets with your everyday tools.
GitGuardian Blog - Code Security for the DevOps generationGuest Expert

FAQs

What are the main types of Kubernetes secrets and how do they impact security?

Kubernetes supports multiple secret types, including Opaque for arbitrary key–value pairs, kubernetes.io/tls for TLS certificates, kubernetes.io/dockerconfigjson for container registry credentials, and kubernetes.io/service-account-token for service account authentication. Each type has specific behaviors and constraints that affect validation and security posture. Prefer the most restrictive type possible and avoid long-lived service account tokens, replacing them with short-lived, rotated credentials to reduce exposure risk.

How can I ensure Kubernetes secrets are encrypted both at rest and in transit?

To encrypt secrets at rest, enable Kubernetes’ built-in encryption providers or integrate an external KMS such as HashiCorp Vault or a cloud provider KMS. For in-transit security, Kubernetes uses TLS by default, but highly sensitive workloads may benefit from envelope encryption and external key management. Always rotate encryption keys independently of application secrets and enforce strict access controls around key usage.

What are common anti-patterns in Kubernetes secret management that should be avoided?

Anti-patterns include storing secrets inside images or config files, overusing environment variables, granting broad RBAC permissions, and hardcoding secrets into manifests or Helm charts. These practices increase the risk of leakage through version control, logs, or container introspection. Instead, enforce least privilege, centralize secret storage, and routinely audit workloads for insecure patterns.

How does Kubernetes External Secrets improve secret management compared to native secrets?

Kubernetes External Secrets integrates Kubernetes with external secret managers such as AWS Secrets Manager, Google Secret Manager, or HashiCorp Vault. Secrets remain outside the cluster and are synced on demand, benefiting from advanced features like centralized audit logs, automated rotation, lifecycle tracking, and granular IAM controls—all while maintaining compatibility with Kubernetes workloads through native Secret objects.

What best practices should security teams follow for managing Kubernetes secrets at scale?

Use fine-grained RBAC, enforce encryption in transit and at rest, automate rotation, and implement expiration policies. Integrate external secret managers for sensitive workloads and add secret scanning to CI/CD pipelines to prevent leaks. Regularly review RBAC roles, audit secret access logs, and maintain standardized policies for secret lifecycle governance across teams and clusters.

How can I minimize the risk of secret exposure in Kubernetes environments?

Store secrets only in Kubernetes secrets or trusted external secret managers. Avoid embedding secrets in images or repositories, restrict access with RBAC, and reduce use of environment variables. Monitor for unauthorized secret access, use automated detectors to find hardcoded secrets, and enforce secret hygiene throughout development and deployment workflows.

How does secret rotation work in Kubernetes, and what are the challenges?

Secret rotation usually requires updating the Secret object and ensuring workloads reload the new values—often necessitating pod restarts. This can be complex at scale, especially for stateful or latency-sensitive applications. Sidecar-based reloaders, application-level reload hooks, and external secret managers with automated rotation and versioning help reduce operational friction and improve reliability.

This article is a guest post. Views and opinions expressed in this publication are solely those of the author and do not reflect the official policy, position, or views of GitGuardian, The content is provided for informational purposes, GitGuardian assumes no responsibility for any errors, omissions, or outcomes resulting from the use of this information. Should you have any enquiry with regard to the content, please contact the author directly.