Greetings! In my upcoming series of articles, I will delve into tutorials and cheat sheets on managing secrets effectively using secrets managers offered by the leading public cloud providers: AWS, GCP, and Azure.
To begin our journey, let's kick it off with Azure first.
But instead of getting hands-on with the practical aspects immediately, it's essential to establish a solid understanding of general data security principles. Let's get started.
1. Zero Trust
The fact that you probably have already heard about it shows how important it is nowadays.
TL;DR: Zero Trust is a security strategy.
I know the word "strategy" doesn't really say much, so let's get into it a bit more. Zero Trust isn't a product or a service, but rather, an approach, a framework, a model, or even a philosophy in system design, based on one simple principle: never trust, always verify.
Instead of unconditionally trusting that everything behind the corporate firewall is safe, the Zero Trust model assumes breach and verifies each request as though it originated from an uncontrolled party. Regardless of where the request originates or what resource it accesses, the Zero Trust model encourages us to "never trust, always verify".
To be more specific, there are three main pillars of zero trust:
- Verify explicitly: always authenticate and authorize based on all available data points.
- Least privilege access: limit user access with Just-In-Time (JIT) and Just-Enough-Access (JEA), risk-based adaptive policies, and data protection.
- Assume breach: minimize blast radius and segment access. Verify end-to-end encryption and use analytics to get visibility, drive threat detection, and improve defenses.
And these are the cores of Zero Trust.
2. Secure App and Data with Zero Trust
In a software system, two of the major components are applications and data. Apps provide the interface to consume data, and data is ultimately what security teams are protecting. So, achieving secure apps and secure data are two of the main goals of the Zero Trust mode.
There are several detailed aspects regarding apps and data:
- Secrets management: apps need access to databases and services using secrets and API keys, which, if stored in plaintext or even hardcoded in code, could increase the risk of unauthorized access in case of a security breach.
- Key management: crucial for protecting sensitive data by data encryption.
- Certificate management: since apps talk to each other, cert management is vital for securing network communications with SSL/TLS certificates.
In summary, secrets management, key management, and certificate management are essential components of a robust security infrastructure, enabling secure storage, controlled access, and encryption of sensitive information in cloud environments. And, they all need to follow the three principles of Zero Trust: verify explicitly, least privilege access, and assume breach. To achieve this goal, dedicated tools are required, and this is where Azure Key Vault comes in.
3. What is Azure Key Vault?
Microsoft Azure offers several options for storing and managing your keys in the cloud, and Key Vault is one of them. Key Vault is more than just a secrets manager; it does secrets management, key management, and certificate management. It's more of a key management solution in Azure.
For the secrets manager part, Azure Key Vault offers centralized storage for application secrets, reducing the risk of accidental leaks and eliminating the need to embed security information in application code.
There are a few reasons you might want to consider Key Vault:
- Proper authentication and authorization through Azure Active Directory and Azure RBAC.
- Keys and secrets can be software-protected or hardware-protected using HSMs for added assurance.
- Simple administration, high availability through replication.
- Key Vault access can be monitored by enabling logging.
- If you are already using other Azure services, Key Vault integrates well with them, like Azure Disk Encryption, SQL server, and Azure App Service.
Today, let's focus on the secrets management part only, and do a hands-on tutorial.
4. How Azure Key Vault Works?
Key Vault provides secure storage of generic secrets, such as passwords and database connection strings.
All secrets in Key Vault are stored encrypted. Key Vault encrypts secrets at rest with a hierarchy of encryption keys, with all keys in that hierarchy protected by modules that are FIPS 140-2 compliant. The encryption is transparent and requires no action from the user. The Azure Key Vault service encrypts your secrets when you add them, and decrypts them automatically when you read them.
Key Vault accepts data, encrypts it, stores and manages secrets as sequences of octets (8-bit bytes), and returns a secret identifier (id). The identifier can be used to retrieve the secret at a later time.
Access Control for secrets managed in Key Vault is provided at the level of the Key Vault that contains those secrets. The access control policy for secrets is distinct from the access control policy for keys in the same Key Vault. Users may create one or more vaults to hold secrets and are required to maintain scenario-appropriate segmentation and management of secrets.
5. How to create a Key Vault using Azure CLI?
The easiest way to use Azure CLI is via the Azure Cloud Shell. Simply go to the Azure portal, click the "Cloud Shell" button in the top-right corner of the navigation bar, and choose "Bash".
5.1 Create a Resource Group
In Azure, a resource group is a logical container into which Azure resources are deployed and managed. We will create a single resource group and use it across this whole tutorial so that it's easy for you to visualize and manage all the resources created in this tutorial.
Run the following command in the cloud shell:
az group create --name "myResourceGroup" --location "EastUS"
5.2 Create a Key Vault
Note: each key vault must have a unique name. Replace the name ("tiexin-keyvault-test") with your own unique name in the following command examples.
Then, let's create a Key Vault using the "myResourceGroup" resource group:
az keyvault create --name "tiexin-keyvault-test" --resource-group "myResourceGroup" --location "EastUS"
5.3 Create a Secret in the Key Vault
Run the following command to create a plain-text secret named "ExampleSecret" with the value "MyAKSExampleSecret" in the Key Vault named "tiexin-keyvault-test":
az keyvault secret set --vault-name "tiexin-keyvault-test" -n ExampleSecret --value MyAKSExampleSecret
Please take note of the Azure tenant ID that the subscription belongs to, which will be used later.
We can verify the secret is created successfully by reading it:
az keyvault secret show --vault-name "tiexin-keyvault-test" -n ExampleSecret
The az keyvault secret show command gets a specified secret from a given key vault.
6. Access Secrets from Virtual Machines
In this section, let's access secrets in Key Vault from a virtual machine.
6.1 Create a VM and Assign Permissions
Now, let's use the az vm create command to create a virtual machine, add a user account named "azureuser", generate an SSH key automatically, and put it in the default key location (~/.ssh):
az vm create \
--resource-group myResourceGroup \
--name myVM \
--image UbuntuLTS \
--admin-username azureuser \
--generate-ssh-keys
Take note of the public IP of the VM.
Please note that this is only a tutorial for demo purposes; do not do this (exposing VM to the public internet) in a production environment without knowing the consequences.
The VM needs an identity, and permissions can be assigned to the identity so that the VM can access Key Vault using those permissions defined in the identity.
Create a system-assigned identity for the VM
az vm identity assign --name "myVM" --resource-group "myResourceGroup"
Copy the "systemAssignedIdentity" value from the output, and assign the previously created identity permissions to the key vault:
az keyvault set-policy --name "tiexin-keyvault-test" --object-id "551c51fd-f86a-41c8-ac81-d47e4f09c4af" --secret-permissions get list
6.2 Create a Sample App to Read the Secret
Login to the VM (replace the public IP address with your own value):
ssh azureuser@172.190.235.224
On the virtual machine, let's write a simple Python app to read the secret. First, install the two Python libraries we'll be using:
sudo apt update && sudo apt upgrade -y
sudo apt install python3-pip
python3 -m pip install --upgrade pip
pip3 install azure-keyvault-secrets
pip3 install azure.identity
Create the Python app main.py, and remember to replace the Key Vault name with your own value:
from azure.keyvault.secrets import SecretClient
from azure.identity import DefaultAzureCredential
key_vault_name = "tiexin-keyvault-test"
key_vault_uri = f"https://tiexin-keyvault-test.vault.azure.net"
secret_name = "ExampleSecret"
credential = DefaultAzureCredential()
client = SecretClient(vault_url=key_vault_uri, credential=credential)
retrieved_secret = client.get_secret(secret_name)
print(f"The value of secret '{secret_name}' in '{key_vault_name}' is: '{retrieved_secret.value}'")
Lastly, run main.py. If all has gone well, it should return the value of the secret:
python3 main.py
The value of secret 'ExampleSecret' in 'tiexin-keyvault-test' is: 'MyAKSExampleSecret'
7. How to Access Secrets from AKS?
In the previous section, we accessed the secret from a VM successfully. In the cloud-native era, we are likely running our workload in containerized applications, in, say, Azure AKS clusters. So, let's try to access the same secret, but this time, from an AKS cluster.
7.1 Create an AKS Cluster
Create an AKS cluster with Azure Key Vault Provider for Secrets Store CSI Driver capability:
az aks create -n myAKSCluster -g myResourceGroup --node-count 1 --enable-addons azure-keyvault-secrets-provider
An identity azureKeyvaultSecretsProvider is created by the add-on to access Azure resources. Take note of the identity's clientId in the output, which will be used to access the Key Vault.
Verify the installation is finished:
kubectl get pods -n kube-system -l 'app in (secrets-store-csi-driver,secrets-store-provider-azure)'
Then, set the policy to access secrets in your key vault
az keyvault set-policy -n tiexin-keyvault-test --secret-permissions get --spn a87c1e0f-ca8e-4064-abbf-b2749c333d65
7.2 Create the SecretProviderClass
Next, we will use Secrets Store CSI Driver for Kubernetes secrets to access secrets from Key Vault. CSI driver integrates secrets stores with Kubernetes via a Container Storage Interface (CSI) volume. For more information, read the official doc here.
Let's create a SecretProviderClass using the following YAML; make sure to use your own values for userAssignedIdentityID, keyvaultName, tenantId, and the objects to retrieve from your key vault:
File spc.yaml:
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: azure-kvname-user-msi
spec:
provider: azure
parameters:
usePodIdentity: "false"
useVMManagedIdentity: "true"
userAssignedIdentityID: a87c1e0f-ca8e-4064-abbf-b2749c333d65
keyvaultName: tiexin-keyvault-test
cloudName: ""
objects: |
array:
- |
objectName: ExampleSecret
objectType: secret
objectVersion: ""
tenantId: 3199f464-7cbd-4c6d-b60f-f87b69ac72a4
Apply it:
kubectl apply -f spc.yaml
Then create a test pod that uses the SPC to mount the secret as a file:
File pod.yaml:
kind: Pod
apiVersion: v1
metadata:
name: busybox-secrets-store-inline
spec:
containers:
- name: busybox
image: registry.k8s.io/e2e-test-images/busybox:1.29-1
command:
- "/bin/sleep"
- "10000"
volumeMounts:
- name: secrets-store01-inline
mountPath: "/mnt/secrets-store"
readOnly: true
volumes:
- name: secrets-store01-inline
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: "azure-kvname-user-msi"
Apply it:
kubectl apply -f pod.yaml
After the pod is up and running, we can verify that the secret is created and mounted as a file in the pod:
kubectl exec busybox-secrets-store-inline -- cat /mnt/secrets-store/ExampleSecret
MyAKSExampleSecret
8. Clean Up
To delete everything created in this tutorial, run:
az group delete --name "myResourceGroup"
Note that the Azure cloud shell, AKS, and network watcher will also create their resource groups; do delete those if you wish.
9. Cheat Sheet
Here's a cheat sheet with some examples for quick references:
- Create resource group:
az group create --name "myResourceGroup" --location "EastUS" - Create Key Vault:
az keyvault create --name "tiexin-keyvault-test" --resource-group "myResourceGroup" --location "EastUS" - Create secret:
az keyvault create --name "tiexin-keyvault-test" --resource-group "myResourceGroup" --location "EastUS" - Read secret:
az keyvault secret set --vault-name "tiexin-keyvault-test" -n ExampleSecret --value MyAKSExampleSecret
SecretProviderClass for Azure AKS example:
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
name: azure-kvname-user-msi
spec:
provider: azure
parameters:
usePodIdentity: "false"
useVMManagedIdentity: "true"
userAssignedIdentityID: a87c1e0f-ca8e-4064-abbf-b2749c333d65
keyvaultName: tiexin-keyvault-test
cloudName: ""
objects: |
array:
- |
objectName: ExampleSecret
objectType: secret
objectVersion: ""
tenantId: 3199f464-7cbd-4c6d-b60f-f87b69ac72a4
Using SPC as volumes/volumeMounts in a pod:
kind: Pod
apiVersion: v1
metadata:
name: busybox-secrets-store-inline
spec:
containers:
- name: busybox
image: registry.k8s.io/e2e-test-images/busybox:1.29-1
command:
- "/bin/sleep"
- "10000"
volumeMounts:
- name: secrets-store01-inline
mountPath: "/mnt/secrets-store"
readOnly: true
volumes:
- name: secrets-store01-inline
csi:
driver: secrets-store.csi.k8s.io
readOnly: true
volumeAttributes:
secretProviderClass: "azure-kvname-user-msi"
Summary
In this tutorial, we had a look at the Zero Trust security strategy, how to use it to secure applications and data, and how the Azure Key Vault can help to achieve the Zero Trust goal. Then we did two tutorials to demonstrate how to create secrets in Key Vault and access them in both virtual machines and Kubernetes clusters.
If you are mainly using AWS or GCP rather than Azure, don't worry; in the next two articles, we will cover GCP Secret Manager as well as AWS Secrets Manager. Please like, comment, and subscribe. See you in the next article!
Additional resources about managing secrets
Ziad Ghalleb
