Configuration management tools like Ansible, Chef, and Puppet been around for almost 20 years. They are starting to become more niche as workloads migrate to serverless and kubernetes deployments. However, there’s still a big need for config management to deploy kubernetes clusters and maintain static infrastructure.
Config management tools were a huge part of ushering in DevOps, which in turn created a need for better authentication practices in automation. Secrets management solutions have evolved to provide flexible and accessible credential storage, and authentication protocols like OpenID Connect (OIDC) have created more robust ways to verify machine identities.
In this article, we’ll break down the different ways you can handle secrets in configuration management tools. We’ll start by taking a brief look at the native secrets management capabilities present in a few tools. Then, we will explore a more generic way to retrieve secrets using CI and Secrets Management infrastructure. By the end of this article, you’ll have a good grasp on the different options you have for handling secrets in configuration management tools.
Ansible secrets management
Ansible is the most widely used configuration management tool, and it provides a couple of native methods for secrets management. Deciding when to use each one really comes down to whether you have Ansible Automation Platform (AAP) and whether you use a supported secret management solution.
Ansible Automation Platform is a paid offering from RedHat. The Automation Controller (previously called Ansible Tower) has some nice integrations with secrets management platforms like HashiCorp Vault and CyberArk Conjur. Without getting too into the weeds, these basically enable you to fetch things like SSH keys and sensitive variables from the secrets management platform in your automation. This is a handy feature, and the only downside is that it’s not a very transferrable skill if you switch jobs and are no longer working with Ansible Automation Platform.
Another feature in Ansible that allows for secure secrets access is Ansible Vault. A vault is an encrypted file that allows you to store your secrets with your Ansible configuration code. Vaults require a password at runtime to unlock. Below is a basic example of running a playbook with an Ansible Vault full of extra variables:
ansible-vault create vault.yml
ansible-vault edit vault.yml
ansible-playbook -i inventory.ini –extra-vars vault.yml –ask-vault-pass playbook.yml
Ansible vaults are simple to work with, but there are a couple of drawbacks to using them. The first drawback is that your encrypted secrets are in an easily accessible place – maybe even committed to the git history. A threat actor who finds your vault could steal it and try to crack it offline for however long they want.
The second drawback of Ansible Vault is that the password to decrypt it is another long-lived secret that needs to be stored somewhere. This “recursive secret” isn’t really making things much easier to manage. In the end, Ansible’s native secret management options are fine, but we have other options as well (more on that later).
Chef secrets management
Chef is another popular config management tool with a few options for handling secrets. The first option is to use the Secrets Management Integration helper. The helper is configured to connect to a secret management platform like Akeyless or HashiCorp Vault, and then you can fetch the string for any secret right from your cookbooks. You can use the helper instead of using a hard-coded value or pull from a data bag. Below is an example of how it might be used:
template ‘/etc/example_service/example_service.conf’ do
source ‘config.erb’
variables(
db_user: secret(name: ‘db_user’, service: aws_secrets_manager)
db_pass: secret(name: ‘db_pass’, service: aws_secrets_manager)
)
end
Encrypted data bags are another option for secret management in Chef. They are a lot like Ansible vaults, and come with the same “recursive secret” tradeoff.
The last Chef secret management option is Chef-Vault, which extends the encrypted data bag concept. Chef-Vault grants individual nodes access to the data bag by using the nodes’ existing public keys. This gets rid of the need for nodes to have another long-lived secret to decrypt the data values. They can just ask for the data using their own identity. The drawback of Chef-Vault is that nodes have to be pre-authorized to access secrets, which can be a challenge for auto-scaling systems.
Puppet secrets management
Puppet is the last config management tool we’ll cover for native secrets management. The common practices for handling secrets with Puppet will sound similar to what we have covered already.
Puppet uses a hierarchical key-value store called Hiera to keep track of configuration data. There is a Puppet module called Hiera-Eyaml that is a backend for Hiera that allows you to encrypt sensitive data in yaml files. Hiera-Eyaml, being an encrypted local file, is similar to Ansible Vault. The difference is that the Hiera-Eyaml is encrypted with a public keypair rather than a password. This means that you don’t have to worry about the file being cracked due to a weak password, but the private key must live on the Puppet server which is a long-lived credential that could be stolen.
Another way to manage secrets within Puppet is to use modules that integrate with secrets management solutions like CyberArk Conjur and HashiCorp Vault. Similarly to Ansible Automation Platform and Chef Secrets Management Integration, these modules establish a connection to the secret manager with long-lived API key used by the Puppet server or individual Puppet agents.
Ferdinand Boas
Workflow identity federation with OIDC
Now we have an idea of how to handle secrets natively with different configuration management tools, but there are similar trade-offs being made no matter what we are using. Let’s take a look at one more method of handling secrets in config management tools that isn’t an encrypted file or API integration.
OIDC is an authentication protocol that allows us to solve the “recursive secret” problem that has been a common theme of the config management tools we’ve covered so far. Rather than explain what OIDC is and how it works, I think it’s easier to show an example of what it can do by walking through how to do it in GitHub Actions.
GitHub Actions allows you to create a JWT for your workflow run that contains “claims”, or information about your workflow that are signed by GitHub’s identity provider. This JWT can be used by other services to verify that a request came from your workflow. To create a JWT for your workflow in GitHub Actions, you can use the following snippet:
jobs:
ansible-job:
runs-on: ubuntu-latest
permissions:
id-token: write
contents: readThe “id-token” line is what grants the workflow permission to request a JWT. Next, we can use the JWT to authenticate the workflow to a secret manager like Hashicorp Vault and retrieve a secret. In this scenario, Vault has already been configured to support federated workflow identities from GitHub, and this specific workflow’s claims have been authorized to access “ansible_secret” in Vault.
steps:
...
- name: Retrieve secret from Vault
id: secrets
uses: hashicorp/vault-action@d1720f055e0635fd932a1d2a48f87a666a57906c
with:
method: jwt
url: https://vault.yourdomain.com/
namespace: VAULT-NAMESPACE # HCP Vault and Vault Enterprise only
role: ROLE-NAME
secrets: secret/data/ci ansible_secret | ANSIBLE_SECRET
Finally, we can use the secret we retrieved from Hashicorp Vault in our Ansible automation by referencing it at ${{ steps.secrets.outputs.ANSIBLE_SECRET }}. Since GitHub Actions variables are converted to strings at runtime, we’ll put the secret in a Linux environment variable to avoid exposing the secret in the command line arguments.
- name: Use secret from Vault
run: |
ansible-playbook -i inventory.ini –extra-vars “secret=$ANSIBLE_SECRET” playbook.yml
env:
ANSIBLE_SECRET: ${{ steps.secrets.outputs.ANSIBLE_SECRET }}The JWT from GitHub that we used to access the secret in Hashicorp Vault is short-lived and will automatically expire when its Time To Live (TTL) is reached. This means we now have a way to access our config manager secrets that doesn’t rely on a recursive secret! Not only that – this pattern can be used to retrieve secrets for other config management tools or different IaC like Terraform. The only requirement is that your CI platform and secret manager need to support OIDC authentication for federated workflow identities.
Wrapping up
Securing secrets in automation can be challenging because we are authenticating non-human identities rather than actual people when we provide access to the secrets. We covered many native ways to handle secrets in configuration management tools, which are all valid ways to handle secrets, but they each had their trade-offs.
Modern authentication protocols like OIDC allow us to create new solutions for handling secrets, which gives us more options when considering what the right choice is for us. Hopefully this blog post helped you think about which trade-offs you’re willing to make, and which method of handling secrets makes the most sense for you.
