How to Handle Secrets in Jenkins

portrait

Keshav Malik

Keshav is a full-time Security Engineer who loves to build and break stuff.
He is constantly looking for new and exciting technologies
and enjoys working with diverse technologies in his spare time.
He loves music and plays badminton whenever the opportunity presents itself.

Jenkins is a popular open-source automation server that is widely used for building, testing, and deploying software. It allows developers to automate many aspects of their software development process, including continuous integration and continuous deployment.

As with any continuous integration software, managing secrets in Jenkins is crucial to ensure the security and integrity of the applications being built and deployed. In this blog post, we will be discussing the best practices for managing secrets in Jenkins, including configuring and managing secrets and how to handle potential security breaches.

The blog post also covers storing secrets with Hashicorp Vault and integrating it with Jenkins.

Prerequisites

Before diving into the specifics of this tutorial, managing secrets in Jenkins, it's important to have the following:

Familiarity with Jenkins and its basic concepts, including how to create jobs and build pipelines.
Access to an already set up Jenkins instance and a Hashicorp Vault for testing and implementation, which will allow you to follow along with the tutorial.
Installed plugins in your Jenkins instance: Credentials | Jenkins plugin, Credentials Binding | Jenkins plugin, and HashiCorp Vault | Jenkins plugin.

Additionally, you can have an application codebase ready to test the secrets management flow in your workflow. We will take as an example a NodeJS application that has Mocha tests and a Docker build setup.

Next, we have a continuous integration (CI) job that is triggered when commits are pushed to the stage branch. The code is then tested and merged into the main branch. After that, a continuous deployment (CD) job is triggered and builds a Docker image before publishing it to DockerHub.

Please note that this blog post is not contingent on this CI/CD workflow: although the presented flow can look pretty basic, you can follow along without any issues, no matter how complex or simple your workflow is!

Understanding Jenkins Secrets

Jenkins secrets, also known as credentials, are used for various authentication and authorization purposes and can include things like passwords, API keys, and private SSH keys.

When it comes to managing these secrets, it’s important to understand there are two scopes in Jenkins: system-wide and job-specific. System-wide secrets are available to all jobs and projects within a Jenkins instance, such as credentials for a shared database or Git repository. Job-specific secrets are only available to a specific job or project and are typically used for specific tasks, like deploying to a particular environment.

It is important to follow best practices when managing secrets in Jenkins. This includes using different secrets for different purposes, restricting their access, and regularly rotating and updating them to minimize the impact of a potential breach.

Configuring Jenkins Secrets

Now that we understand the basics of Jenkins secrets, let's take a look at how to set up and configure them. The process of configuring secrets in Jenkins will vary depending on the type of secret and the specific use case.

One way to configure Jenkins secrets is through the Jenkins web interface. To do this, navigate to the "Credentials" page in the Jenkins settings.

This is Jenkins' official credential management tool. You can add, modify, and delete secrets as needed. You can also specify which jobs or projects a secret is available for, and assign permissions to specific users or groups. There are multiple ways to store secrets using the Credentials Plugin.

One good practice when configuring a secret is to give it a descriptive and unique name that clearly indicates what the secret is used for. Avoid using generic or ambiguous names that could cause confusion or make it difficult to identify the correct secret.

Also, it is important to scope credentials appropriately based on the intended usage. For example, if a credential is only needed for a specific job or pipeline, it should be scoped to that job or pipeline rather than being made available globally. This helps to minimize the potential for unauthorized access or misuse of the credential.

Here we have stored DockerHub username and password with a global scope:

Here is an example of storing SSH keys (GitHub credentials in this case):

We can also store a text-based secret like an OAuth client secret for example:

If you have an .env holding environment variables needed in multiple steps in the build pipeline:

This is where the "Credentials Binding" plugin comes into play. This plugin allows you to bind secrets to environment variables, making them available to your build scripts. It gives you an easy way to package up all a job’s secret files and passwords and access them using a single environment variable during the build.

To use this plugin, you'll need to specify the ID of the secret you want to use, as well as the environment variable it should be bound to. The following example is for a freestyle project in Jenkins.

And below is an example for injecting the credentials PORT into a pipeline project in Jenkins. (full config here)

pipeline {
  …
  stages{
    …
    stage('Building Docker Image') {
      steps {
        script {
          withCredentials([string(credentialsId: 'PORT', variable: 'PORT')]) {
            dockerImage = docker.build("$registry:$BUILD_NUMBER", "--build-arg PORT=$PORT .")
          }
        }
      }
    }
  …
  }
  …
}

How to Use Vault Secrets in Jenkins

In addition to the built-in Jenkins Secrets manager, it is also possible to integrate with third-party secret management tools such as Hashicorp Vault, AWS KMS, etc. This can be particularly useful if you are already using Vault to manage secrets for other applications or services.

In this section, we will walk through the process of configuring Jenkins to retrieve secrets from Vault and use them in your pipeline.

Run your Vault with the following command and leave this terminal session open as dev servers run in-memory, i.e., shutting them off would wipe the slate clean.

# we are running the vault inside an ec2 machine thus the private ip
$ vault server -dev -dev-listen-address=”<private ip of ec2>:8082”

Now, run the following commands in a separate terminal session:

$ export VAULT_ADDR='http://<private ip of ec2>:8082'
# check the status just to be sure
$ vault status

It should print information similar to this:

Next, our Jenkins instance needs to communicate with the Vault. The following commands will take care of the auth-related formalities. Jenkins will later use this role to talk to the Vault.

# firstly enable the vault approle authentication method
$ vault auth enable approle

# ensure a v2 kv secrets engine is in use
$ vault secrets enable kv-v2

# create a local file “jenkins-policy.hcl” that houses a policy allowing “read” access to the path “secret/data/jenkins” under which we’ll store our secret

$ tee jenkins-policy.hcl <<"EOF"
  path "secret/data/jenkins/*" {
    capabilities = [ "read" ]
  }
  EOF

# create a vault policy “jenkins” from the local policy file “jenkins-policy.hcl”
$ vault policy write jenkins jenkins-policy.hcl

# create a vault approle “jenkins” and associate the vault policy “jenkins” with it

$ vault write auth/approle/role/jenkins token_policies=”jenkins” token_ttl=1h token_max_ttl=4h
  
# Next, we need the strings roleId and secretId with 
$ vault read auth/approle/role/jenkins/role-id
$ vault write -f auth/approle/role/jenkins/secret-id

These commands will ensure that apps and/or services like Jenkins can connect with the Vault. The access policy we’ve defined allows the path secret/data/jenkins/* to be accessed by the policyholder. Then, this policy is registered with the Vault and a role is attached to this policy.

Copy the role_id and secret_id for later use. These are what Jenkins will use to identify itself while communicating with the Vault, and to access secrets in the path secret/data/jenkins/*. We’ll store these as a credential in Jenkins.

Finally, on Vault’s end, we need to store the secret. There are multiple ways to achieve this, but we’ll store it as a secret text here. The following command is used to store a secret identified as my-secret in the kv secret engine in the path we earlier declared in the approle policy.

$ vault kv put secret/data/jenkins/my-secret PORT=4000

Then create credentials in Jenkins via the Jenkins web interface. To do this, navigate to the "Credentials" page in the Jenkins settings. Add a new credential under Global credentials. Select the “Kind” as “Vault App Role Credential”. And paste in the IDs role and secret we copied from the terminal. The form should look like the following:

Once the credentials are added, copy the credential’s ID for later reference in the Jenkins pipeline.

Lastly, the following example is for a pipeline project in Jenkins. (full config here)

def secrets = [
  [
    path: 'secret/data/jenkins/my-secret',
    engineVersion: 2,
    secretValues: [
      [envVar: 'PORT', vaultKey: 'PORT']
    ]
  ],
]

def configuration = [
  vaultUrl: 'http://172.31.36.198:8082',
  vaultCredentialId: 'ccc2cf0c-1b1a-40a1-a7bd-81ef1dc764f1',
  engineVersion: 2
] 

pipeline {
  …
  stages{
    …
    stage('Building Docker Image') {
      steps {
        script {
          withVault([configuration: configuration, vaultSecrets: secrets]) {
            dockerImage = docker.build("$registry:$BUILD_NUMBER", "--build-arg PORT=${env.PORT} .")
          }
        }
      }
    }
  …
  }
  …
}

In this section, we have demonstrated how to integrate Jenkins with Hashicorp Vault to securely manage your secrets. By leveraging the power of Vault's secret management capabilities and Jenkins' flexible pipeline configuration, you can ensure that your sensitive information remains protected throughout your CI/CD process.

Managing Secrets in Jenkins

Managing secrets in Jenkins is crucial for maintaining the security of your applications. To ensure this, it is important to follow best practices such as rotating secrets regularly, using strong passwords, and avoiding storing plaintext secrets in Jenkins or in version control.

In addition, regular auditing and monitoring of secrets are also important. This involves keeping track of access to secrets, and their usage, and updating them as necessary. Alerts should also be set up to notify of potential breaches, such as unauthorized access to secrets or failed login attempts.

In case of a security breach or secrets leakage, having a plan in place for revoking access to the compromised secrets, rotating all secrets, and conducting a thorough investigation to prevent similar incidents in the future is crucial.

Conclusion

As we have seen, Jenkins secrets are used to store sensitive information such as passwords, tokens, and keys that are needed by your applications to function properly. However, if not managed properly, these secrets can become a security risk.

Throughout this blog post, we have discussed the importance of managing secrets in Jenkins, the types of secrets that can be managed, and best practices for securing and managing these secrets. We also provided step-by-step instructions for setting up and configuring secrets in Jenkins natively or with an external secrets management solution.

It is important to remember that hard-coding secrets or storing them in plaintext can lead to security breaches. Therefore, it is crucial to follow best practices for secret management and to have a plan in place for handling potential breaches.