portrait

Tiexin Guo

Senior DevOps Consultant, Amazon Web Services
Author | 4th Coffee

In the first part of the supply chain security series, we had an overview of the supply chain, its attacks, and the SLSA framework to understand where supply chain security is headed. If you still need to read it, here's the link for you.

In this second part, we are going to explore Sigstore, a set of tools from which developers and security experts can benefit, and then we will learn how to use one of these tools named cosign to sign artifacts.

Without further ado, let's get started!

1. What is Sigstore?

Before we talk about Sigstore, let's quickly review the typical software development workflow and the possible threats in the supply chain, which we already mentioned in the first part of this series:

Supply Chain Threats

These threats are hard to spot when you don't know where all the software, dependencies, and packages come from. Securing the supply chain consists in maintaining the integrity of each link in the chain. Without constant identity checks, packages or dependencies can open the door to breaches, exploits, and supply chain attacks.

And this is precisely the problem Sigstore is been trying to solve.

Simply put, Sigstore is a set of new tools with the objective of establishing a new standard for signing, verifying, and protecting software, ensuring the software is what it claims to be, and that it has not been tampered with. It's a direct response to today's supply chain challenges, a work in progress for a future where the integrity of what we build and use will be adhering to a standard. It is a proposal to automate how we digitally sign and check components to achieve a safer chain of custody, tracing software back to the source.

Diving deeper into the backstage of Sigstore, it combines several tools and technologies:

  • Cosign: a tool to sign and verify containers (and other artifacts) that ties the rest of Sigstore together, making "signature invisible" infrastructure. More on that below.
  • Fulcio: A free root certification authority, issuing temporary certificates to authorized identities.
  • Rekor: a built-in transparency and timestamping service that records signed metadata to a ledger that can be searched but can't be tampered with.
  • OpenID Connect: a means of authentication, an identity layer that checks you're who you say you are.

Today, we will take a deeper look at what Cosign can do and how to integrate it.

2. What is Cosign?

In short, Cosign aims to create a "signature invisible" infrastructure: it signs container images and artifacts, stores the signature in an Open Container Initiative (OCI) registry, and verifies the artifacts against the stored signature, all of which are done without you having to worry about the signatures themselves: where to keep them, where to get them when verifying, etc.

Cosign has many features, such as:

  • Hardware and KMS signing, supports multiple KMS providers
  • Sign anything in an OCI registry
  • Kubernetes/OPA gatekeeper integration

Under the hood, when signing a Docker image, cosign actually creates a tag that stores the signature, and the tag has the Docker image's digest (the immutable, build-time id of the image) as part of its name: so, given any Docker image digest, you can find the corresponding tag which contains the signature of that image.

Since Cosign uses a naming convention (tag based on the sha256 of what we're signing) for locating the signature index and stores signatures in an OCI registry, if we sign reg.example.com/ubuntu@sha256:abc using Cosign, the signature of it would be stored at reg.example.com/ubuntu:sha256-abc.sig. If this sounds complicated, don't worry, the following tutorial should clear it up.

cosign signature

Next, let's install it and see what's what.

3. Using Cosign

3.1 Install

The easiest way to install Cosign is using Homebrew/Linuxbrew. If you are using Homebrew (or Linuxbrew), you can install Cosign by running:

brew install cosign

For other installation methods for different OS, see the official documentation here.

After the installation, we can verify the installation is successful by running the following:

$ cosign version

And you should get similar output to the following:

  ______   ______        _______. __    _______ .__   __.
 /      | /  __  \      /       ||  |  /  _____||  \ |  |
|  ,----'|  |  |  |    |   (----`|  | |  |  __  |   \|  |
|  |     |  |  |  |     \   \    |  | |  | |_ | |  . `  |
|  `----.|  `--'  | .----)   |   |  | |  |__| | |  |\   |
 \______| \______/  |_______/    |__|  \______| |__| \__|
cosign: A tool for Container Signing, Verification and Storage in an OCI registry.

GitVersion:    1.13.1
GitCommit:     d1c6336475b4be26bb7fb52d97f56ea0a1767f9f
GitTreeState:  "clean"
BuildDate:     2022-10-17T18:00:05Z
GoVersion:     go1.19.2
Compiler:      gc
Platform:      darwin/arm64

3.2 Generating Keys

Since Cosign does digital signature, which is based on asymmetric encryption, we need to generate key pairs. Here, we do not need to use another tool; Cosign can do this:

$ cosign generate-key-pair
Enter password for private key:
Enter again:
Private key written to cosign.key
Public key written to cosign.pub

Note that you need to set a password for the private key. Store the password, which will be used when you sign stuff using the private key.

3.3 Signature and Verification

In this tutorial, I will try to sign my own image ironcore864/helloworld:0.0.1, which has the sha256 digest value ec57e65246d331177a3cf831a1346b2f6e1d751d4dceeecfbc906b21ebb926a8. You can do the same on any image you have registered under your Docker Hub account.

If we go to the Docker Hub registry for this image, we can get the digest sha256:ec57e65246d331177a3cf831a1346b2f6e1d751d4dceeecfbc906b21ebb926a8 for ironcore864/helloworld:0.0.1

I will use this sha256 digest to sign with Cosign:

$ cosign sign --key cosign.key ironcore864/helloworld@sha256:ec57e65246d331177a3cf831a1346b2f6e1d751d4dceeecfbc906b21ebb926a8
Enter password for private key:
Pushing signature to: index.docker.io/ironcore864/helloworld

Now, if we go to the image repo registry again, we will see a new tag has been created: sha256-ec57e65246d331177a3cf831a1346b2f6e1d751d4dceeecfbc906b21ebb926a8.sig

As you can see, the tag name embeds the digest we just signed. This is how Cosign maps digests to signatures and stores. We have just signed a public container with our signature, and this has been pushed to the registry.

Next step, logically, is to verify the signature. Here we will do it manually and verify our own signature, which is of limited interest, but is still helpful to understand the basics. To verify the signature matches the public key:

$ cosign verify --key cosign.pub ironcore864/helloworld@sha256:ec57e65246d331177a3cf831a1346b2f6e1d751d4dceeecfbc906b21ebb926a8

Verification for index.docker.io/ironcore864/helloworld@sha256:ec57e65246d331177a3cf831a1346b2f6e1d751d4dceeecfbc906b21ebb926a8 --
The following checks were performed on each of these signatures:
  - The cosign claims were validated
  - The signatures were verified against the specified public key

[{"critical":{"identity":{"docker-reference":"index.docker.io/ironcore864/helloworld"},"image":{"docker-manifest-digest":"sha256:ec57e65246d331177a3cf831a1346b2f6e1d751d4dceeecfbc906b21ebb926a8"},"type":"cosign container image signature"},"optional":null}]
$ # This command returns 0 if at least one Cosign formatted signature for the image is found matching the public key.
$ echo $?
0

It's worth noting that we are signing and verifying using image digest directly instead of image tags in the example above. This is because a tag can change since we can push different stuff to the same tag.

OK, now we have figured out how Cosign works, but it seems a bit pointless, at least so far: if we have to generate a key pair locally and then do the signature and verification, it doesn't scale. How do I share key pairs with my team? How do I integrate the signing/verification process with my automagic continuous integration workflows?

Now, let's tackle these two questions in turn.

4. Using Cosign with a Kubernetes Secret

Cosign can use keys stored in Kubernetes Secrets to sign and verify signatures. The secret can be shared if team members all have access to the same Kubernetes cluster.

In order to create a secret, pass a k8s://[NAMESPACE]/[NAME] URI specifying the namespace and secret name to cosign generate-key-pair as follows:

cosign generate-key-pair k8s://default/testsecret
Enter password for private key: ****
Enter again: ****
Successfully created secret testsecret in namespace default
Public key written to cosign.pub

After generating the key pair, Cosign will store it in a Kubernetes secret using your current context. The secret will contain the private and public keys plus the password to decrypt the private key, like this:

apiVersion: v1
kind: Secret
metadata:
  name: testsecret
  namespace: default
type: Opaque
data:
  cosign.key: xxx
  cosign.password: yyy
  cosign.pub: zzz

To sign, we run the following:

$ cosign sign --key k8s://default/testsecret ironcore864/helloworld@sha256:ec57e65246d331177a3cf831a1346b2f6e1d751d4dceeecfbc906b21ebb926a8

And to verify:

cosign verify --key k8s://default/testsecret ironcore864/helloworld@sha256:ec57e65246d331177a3cf831a1346b2f6e1d751d4dceeecfbc906b21ebb926a8

5. Putting it Together with Your Continuous Integration

OK, now we got rid of local key pairs by using keys stored in a Kubernetes Secret. What about CI?

Luckily, we can basically automate the process we've done in any CI system:

  • Install Cosign
  • Docker hub login
  • Docker build, tag it with sha256 value
  • Cosign CLI sign and store signature

Here is an example GitHub Actions:

...
    steps:
      - name: Install Cosign
        uses: sigstore/cosign-installer@main

      - id: docker_meta
        uses: docker/metadata-action@v3.6.0
        with:
          images: ironcore864/helloworld
          tags: type=sha,format=long

      - name: Login to Docker Hub
        uses: docker/login-action@v2
        with:
          username: ${{ secrets.DOCKERHUB_USERNAME }}
          password: ${{ secrets.DOCKERHUB_TOKEN }}

      - name: Build and Push container images
        uses: docker/build-push-action@v2
        with:
          platforms: linux/amd64
          push: true
          tags: ${{ steps.docker_meta.outputs.tags }}
          labels: ${{ steps.docker_meta.outputs.labels }}

      - name: Sign image with a key
        run: |
          echo ${TAGS} | xargs -I {} cosign sign {}@${DIGEST}
        env:
          TAGS: ${{ steps.docker_meta.outputs.tags }}
          DIGEST: ${{ steps.docker_meta.outputs.digest }}
          COSIGN_PRIVATE_KEY: ${{secrets.COSIGN_PRIVATE_KEY}}
          COSIGN_PASSWORD: ${{secrets.COSIGN_PASSWORD}}
...

This code first installs cosign using the sigstore/cosign-installer action, then installs the docker/metadata-action and docker/build-push-action to build, tag, and push the Docker image.

To sign the image with Cosign (Sign image with key step), we use xargs -I {} to read each tag output by the build and push step, and then concatenate with the image digest to form the input for the cosign sign command.

For example, if the echo "${{ steps.docker_meta.outputs.tags }}" command prints the following output:

ironcore864/helloworld:latest

The xargs -I {} command will execute the cosign sign command for each item read from standard input, with the {} placeholder replaced by the image reference:

cosign sign ironcore864/helloworld:latest@sha256:abcdef1234567890

The @sha256:abcdef1234567890 is the digest of the image, which is obtained from the docker/metadata-action step.

To view the complete source code of this workflow, see here.

If you are using other CI systems, the principles apply: install Cosign and run the CLI tool to do the signature, which also saves the signature in the registry.

6. Summary

In this article, the second part of the Supply Chain Security series, we did a basic introduction to Sigstore/Cosign, learned how to use Cosign locally, with Kubernetes Secrets, and how to integrate it with continuous integration workflows.

On the surface, Cosign only covers one part of the supply chain (or the SDLC, for that matter), but it's more than a single-point solution: it signs images and everything stored in an OCI registry.

And the best part is that you don't have to worry about the signature storage at all. Imagine if you have to store the signature somewhere and then try to find it when you want to do a verification. I can't.

Moreover, Cosign integrates well with Kubernetes and OPA gatekeeper so that pulling images and verification using Cosign, then deploying containers in Kubernetes clusters can be fully automated. Amazing, right? For advanced Kubernetes users, if you want to know more about these applications, read the following links (advanced):

Next time, in the third (and final) part of this series, we will try to answer one question: how does secret detection fit in modern security frameworks like SLSA and NIST?
If you are interested, please subscribe. See you next time!