portrait

Tiexin Guo

OS Developer @Ubuntu
CNCF ambassador | LinkedIn

In my previous blogs, I have covered how to handle secrets in Helm and Jupyter notebooks. If you haven't read them yet, here are the links:

They are part of the "How to Handle Secrets" series, and there are more:

To complete the list, today, let's examine how to handle secrets in another popular programming language: Golang.

According to the TIOBE Index, Go has grown from No.17 in 2019 to No.9 in 2024, and it constantly ranks in the top 10 this year. Plus, Go is a first-class citizen in the cloud-native ecosystem: Kubernetes, Helm, and many other things are written in Go, which means that Go can smoothly interact with their APIs, making it an ideal choice for developing cloud-native tooling or applications running in a cloud environment.

As Go's popularity grows, it's becoming increasingly important to handle secrets properly. So, next, let's examine different ways of handling secrets in Go and some recommendations and best practices.

1 Read Secrets from an ENV File

Like many other programming languages, the simplest method to read secrets is from an ENV file. You can do this with the godotenv package (ported From Ruby's dotenv lib):

First, let's create a project:

mkdir test && cd test
go mod init github.com/ironcore864/test

Then, create a .env file containing fake secrets for demo purposes:

cat > .env<< EOF
S3_BUCKET=YOURS3BUCKET
SECRET_KEY=YOURSECRETKEYGOESHERE
EOF

Next, create the main source code file main.go, and use godotenv to read the key-value pairs from .env:

package main

import (
	"fmt"
	"log"
	"os"

	"github.com/joho/godotenv"
)

func main() {
	err := godotenv.Load()
	if err != nil {
		log.Fatal("Error loading .env file")
	}

	s3Bucket := os.Getenv("S3_BUCKET")
	secretKey := os.Getenv("SECRET_KEY")

	// now do something with s3 or whatever
	fmt.Println(s3Bucket)
	fmt.Println(secretKey)
}

Install dependencies and run it:

go mod tidy
go run main.go

As you can see, it's simple and intuitive: just one line of code (godotenv.Load()) and then you can treat the secrets like environment variables as if they were in there. But beware!

Although it's simple, this approach is not recommended in the modern cloud-native world (we only start with this method for the sake of completeness). Here's why:

In many cases, one of the main reasons to choose Go, besides its popularity, is that it is extremely suitable for containerized apps. As Go is a compiled language, we don't need a runtime (unlike Python): we just need to deploy a pre-built binary to run it. As a result, Go images are usually small, hence fast to pull to the runtime platform (especially if we use a lightweight base image like Alpine Linux).

However, we can't hard-code the ENV file containing secrets in the Docker image.

Although, technically, it's possible to mount a file to a container (for example, in K8s you can inject configuration data into Pods using a ConfigMap), this isn't ideal for two reasons:

  • You need to create that ConfigMap from a file containing secrets. How can you securely store and use the file containing sensitive information?
  • It's a little complicated because we need to handle ConfigMaps and volumes in the Pod.

So, reading secrets from a file doesn't really meet the needs of a cloud-native environment.

What's the alternative, then? Let's continue exploring.

2 Read Secrets from a Secret Manager

To regular readers of the GitGuardian blog and my articles, a secret manager is not an unfamiliar concept. And yes, it's much better (not only more secure but also more compatible with the cloud-native environment) to read secrets from a secret manager.

If you haven't used a secret manager in a major public cloud, read more about it in my previous blogs:

2.1 How to Authenticate with a Secret Manager

Before diving into how to read secrets from a secret manager (which is kind of simple; we only need to call some APIs), it's more important to figure out how to authenticate with a secret manager.

Since secret managers are services hosting sensitive information, our apps need some form of secret token to authenticate with them. Generally speaking, there are three basic ways to get a token for our apps:

  1. Using a token provided directly to the app.
  2. Using an authentication method provided by secret managers rather than the runtime platform. For example, username/password authentication.
  3. Using the runtime platform's identity. For example, the cloud provider's IAM role can be used to authenticate with secret managers.

Next, let's analyze the three methods.

2.2 Token-Based Authentication

Like many web services, secret managers can generate tokens (e.g., JWT, OAuth2), which we can then provide directly to our apps so that they can call secret managers' APIs.

Although this is simple, it's the least preferred method of the three aforementioned choices:

  • We have to guarantee the secure delivery of the tokens ourselves instead of delegating this to some automation mechanism. If we have 100 apps to operate, we'd need to deliver 100 tokens, which is extra operational overhead and extra attack surface.
  • We don't get the app identities to token mapping. This means that if a token is compromised, we wouldn't know which identity is using which token, and hackers can use the token as well, instead of the app that is supposed to use it.

So, let's skip this non-ideal method, and continue exploring.

2.3 Non-Platform Credentials

Using a non-platform credential, like a username/password authentication (non-platform means that it's not provided by the runtime platform, to differ from the option in the next section), is better. This kind of credential can associate an identity with the app that is using it.

Think of this method as using a web service that requires login: first, you log in to a website using your username and password so that the service knows it's you who's using it (the identity). Then, after logging in, the backend returns some token to the frontend so that logged-in users can access protected APIs with that returned token.

This is already a significant improvement over the token authentication method, but still, there are some challenges:

  • If we deliver the username and password to an app in the same way we would deliver a token, the risk factor is the same: although the credential consists of two parts (username and password), they are delivered together in the same way, meaning the risk of leaking both is the same as leaking a token.
  • If the password is never rotated and valid forever, it still somehow is like a token: If it's compromised, others can use it for quite a long time.

Some secret managers, such as HashiCorp Vault, handle these challenges automatically by using a mechanism called an AppRole, which basically is a username/password authentication mechanism, but with some interesting implementation:

  • The AppRole consists of two parts: the username and the password.
  • The username is non-sensitive and can be delivered to the app in a normal way, for example, defined in a ConfigMap, passed in as an environment variable, etc.
  • The password is sensitive, but it's not delivered at the same time in the same way as the username. The app's runtime platform is pre-configured as a trusted entity of the secret manager, it's the app's runtime platform, not the app itself, that requests a password, and the runtime platform will need a way to deliver the password to the app (for example, store it in a file at a location so that the app can read it, or call the app's API to deliver the password to the app, etc.)
  • The idea is that the password is scoped, short-lived, and linked to the app.

If the description above confuses you, check out this diagram:

hashicorp vault approle auth

In this way, since the username and password are delivered by different channels at different times, and the password is scoped and short-lived, it's much more secure than the token method.

For more details on this mechanism, read this blog and this tutorial.

However, this mechanism has a drawback: not all major secret managers have this kind of mechanism. Because of this, we won't dive deep into a tutorial on this mechanism, but if you are interested, check out this sample app here. For more information on using Vault, check out this blog.

2.4 Platform-Managed Identity - Tutorial: AWS Secrets Manager + OIDC Provider

The third option is to use the platform's identity to authenticate with the secret manager so that the app itself won't need to worry about the authentication part.

For example, an app running in AWS EKS can have a service account, which can be mapped to an AWS IAM role. We can grant the role access to AWS Secrets Manager and certain secrets. Let me walk you through this process.

First, let's prepare an AWS EKS cluster with OIDC enabled. The easiest way to do so is by using eksctl, the official CLI for AWS EKS:

eksctl create cluster --name my-cluster --with-oidc --region ap-southeast-1
Note:
service account
I assume AWS CLI is already installed and configured in your environment. If you haven't done so, follow this guide.

You can install eksctl by following the official doc here.

If you already have an EKS cluster, make sure the OIDC is enabled.

Then, let's create an IAM policy to be used with our app:

cat >my-policy.json <<EOF
{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
              "secretsmanager:GetSecretValue",
              "secretsmanager:DescribeSecret"
            ],
            "Resource": "*"
        }
    ]
}
EOF

aws iam create-policy --policy-name my-policy --policy-document file://my-policy.json

Next, create an IAM role and associate it with a Kubernetes service account (remember to replace the AWS account in the policy arn):

eksctl create iamserviceaccount --name my-service-account --namespace default --cluster my-cluster --role-name my-role \
    --attach-policy-arn arn:aws:iam::202449767955:policy/my-policy --approve

Create a secret in AWS Secrets Manager for testing:

aws secretsmanager create-secret \
  --name MySecret \
  --secret-string "{'user':'tiexin','password':'PASSWORD'}"

We deploy a Pod with the service account my-service-account, which we mapped to a newly created IAM role my-role with policy my-policy that allows it to describe and get secrets from AWS Secrets Manager:

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: test
spec:
  serviceAccountName: my-service-account
  containers:
  - name: go-container
    image: golang
    command: [ "sleep", "3600" ]
EOF

Finally, let's get into the Pod to write some Go code to simulate an app accessing the secret:

kubectl exec -it test -- /bin/bash

In the pod:

mkdir test && cd test
go mod init test

cat >main.go <<EOF
package main

import (
	"context"
	"fmt"
	"log"

	"github.com/aws/aws-sdk-go-v2/aws"
	"github.com/aws/aws-sdk-go-v2/config"
	"github.com/aws/aws-sdk-go-v2/service/secretsmanager"
)

func main() {
	secretName := "MySecret"
	region := "ap-southeast-1"

	config, err := config.LoadDefaultConfig(context.TODO(), config.WithRegion(region))
	if err != nil {
		log.Fatal(err)
	}

	svc := secretsmanager.NewFromConfig(config)

	input := &secretsmanager.GetSecretValueInput{
		SecretId:     aws.String(secretName),
		VersionStage: aws.String("AWSCURRENT"),
	}

	result, err := svc.GetSecretValue(context.TODO(), input)
	if err != nil {
		log.Fatal(err.Error())
	}

	var secretString string = *result.SecretString

	fmt.Println(secretString)
}
EOF

go mod tidy
go run main.go

You should get the content of the secret successfully.

As we can see from the code above, the app doesn't do any authentication, but rather, since it's running with a K8s service account, which is bound to an IAM role. The IAM role handles the access to the secrets manager.

It's worth mentioning that although in this example we demonstrated authenticating with AWS Secrets Manager using IAM roles, authenticating with HashiCorp Vault using IAM roles can be achieved as well.

Apparently, authenticating through the platform is the best, because not only it's secure, but also the apps don't need to worry about identities and authentications altogether. However, there are some limitations: not every cloud platform supports this feature for all secret managers. For example, maybe you are using a cloud with HashiCorp Vault, but that cloud doesn't have an authentication plugin for Vault. Or, for another example, maybe you are running your workload on bare metal servers but you want to use AWS Secrets Managers, and your bare metal servers can't have IAM roles.d

To tear down the IAM role and policy (the EKS cluster is kept for the next section):

3 The Ultimate Solution: Not Worrying about Secrets, At All

3.1 Issues with Explicitly Using a Secret Manager

We have come quite far from reading secrets from an ENV file, but there are still possible improvements.

Note that our in the previous section's tutorial, the Go app is not "secret-manager-agnostic": It still needs to know which secret manager is used, and it still has the code to call the secret manager's APIs. This can be suboptimal for a few reasons:

  • You are locked in with a specific secret manager.
  • The Go app is not "secret-manager-agnostic", meaning you need to know where the app is running and adjust the code for that platform. This inflexibility can be especially problematic if you are migrating from one cloud to another, or running a multi-cloud architecture where your app needs to run in multiple clouds and needs access to different secret managers. (For more information on multicloud security architecture, see this blog here).
  • If you have multiple teams managing dozens of apps, each app needs to repeat the secret manager's API calls.

And maybe more.

3.2 The Solution

Is there a way to benefit from secret managers while not worrying about them at all, achieving some "secret-manager-transparent" state?

Yes there is, and it could be the ultimate simple solution: don't even think about secret when writing Go apps, let somebody else worry about it.

Allow me to expand:

  • You write your Go app in a normal way and get secrets from ENV vars as if they are already there, following the 12-factor app methodology.
  • You let someone else (well, maybe it's still you, but you get the idea) worry about the secrets: Where are they stored? In a secret manager somewhere. How do they become available as your app's ENV vars? Use some synchronization mechanism to get them from the single source of truth (secret manager) to the runtime platform where your app is running.

You may think pushing the task of secret handling to others is irresponsible, but if you look at the big picture, this is the most responsible way possible: the secret handling for all apps, no matter what they are, in what languages they are written, is tackled once and for all, using a unified solution. No duplicated API calls in your dozens of microservices anymore, and no worrying about multicloud anymore.

So, the key is to synchronize secrets from a secret manager to the runtime platform where your app is running, so that your Go app can read them directly from ENV vars without doing anything.

3.3 Syncing External Secrets - External Secrets Operator Tutorial

Next, let's have a look at how to achieve this.

There are many ways to achieve this, for example, using the External Secrets Operator to sync secrets from secret managers to K8s, using the Vault Secrets Operator on Kubernetes to sync secrets from HashiCorp Vault to K8s, and more. In this tutorial, let's continue using the EKS cluster and the secret created in the previous section, and use the External Secrets Operator to sync secrets to our Go app.

First, let's install the External Secret Operator:

helm repo add external-secrets https://charts.external-secrets.io
helm repo update
helm install external-secrets \
  external-secrets/external-secrets \
  -n external-secrets \
  --create-namespace

Then, let's create an IAM role with policies enabling the external secret operator to get secrets from AWS Secrets Manager, solving the authentication problem using the "platform's identity" method:

cat >external-secret-policy.json <<EOF
{
  "Version": "2012-10-17",
  "Statement": [
    {
      "Effect": "Allow",
      "Action": [
        "secretsmanager:GetResourcePolicy",
        "secretsmanager:GetSecretValue",
        "secretsmanager:DescribeSecret",
        "secretsmanager:ListSecretVersionIds"
      ],
      "Resource": "*"
    }
  ]
}
EOF

aws iam create-policy --policy-name external-secret-policy --policy-document file://external-secret-policy.json

Then, create an IAM role for the service account with the above policy:

eksctl create iamserviceaccount --name my-external-secret-store --namespace default --cluster my-cluster --role-name my-external-secret-role \
--attach-policy-arn arn:aws:iam::202449767955:policy/external-secret-policy --approve

Then, let's create a SecretStore pointing to AWS Secrets Manager using the IAM role for serviceaccount for authentication (the production-ready way):

cat <<EOF | kubectl apply -f -
apiVersion: external-secrets.io/v1beta1
kind: SecretStore
metadata:
  name: aws-secretsmanager
  namespace: default
spec:
  provider:
    aws:
      service: SecretsManager
      region: ap-southeast-1
      auth:
        jwt:
          serviceAccountRef:
            name: my-external-secret-store
EOF

cat <<EOF | kubectl apply -f -
apiVersion: external-secrets.io/v1beta1
kind: ExternalSecret
metadata:
  name: test-secret
  namespace: default
spec:
  refreshInterval: 1h
  secretStoreRef:
    name: aws-secretsmanager
    kind: SecretStore
  target:
    name: test-secret
    creationPolicy: Owner
  data:
  - secretKey: secret
    remoteRef:
      key: MySecret
EOF

Then, we deploy a Pod referencing a K8s Secret as ENV var. We did not create the K8s Secret, it is automatically synchronized by the external secret operator:

cat <<EOF | kubectl apply -f -
apiVersion: v1
kind: Pod
metadata:
  name: test-es
spec:
  containers:
  - name: go-container
    image: golang
    envFrom:
    - secretRef:
        name: test-secret
    command: [ "sleep", "3600" ]
EOF

Get into the Pod to write normal Go code to read from ENV vars:

kubectl exec -it test-es -- /bin/bash

In the pod:

mkdir test && cd test
go mod init test

cat >main.go <<EOF
package main

import (
	"fmt"
	"os"
)

func main() {
	envVarValue := os.Getenv("secret")
	fmt.Printf("%s\n", envVarValue)
}
EOF

go mod tidy
go run main.go

The secret value is read from ENV var.

As you can see, our simple Go program can transparently read the environment variable at runtime. This is made possible thanks to the job done automatically by the external secret operator.

Note that this is only a demo, in the real world, we would use Helm charts to package the ExternalSecret object and our app, and in the app, we don't need to think about secret managers, at all. For more information on how to handle secrets in Helm charts, read this blog.

To tear down after this tutorial:
eksctl delete iamserviceaccount --name my-external-secret-store --namespace default --cluster my-cluster
aws iam delete-policy --policy-arn arn:aws:iam::202449767955:policy/external-secret-policy
aws secretsmanager delete-secret --secret-id MySecret
eksctl delete cluster --name my-cluster

4 Summary

In this blog, we covered various ways of secrets handling in Go.

The ultimate solution, following the 12-factor app methodology, is not to worry about secrets at all: Let someone else worry about how to securely synchronise secrets to the app's runtime platform. No matter which public cloud you are using, chances are there is a solution to achieve this goal, just as demonstrated in the previous tutorial.

If, for some reason, the best choice above isn't available to you, there are also some okayish ways (with some limitations), like using the underlying platform's identity to authenticate your app to a secret manager (but not all platforms support all secret managers) or using a secure auth method like the Vault AppRole (but not all secret managers support this method).

Authenticating using a token is the least preferred method. Use it only when the previous options are absolutely impossible.

The last resort is to read from an ENV file, which is not recommended in today's fast-paced, security-first, cloud-native world.

As a finishing note, if you want to learn more about HashiCorp Vault and data security in general, read my other blog, "Data Security: an Introduction to AWS KMS and HashiCorp Vault."

If you enjoyed reading this article, as always, please share and subscribe. See you in the next piece!