DevOps engineers must handle secrets with care. In this series, we summarize best practices for leveraging secrets with your everyday tools.

We live in a world where applications are used to do everything, be it stock trading or booking a salon, but behind the scenes, the connectivity is done using secrets. Secrets such as database passwords, API keys, tokens, etc., must be managed appropriately to avoid any breach.

The need for managing secrets is critical for any organization. Secrets can be leaked in many ways, including through version control systems (never hardcode any secrets in your code!), private messages, email and other communication channels. If secrets are leaked, it can lead to a loss of trust, credibility, and even business (you can learn why here). In some cases, leaked secrets can also lead to legal action. That's why it's so important to have a plan for managing secrets.

In this post, we will discuss 4 different ways to manage secrets in Python efficiently.

Pre-requisites

Before getting started, below are a few things to keep in mind to avoid any issues later.

• Python & pip installed on your machine
• Understanding of Python & CLI
• Python IDE such as PyCharm or VS Code
• Basic understanding of Cloud

4 Ways to Manage Secrets in Python

In this section, we will discuss 4 different ways to manage your secrets in Python.

1. From a file

Using an .env file

The .env file is a file used to store environment variables in Python. Environment variables are variables set outside of the Python code and are used to configure the Python code. The .env file is typically used to store secret keys and passwords.

We will use the python-dotenv package for accessing the content of the .env file. To get started, first install the package using the following command.

$ pip install python-dotenv

Create a .env file for testing purposes and paste the following secrets:

API_KEY=test-key
API_SECRET=test-secret

Of course, this file should not be committed to your git repo! Otherwise, it would be versioned and readable even after you delete it.

Add this line to your .gitignore file:

.env

Already committed the file? No worries, follow the instructions here:

Git Clean, Git Remove file from commit - Cheatsheet - GitGuardian Blog

Exposing secrets in a git repository is bad but mistakes happen. This is a complete guide and cheatsheet to rewrite and remove files from git history

GitGuardian Blog - Automated Secrets DetectionGuest Expert

Once done, create a main.py file and paste the below-mentioned code snippet. In this code, we are using the load_dotenv() function to load content from the .env file.

from dotenv import load_dotenv
import os

load_dotenv()

api_key = os.getenv("API_KEY")
api_secret = os.getenv("API_SECRET")

print("API_KEY: ", api_key)
print("API_SECRET: ", api_secret)

We can also use dotenv_values() function, which converts the secrets into a dictionary. Secrets can be accessed by using the following code snippet:

from dotenv import dotenv_values

secrets = dotenv_values(".env")

def main():
   print(secrets["API_KEY"])
   print(secrets["API_SECRET"])

if __name__ == "__main__":
   main()

When working on a large project, you may find that you need multiple .env files. For example, you may have a .env file for your local development environment and a .env.dev file for your cloud development production environment. The following code snippet can be helpful if you have multiple env files.

from dotenv import dotenv_values

secrets = dotenv_values(".env")
local_secrets = dotenv_values(".env.dev")

def main():
   print(secrets["API_KEY"])
   print(local_secrets["API_SECRET"])

if __name__ == "__main__":
   main()

Using a JSON file

To keep the secrets more organized, you can also use a JSON file. Let's create a secrets.json file and paste the following secrets into it.

{
   "db": {
       "host": "localhost",
       "port": 27017,
       "name": "test"
   },
   "server": {
       "host": "localhost",
       "port": 3000
   }
}

Now that we have the JSON file ready, let’s write a function to access secrets from the file:

# main.py
import json

def get_value_from_json(json_file, key, sub_key):
   try:
       with open(json_file) as f:
           data = json.load(f)
           return data[key][sub_key]
   except Exception as e:
       print("Error: ", e)
       
print(get_value_from_json("secrets.json", "db", "host")) # prints localhost

2. Using Environment Variables

Environment variables are variables that are set by the operating system or a specific user and are used by programs to determine various settings. We can use these variables to store our secrets and then access them in our program. You can use the following syntax to create an environment variable in macOS or a Linux machine.

$ export variable_name=value
$ export API_KEY_TEST=dummykey

On a Windows machine, you use GUI to add environment variables or use the following command to add a variable.

$ setx [variable_name] “[value]”

You can use the os package to access the os environment variable. Below mentioned is the sample code:

import os

# Get the secret key from the environment
secret_key = os.environ.get('api_key_test')
print(secret_key) // prints dummykey

Secrets at the command line should be handled with special care too! Have a look at the best practices with this:

Secrets at the Command Line [cheat sheet included]

Developers need to prevent credentials from being exposed while working on the command line. Learn how you might be at risk and what tools and methods to help you work more safely.

GitGuardian Blog - Automated Secrets DetectionDwayne McDaniel

3. Use a Cloud Secrets Manager

Most cloud service providers offer an inbuilt secrets manager that can be used to create and use secrets across cloud infrastructure. Following the secret managers offered by cloud providers:

• AWS Secrets Manager
• Secret Manager (By Google Cloud)
• Azure Key Vault

AWS Secrets Manager is widely used across the industry. Let’s write a function to create and access a secret in AWS using Boto3.

import boto3

def fetch_secret_from_aws(secret_name):
   try:
       session = boto3.session.Session()
       client = session.client(service_name='secretsmanager', region_name='us-east-1')
       get_secret_value_response = client.get_secret_value(SecretId=secret_name)
       return get_secret_value_response['SecretString']
   except Exception as e:
       print(e)
       return None
       
def create_secret_in_aws(secret_name, secret_value):
   try:
       session = boto3.session.Session()
       client = session.client(service_name='secretsmanager', region_name='us-east-1')
       client.create_secret(Name=secret_name, SecretString=secret_value)
       return True
   except Exception as e:
       print(e)
       return False

4. Using a KMS

A KMS is a key management system that is used to manage cryptographic keys. It is typically used in organizations in order to centrally manage and secure keys. A KMS can be used to generate, store, and distribute keys. It can also be used to revoke keys and to monitor key usage.

KMS is a convenient way to centrally manage the keys used by your applications and services and helps to ensure that only authorized users have access to them.

Hashicorp Vault is one of the best open-source KMS available in the market that offers a number of features and benefits, including the ability to manage secrets and keys across multiple environments, strong security controls, and good scalability.

Let’s write a function to read and write secrets to a specific path in the vault.

import hvac

def read_secret_from_vault(secret_path, token, secret_name):
   try:
       client = hvac.Client(
           url='http://localhost:8200',
           token=token,
       )
       read_response = client.secrets.kv.read_secret_version(path=secret_path)
       return read_response['data']['data'][secret_name]
   except Exception as e:
       print(e)
       return None

def write_secret_to_vault(secret_path, token, secret_name, secret_value):
   try:
       client = hvac.Client(
           url='http://localhost:8200',
           token=token,
       )
       create_response = client.secrets.kv.v2.create_or_update_secret(
           path=secret_path,
           secret={secret_name: secret_value},
       )
       return create_response
   except Exception as e:
       print(e)
       return None

Understanding the Python Secrets Module

The Python secrets module, introduced in Python 3.6, provides cryptographically strong random number generation specifically designed for security-sensitive applications. Unlike the standard random module which uses pseudo-random generators suitable for modeling and simulation, the secrets module accesses your operating system's most secure randomness sources.

The module offers several key functions for generating secure tokens: token_bytes() returns random bytes, token_hex() provides hexadecimal strings, and token_urlsafe() creates URL-safe Base64-encoded strings. For random selection, secrets.choice() securely picks elements from sequences, while randbelow() generates integers within specified ranges. The compare_digest() function performs constant-time string comparisons to prevent timing attacks.

import secrets

# Generate secure tokens
api_token = secrets.token_urlsafe(32)
session_id = secrets.token_hex(16)

# Secure random selection
secure_password_char = secrets.choice('abcdefghijklmnopqrstuvwxyz')

When generating tokens for security purposes, the module recommends using at least 32 bytes (256 bits) of randomness to resist brute-force attacks. This makes the secrets module essential for creating API keys, session tokens, and other security-critical random values in Python applications.

Advanced Security Considerations for Python Secrets

Beyond basic secrets storage, Python applications require sophisticated security measures to protect sensitive data throughout its lifecycle. Implementing proper secret rotation, access logging, and encryption at rest ensures comprehensive protection against various attack vectors.

The secrets.compare_digest() function prevents timing attacks by performing constant-time comparisons, essential when validating API keys or authentication tokens. This function should replace standard string comparison operators when dealing with security-sensitive data:

import secrets

def validate_api_key(provided_key, stored_key):
    return secrets.compare_digest(provided_key, stored_key)

For applications requiring high-entropy passwords, combine the secrets module with string manipulation to create complex credentials meeting specific requirements. Consider implementing secret validation patterns that verify entropy levels and character complexity before accepting user-provided secrets.

Memory management becomes critical when handling secrets in Python. Use techniques like immediately overwriting variables containing sensitive data and avoiding string concatenation that might leave traces in memory. Integration with hardware security modules (HSMs) or secure enclaves provides additional protection for the most sensitive applications, ensuring cryptographic operations occur in tamper-resistant environments.

Wrapping it up

Managing secrets is an essential part of application development. When developers hardcode secrets in cleartext into their applications, it creates a potential security breach. An attacker can use those secrets to access sensitive data if those secrets are discovered.

Another alternative to the proposed methods here is to check secrets into source code and share them with your team encrypted. This can be a very flexible solution if you learn how to leverage a tool such as Mozilla SOPS. To learn more about SOPS and get started using it, read this tutorial:

A Comprehensive Guide to SOPS: Managing Your Secrets Like A Visionary, Not a Functionary

Have you heard about SOPS? If you have already been in a situation where you needed to share sensitive information with your teammates, this is for you.

GitGuardian Blog - Automated Secrets DetectionGuest Expert

Did you know that GitGuardian detects more than 16000 secrets hardcoded into GitHub commits daily? When a secret is detected, GitGuardian notifies the repository owner so they can take action to protect their data.

If you want a better view of how many secrets could be hidden inside your repositories, sign up (for free) or book a demo to the GitGuardian Platform to scan every single commit and take preventive action now!

We hope this blog post has provided you with a better understanding of how to manage secrets in Python and keep your applications secure.

It is part of a series on secrets management with popular technologies, have a look!

4 Ways to Store & Manage Secrets in Docker

DevOps engineers must handle secrets with care. In this series, we summarize best practices for leveraging secrets with your everyday tools.

GitGuardian Blog - Automated Secrets DetectionGuest Expert

How to Handle AWS Secrets

In this blog post, we’ll cover some best practices for managing AWS secrets when using the AWS SDK in Python.

GitGuardian Blog - Automated Secrets DetectionGuest Expert

How to Handle Secrets in Jenkins

DevOps engineers must handle secrets with care. In this series, we summarize best practices for leveraging secrets with your everyday tools.

GitGuardian Blog - Automated Secrets DetectionGuest Expert

This article is a guest post. Views and opinions expressed in this publication are solely those of the author and do not reflect the official policy, position, or views of GitGuardian, The content is provided for informational purposes, GitGuardian assumes no responsibility for any errors, omissions, or outcomes resulting from the use of this information. Should you have any enquiry with regard to the content, please contact the author directly.