API keys are a simple authentication method, essentially a unique code used to identify an application. However, as an authentication mechanism, API keys have relative security limitations: like any password, they are susceptible to exposure, lack fine-grained access control, and provide minimal audit capabilities.
Given these security limitations, you might wonder, Why are they still widely used? Well, it all comes down to one word: simplicity - the ease of implementation, together with low cost (only true in certain cases, though), makes it an attractive option in many situations, like public APIs with low-sensitivity data, internal APIs, prototyping, and simple scripting.
Over the years, I've come to realise that in security, there is no one definitive answer that fits all; it's all relative. Although there are authentication methods with a higher level of security for sensitive APIs, the decision of whether to use API keys is based on careful assessments of your specific requirements and needs. At the end of the day, it's a trade-off between convenience and security, and the right answer depends on the context.
That said, if you have decided API key is the best choice for your application, what are the best practices to mitigate some of the risks?
1 Never Embed API Keys in Code
To prevent API keys from leaking, the first and foremost rule is, as you guessed, never store them in the code. Embedding API keys directly in client-side code or committing them to version control systems is, no doubt, a recipe for disaster: Anyone who can access the code or the repository can steal the keys.
To read more about the actionable items on this, read Best Practices for Managing and Storing Secrets Including API Keys and Other Credentials.
2 Securely Store API Keys with Secret Managers
If we don't store API keys in code or config files, where to store them, and how?
Implementing an API key storage system? Out of the question (for most companies and teams), because securely storing and managing API keys bring tremendous operational overhead, like storage overhead (handling encryption at rest and in transit), management overhead (backup, recovery, audit logging, etc.), usage overhead (rotation, how to update and fetch), and distribution overhead (think insecure channels like email, file transfer, and plain text in chat).
The short answer: Don’t reinvent the wheel. Use tools like HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, or Google Cloud Secret Manager.
Using purpose-built systems for managing secrets and API keys brings many benefits:
- They handled all the operational overhead: encryption at rest and in transit, backup/recovery, etc.
- They can handle automated key rotation and expiration. If an API key is compromised, it can be used to access our API indefinitely unless we revoke it. So, regular rotation is not optional; it's a must, and automated rotation reduces not only overhead but also risk of human error.
- They serve as the single source of truth, not only for API keys, but all secrets in general, and work as a holistic centralized secret management solution.
- They may even provide more secure access control, following the principle of least privileges and audit logging capabilities.
- Almost all secret managers provide ways to inject secrets as environment variables in containerized environments, following the 12-factor app methodology: This separates the configuration from the code, making it easier and more secure to manage and update keys without modifying the app.
The list goes on, and they are all essential for enterprise-grade security.
Greg Bulmash
3 Use API Gateways for Centralized Security Control
Now that we covered how to store API keys securely, let's talk about how to use them securely.
When it comes to using API keys, there are a few challenges, including:
- If an API key is compromised, it can be used to launch denial-of-service (DoS) attacks or to exhaust our services' resources - That's why we need to implement rate limiting, a mechanism controlling the number of requests a client can make in a specific time period, and throttling, which slows down or delays excess requests instead of outright rejecting them. Both could help manage traffic spikes and prevent server overload.
- Even if we could implement this by ourselves, in the modern microservice architecture, it means we need to implement it for every service. Managing API security across dozens or even hundreds of services is a big challenge.
- Even if we have implemented all security policies for all APIs, the lack of centralized monitoring is another big problem.
You see where I'm going with this, and you are right: In the modern microservice architecture era, we need a centralized security control plane for all of our APIs, and this is where API gateways come to the rescue.
API Gateways, like AWS API Gateway, Kong, etc., are designed to solve these problems, simplifying and centralizing the management of all APIs, providing a single entry point for all requests. Features like limiting, throttling, and DDoS protection are baked in; API gateways can also provide centralized logging and monitoring; they even provide more features like input validation, data masking, and response filtering. All of these help enforce consistent security policies across all APIs, reduce operational overhead, enhance visibility and control, and make scalability a reality.
4 Logging/Monitoring/Alerting and Anomaly Detection
Now that we can securely store and use API keys, next, we need to pay attention to compromised API keys, which are a significant threat. A single leaked key can grant malicious actors unauthorized access to our systems, potentially leading to data breaches, service disruptions, and even financial losses. The answer to this is proactive monitoring, robust logging, and intelligent anomaly detection.
The basis of any effective API security monitoring strategy starts with logging. We need to capture every API request, including the details, which are already handled by API gateways. Then, raw logs are only useful if we can analyze them effectively, and this is where log aggregation and analysis tools come in. Examples:
- Cloud-Based Logging Services: AWS CloudWatch Logs, Azure Monitor Logs, and Google Cloud Logging offer scalable and managed logging solutions within their respective cloud environments. This is especially helpful if you are using API gateways provided by the cloud providers.
- The ELK Stack: A popular open-source alternative for log management and analysis. Elasticsearch provides search and storage, Logstash handles log ingestion and processing, and Kibana offers visualization and dashboarding.
- Security Information and Event Management (SIEM) Systems: SIEMs like Splunk, Sumo Logic, or Datadog Security Monitoring are designed to collect, analyze, and correlate logs from various sources. They provide powerful search capabilities, alerting, and reporting features.
True security comes from detecting deviations from the norm: We want to track API usages, look for suspicious patterns in real-time, and be notified when something happens. And this is where monitoring/alerting kicks in. In general, we want to monitor unexpected spikes in traffic, requests from unusual locations, access to unauthorized resources (might indicate attempts to escalate privileges), and high error rates/repeated failures (might indicate attempts to exploit a vulnerability).
The tools mentioned above also provide monitoring/alerting capabilities. For example, cloud monitoring services like AWS CloudWatch allow us to create alerts based on metrics and log data, and SIEM Systems often offer built-in alerting based on log analysis.
With the rise of AI/ML, we can use tools or even custom ML models to train on our API logs to identify subtle anomalies that are not visible to human eyes. By establishing behavioral baselines and implementing geographic and temporal analysis, we can use AI to detect: request volume spikes and patterns, error rate anomalies, unusual geographic access, off-hours usage patterns, permission escalation attempts, and so on.
5 Securely Handling API Keys in CI/CD Pipelines
All the above practices enhance API security in either the usage/storage or production environment, but there is another area where API keys could be compromised: the continuous integration/continuous deployment systems and pipelines. By nature, CI/CD involves running automation scripts and executing commands in a non-interactive way, which sometimes requires API keys, and this means the keys need to be stored somewhere and passed to the pipelines at runtime.
One risky area is the pipeline definition files (e.g., .gitlab-ci.yml, Jenkinsfile, GitHub Actions workflows). The rule of thumb is that, since these files are also stored in the code repo, do not hard-code API keys in CI/CD definitions.
A better solution is to use API keys in pipelines in the form of environment variables, whose values are set in the CI/CD system's UI. This approach, however, is better, but not good enough. The variables might still be visible to users with access to the CI/CD system, with a risk of being accidentally logged or exposed. Another reason is that CI/CD systems aren't designed to store secrets in the first place: not a centralized place to manage all (not just for CI/CD) secrets, not a suitable single source of truth, copy-paste across projects, weak or no fine-grained access control (a violation of the least-privilege principle), no automated rotation mechanisms (more operational overhead), no audit capabilities, and last but not least, probably not the most secure solution, as shown by the 2022 CircleCI breach that exposed all their customers' secrets.
The solution is, of course, to use secret managers to store API keys for CI/CD pipelines and securely synchronize them into the pipelines. To know more about using a secret manager, for example, the AWS Secrets Manager, see this blog post, which also includes an example of accessing secrets stored in the secret manager from the CI pipeline. There is also an equivalent piece on Google Cloud Secret Manager. The security of the CI - Cloud Secret Manager integration could be further enhanced by OpenID Connect (OIDC), which completely eliminates short-lived keys/tokens. This blog post details how to achieve this. And, for a more comprehensive guide on handling secrets in CI/CD pipelines, see this tutorial.
6 Develop an Incident Response Playbook
With all the practices above, we've made our API keys secure in code repos, in production, in CI/CD systems, with monitoring, which seems to be good enough. In security, however, "good enough" isn't enough: There still could be a leakage not caught by our comprehensive security framework, and we need a structured process to quickly respond to security breaches and to limit the damage, and this is the incident response, which is part of the DevSecOps discipline.
If done properly, incident response can greatly improve the overall mean time to acknowledge (MTTA) and mean time to remediate (MTTR). However, in the event of a security breach, we humans tend to panic, and when we panic, we don't think clearly or do things logically. This is why we need a playbook: to define a standard process so that we can handle incidents reliably.
In general, the lifecycle of an incident starts with an alert from our centralized monitoring system, then we need to analyze it, limit the damage, remove the root cause, and recover. During the course, we need to identify the severity level of the incident, and may escalate or ask for help from other colleagues and teams. Afterwards, post-incident activities like review and postmortem are mandatory.
A well-defined incident response playbook should cover all these aspects. While it sounds like a lot, there are some incident management platforms, like OpsGenie, PagerDuty, etc., which can be integrated with monitoring tools, define on-call schedules, escalation policies, and incident tracking.
7 Conclusion
Last but not least, for sensitive APIs that require a higher level of security, it's crucial to consider more robust authentication methods that offer enhanced security, flexibility, and control.
- JWTs: Fine-Grained Access Control. JSON Web Tokens (JWTs) are a compact, URL-safe means of representing claims to be transferred between two parties. In the context of API security, JWTs are used for authentication and authorization. They are suitable for microservices architectures, mobile applications, and scenarios where you need fine-grained access control.
- OAuth 2.0 / OIDC: Delegated Authorization. OAuth 2.0 (Authorization Framework) and OpenID Connect (OIDC) are industry-standard protocols for delegated authorization. Instead of sharing their credentials, users can grant limited access to their resources to third-party applications. This is ideal for scenarios where third-party applications need to access user data or resources.
- Mutual TLS (mTLS): The Gold Standard for High-Security APIs. mTLS is a two-way authentication process where both the client and the server authenticate each other using digital certificates. This goes beyond traditional TLS, where only the server's identity is verified. It's ideal for financial services, healthcare, government, and any other industry where security is paramount.
Summary
In this blog, we covered a few enterprise-proven best practices and methods to enhance API key security:
- Never Embed API Keys in Code
- Securely Store API Keys with Secret Managers
- Use API Gateways for Centralized Security Control
- Logging/Monitoring/Alerting and Anomaly Detection
- Securely Handling API Keys in CI/CD Pipelines
- Develop an Incident Response Playbook
Then, we briefly covered a few alternatives to API keys: JWT, OAuth 2.0/OIDC, and mTLS.
Of these three options, mTLS offers a significant step up in security compared to API keys, because it not only eliminates the risk of API key leakage (since the client's identity is tied to a cryptographic certificate), but also provides a robust defense against various attacks like man-in-the-middle attacks, replay attacks, and credential stuffing.
In my next blog posts, I'll be diving deep into the details of mTLS, covering topics such as:
- How mTLS works in detail
- Generating and managing certificates for mTLS
- Configuring your server and client for mTLS
- Use cases and best practices for mTLS
- Tools and libraries for implementing mTLS
So, stay tuned for a comprehensive guide to securing your APIs with mutual TLS!
