Secret Manager

Secret Manager
Author

Benedict Thekkel

What is AWS Secrets Manager & Why Use It

AWS Secrets Manager is a managed service for storing, retrieving, and rotating secrets (e.g. database credentials, API tokens). It offers:

  • Encryption at rest and in transit
  • Fine-grained access control (IAM)
  • Automated secret rotation (via Lambda)
  • Auditability (via CloudTrail)
  • Centralized secret management across your AWS environment
  • Integration with various AWS services (RDS, etc.) ([AWS Documentation][1])

So in a Django context, instead of storing credentials in settings.py, .env files, or config repos, you can fetch them at runtime (or at startup) from Secrets Manager.

That said, doing this properly requires care: you must balance security, performance, resilience, and maintainability.

Key Concepts & Capabilities

Before diving into Django details, here are key features of Secrets Manager to understand:

Feature Description
Secret A named object (often JSON) storing key/value pairs (e.g. {"username": "...", "password": "...", "host": "..."})
Version / Version Stages Each secret can have multiple versions (e.g. “AWSCURRENT”, “AWSPREVIOUS”)
Secret Rotation You can hook up a Lambda that periodically changes the secret and updates dependent services
Access control IAM policies can restrict which identities (roles, users) can access which secrets
Caching & Rate Limits API calls have latency and quotas, so often you’ll want to cache secrets locally
Failure / fallback If Secrets Manager is unavailable or permissions fail, you must decide what to do (fail, fallback)

Also, you should understand IAM roles, network access (if your Django service runs in a VPC), and AWS SDK (boto3) for fetching.

Integration Strategies in Django

There are multiple ways to integrate AWS Secrets Manager into a Django project. Each has tradeoffs. I’ll walk through them, and then propose a robust pattern.

Where to fetch secrets: startup vs runtime

  • At Django startup (in settings module) Load secrets when settings.py is imported (or in settings_base, etc.). This ensures all configuration is known at start time. Pros: Simpler, easier to reason about. Cons: If secrets fail to load, the app fails to start. Also, if you rotate secrets, app restart may be required, unless you build refresh logic.

  • Lazy / on-demand fetch Only fetch the secret when a component first needs it (e.g. when connecting to the DB). Use caching so you don’t hit the Secrets API on every request. Pros: More resilient to transient failures; you can refresh. Cons: More complexity, possibly slight runtime overhead on first access.

  • Middleware / dynamic refresh More advanced: periodically re-fetch the secret while the app is running (for secret rotation without downtime). This is trickier and must be done carefully to avoid mid-request inconsistencies.

Secret structure & naming conventions

  • You’ll usually store secrets as JSON structured objects. For instance:

    {
  "username": "dbuser",
  "password": "supersecret",
  "engine": "postgres",
  "host": "mydb.xxxx.region.rds.amazonaws.com",
  "port": 5432,
  "dbname": "myappdb"
}

  • You might have separate secrets for DB credentials, for third-party API keys, for other secrets (e.g. django/SECRET_KEY). Use naming or path conventions, e.g. project/prod/db, project/prod/django, etc.

  • Use versioning and rotation carefully. When rotating, ensure the new version is “promoted” before old clients fail.

Caching & client reuse

  • Boto3 clients should be reused rather than recreating per call (to avoid overhead).
  • Implement caching (in-memory, or a small TTL) so you don’t hit Secrets Manager for every request.
  • If you support refresh, you might invalidate the cache contextually (e.g. on DB connection failure).

Permissions and access

  • Whatever service is running Django (EC2, ECS, Lambda, etc.) should use an IAM role (instance profile or task role) with minimal permissions to read the required secrets.
  • Use least privilege: only allow secretsmanager:GetSecretValue (and perhaps ListSecrets) on the secrets your app needs.
  • If you use secret rotation, you may need secretsmanager:RotateSecret or permissions to invoke the rotation Lambda, depending on configuration.

Network & environment

  • If your Django app is running inside a VPC, ensure it has outbound access to Secrets Manager (which is a public endpoint or via VPC endpoints).
  • Optionally, use VPC interface endpoints (AWS PrivateLink) for Secrets Manager, so traffic stays within your VPC.
  • Ensure DNS / routing works so your app can call AWS APIs.

Libraries & Packages to Help

You don’t have to write everything from scratch. Several Django / Python libraries exist to help integrate with Secrets Manager:

Package What it provides / pros & cons
django-secrets-manager (LeeHanYeong) A library to help fetch secrets into Django settings. ([GitHub][2])
django-simple-secrets Provides a simple interface, caching, lazy loading for AWS Secrets Manager. ([PyPI][3])
govtech-csg-xcg-secretsmanager Offers integration for fetching DB credentials and rotating Django SECRET_KEY. Supports custom DB backends to transparently fetch. ([GitHub][4])
django-aws-secrets-env-setup A helper that sets environment variables from AWS Secrets Manager, so you can treat them as “normal” env variables. ([GitHub][5])

These libraries relieve some boilerplate (caching, error handling, rotation hooks). But they also add dependencies, so evaluate maintenance and stability.

Best Practices & Recommendations

Here’s a set of guidelines and tips to follow when integrating Secrets Manager with Django.

  1. Use the IAM role / instance profile approach Don’t embed AWS access keys in your code. Let your runtime environment assume an IAM role that has permission to access required secrets.

  2. Minimize secret requests with caching Don’t call AWS Secrets API per request. Use a local cache with an expiration (e.g. 15 min or more). Use client reuse (keep boto3 client), and exponential backoff / retry for transient failures.

  3. Graceful fallback & error handling

    • If fetching secret fails, either fail fast (so you see the error) or fallback to a safe default or earlier-known secret.
    • For database credentials, if the secret changes mid-flight, catch failures, clear cache, re-fetch, retry.
    • Handle timeouts, throttling (Secrets Manager has API limits).

  4. Secret rotation & zero downtime

    • Use AWS’s built-in rotation or custom Lambda to rotate secrets.
    • Ensure the Django app can pick up new credentials without manual redeploy (or gracefully swap).
    • You may need to build refresh logic at the database connector layer.

  5. Versioning and staging Use version stages (AWSCURRENT, AWSPENDING) so new secret versions can be tested before promoting. If your app is multi-instance, ensure all instances see the new secret smoothly.

  6. Limit secret scopes Don’t bundle all secrets into one JSON blob unless they always rotate together. Use separate secrets for independent concerns (DB credentials, third-party API keys, etc).

  7. Audit & monitoring

    • Enable CloudTrail to log secrets access.
    • Monitor GetSecretValue calls, errors, throttling.
    • On rotation failures, alert early.

  8. Local development & CI support

    • For local dev, you may fallback to local .env or AWS credentials in ~/.aws/credentials.
    • Consider a “mock” or stub secret fetcher in non-production environments.
    • Secure secrets (e.g. not commit them in tests or repos).

  9. Avoid doing heavy logic in settings.py While it’s common to fetch secrets inside settings.py, that logic runs at startup and can break introspection, builds, or non-web commands. Keep it lean, with timeouts and fallback. (Many discussions on whether it’s OK). ([Stack Overflow][6])

  10. Test rotation / failover Simulate secret changes, and verify that your application recovers gracefully (e.g. reconnect using new credentials).

Example Integration Pattern

Here’s a sample pattern for integrating AWS Secrets Manager for database credentials in Django.

1. Create a Secret in AWS

Name: myapp/prod/db Value (JSON):

{
  "username": "dbuser",
  "password": "supersecret",
  "engine": "postgres",
  "host": "mydb.xxxx.rds.amazonaws.com",
  "port": 5432,
  "dbname": "myappdb"
}

Ensure your Django app’s IAM role has secretsmanager:GetSecretValue permission for arn:aws:secretsmanager:region:acct-id:secret:myapp/prod/db.

2. Write a small helper to fetch & cache the secret

# secrets.py

import json
import threading
import time
import boto3
from botocore.exceptions import ClientError

_lock = threading.Lock()
_cache = {}
_CACHE_TTL = 300  # seconds

def get_secret(secret_name, region_name=None):
    """Fetch secret and cache it."""
    now = time.time()
    entry = _cache.get(secret_name)
    if entry and now < entry["expires_at"]:
        return entry["value"]
    # Acquire lock to refresh
    with _lock:
        entry = _cache.get(secret_name)
        if entry and now < entry["expires_at"]:
            return entry["value"]
        # Actually fetch
        session = boto3.session.Session()
        client = session.client("secretsmanager", region_name=region_name)
        resp = client.get_secret_value(SecretId=secret_name)
        secret_str = resp.get("SecretString")
        if secret_str is None:
            # You may need to handle binary secrets
            raise RuntimeError("Secret has no string value")
        data = json.loads(secret_str)
        _cache[secret_name] = {
            "value": data,
            "expires_at": time.time() + _CACHE_TTL
        }
        return data

This code:

  • Caches fetched secrets for 5 minutes
  • Uses locking to avoid duplicate fetches in concurrent threads
  • Raises errors if things go wrong

You can extend it (retry, fallback, clear cache) as needed.

3. Use in settings.py

# settings.py (or settings/base.py)

from .secrets import get_secret

# Optionally get AWS region from env
AWS_REGION = os.getenv("AWS_REGION", "ap-southeast-1")
DB_SECRET_NAME = os.getenv("DB_SECRET_NAME", "myapp/prod/db")

_db_secrets = get_secret(DB_SECRET_NAME, region_name=AWS_REGION)

DATABASES = {
    "default": {
        "ENGINE": "django.db.backends.postgresql",
        "NAME": _db_secrets["dbname"],
        "USER": _db_secrets["username"],
        "PASSWORD": _db_secrets["password"],
        "HOST": _db_secrets["host"],
        "PORT": _db_secrets.get("port", 5432),
        "OPTIONS": {
            # e.g. ssl mode if you require TLS
            "sslmode": "require",
        },
    }
}

# Similarly, load SECRET_KEY from a separate secret
DJANGO_SECRET_KEY = get_secret("myapp/prod/django", region_name=AWS_REGION)["SECRET_KEY"]
SECRET_KEY = DJANGO_SECRET_KEY

You could also fetch other secrets (e.g. third-party APIs) similarly.

4. Handle rotation / secret change gracefully

When rotation happens:

  • If a new credential version becomes current, your next fetch (after cache expiry) will see it
  • But if your DB connection is already open with old credentials, you may need to detect failures (e.g. “authentication failed”) and on such exception, clear the cache, re-fetch, and retry connection
  • Optionally, you can implement a background thread that periodically refreshes the cache (if your application is long-running).

Alternatively, libraries like govtech-csg-xcg-secretsmanager provide custom DB backends that perform such logic transparently. ([GitHub][4])

Tradeoffs, Challenges & Gotchas

  • Latency / cold start overhead: The first fetch adds latency. Mitigate via caching or fetching early (e.g. at app bootstrap).
  • Secret Manager availability / throttling: If the AWS API is temporarily unavailable or exceeds rate limits, your app must cope (timeouts, fallback).
  • Secret rotation complexity: If secrets change (e.g. DB password), existing connections may break. You need handling logic (clear cache, reconnect).
  • Testing / local development: In local dev or CI, you may not have Secrets Manager access. Provide fallback (e.g. local env files).
  • Doing too much in settings: If your settings.py imports dependencies or has network calls, tools (e.g. manage.py or django-admin) might fail or misbehave in contexts (e.g. collectstatic)
  • Thread safety & concurrency: If your app has multiple threads or processes, ensure your caching logic is thread/process-safe.
  • Secrets growth & cost: If you have many secrets or frequent rotations, watch costs and API usage.
  • IAM misconfiguration: Lack of correct IAM permissions is a usual cause of runtime errors.

Summary / Checklist

Here’s a distilled checklist for using AWS Secrets Manager in Django:

  1. Define secrets in AWS (structure as JSON, name them well)
  2. Grant minimal IAM permissions to your app’s runtime role
  3. Implement or use a library to fetch and cache secrets
  4. Integrate secret fetch into settings.py (or lazy load)
  5. Use retries, error handling, fallback mechanisms
  6. Plan for secret rotation and graceful application behavior on credential change
  7. For local/dev, provide alternative (env vars or mock)
  8. Monitor usage / audit accesses via CloudTrail
  9. Test secret rotation and failure recovery
  10. Keep secret logic simple and maintainable
Back to top