CODE WITH SIBIN

Solving Real Problems with Real Code


Django AWS Secrets Manager Integration
admin
April 30, 2025
Django

This comprehensive guide will walk you through integrating AWS Secrets Manager with a Django application from scratch, covering everything from initial setup to production deployment.

Table of Contents

  1. Prerequisites
  2. AWS Setup
  3. Django Project Setup
  4. Basic Integration Approach
  5. Advanced Integration Patterns
  6. Local Development Considerations
  7. Security Best Practices
  8. Deployment Strategies
  9. Monitoring and Rotation
  10. Troubleshooting

Prerequisites

Before starting, ensure you have:

  • An AWS account with appropriate permissions
  • Python 3.8+ installed
  • Django 3.2+ project
  • AWS CLI configured with credentials
  • Basic understanding of IAM roles and policies

AWS Setup

1. Create an IAM Policy for Secrets Manager Access

Create a policy that allows your application to access Secrets Manager:

{
    "Version": "2012-10-17",
    "Statement": [
        {
            "Effect": "Allow",
            "Action": [
                "secretsmanager:GetSecretValue",
                "secretsmanager:DescribeSecret",
                "secretsmanager:ListSecrets"
            ],
            "Resource": "*"
        }
    ]
}

2. Create an IAM Role for Your Application

  1. Go to IAM > Roles > Create role
  2. Select "AWS service" and choose "EC2" (or appropriate service for your deployment)
  3. Attach the policy created above
  4. Name the role (e.g., DjangoAppSecretsManagerAccess)

3. Create a Secret in AWS Secrets Manager

  1. Navigate to AWS Secrets Manager
  2. Click "Store a new secret"
  3. Select "Other type of secrets"
  4. Add your key-value pairs (e.g., DATABASE_URL, SECRET_KEY)
  5. Name your secret (e.g., prod/django/app)
  6. Configure rotation if needed (we'll cover this later)
  7. Complete the secret creation

Django Project Setup

1. Install Required Packages

pip install boto3 django-environ

2. Create a Secrets Manager Utility Module

Create utils/aws_secrets.py:

import json
import boto3
from botocore.exceptions import ClientError
from django.conf import settings
import logging

logger = logging.getLogger(__name__)

class SecretsManager:
    def __init__(self):
        self.client = boto3.client(
            'secretsmanager',
            region_name=getattr(settings, 'AWS_REGION', 'us-east-1')
        )
        self.secret_name = getattr(settings, 'AWS_SECRETS_NAME', None)
        
    def get_secret(self, secret_name=None):
        """Retrieve secret from AWS Secrets Manager"""
        secret_name = secret_name or self.secret_name
        
        if not secret_name:
            raise ValueError("No secret name configured")
            
        try:
            response = self.client.get_secret_value(SecretId=secret_name)
        except ClientError as e:
            logger.error(f"Error retrieving secret {secret_name}: {e}")
            raise
            
        if 'SecretString' in response:
            secret = response['SecretString']
            return json.loads(secret)
            
        return None

def load_secrets_to_env():
    """Load secrets into environment variables"""
    secrets_manager = SecretsManager()
    secrets = secrets_manager.get_secret()
    
    if secrets:
        for key, value in secrets.items():
            os.environ[key] = str(value)

3. Configure Django Settings

Modify your settings.py:

import os
from utils.aws_secrets import load_secrets_to_env

# Try to load secrets from AWS if in production
if os.getenv('AWS_EXECUTION_ENV') or os.getenv('AWS_SECRETS_NAME'):
    try:
        load_secrets_to_env()
    except Exception as e:
        print(f"Warning: Failed to load AWS secrets: {e}")

# Now use environment variables as usual
SECRET_KEY = os.getenv('SECRET_KEY', 'django-insecure-default-key-for-dev')

DATABASES = {
    'default': {
        'ENGINE': 'django.db.backends.postgresql',
        'NAME': os.getenv('DB_NAME'),
        'USER': os.getenv('DB_USER'),
        'PASSWORD': os.getenv('DB_PASSWORD'),
        'HOST': os.getenv('DB_HOST'),
        'PORT': os.getenv('DB_PORT', '5432'),
    }
}

# Optional AWS-specific settings
AWS_REGION = os.getenv('AWS_REGION', 'us-east-1')
AWS_SECRETS_NAME = os.getenv('AWS_SECRETS_NAME', 'prod/django/app')

Basic Integration Approach

1. Lazy Loading Approach

For applications where you don't want to load secrets at startup:

from django.conf import settings
from utils.aws_secrets import SecretsManager

def get_database_config():
    """Lazy-load database config from Secrets Manager"""
    if not hasattr(settings, '_database_config'):
        secrets_manager = SecretsManager()
        secrets = secrets_manager.get_secret()
        settings._database_config = {
            'ENGINE': 'django.db.backends.postgresql',
            'NAME': secrets['DB_NAME'],
            'USER': secrets['DB_USER'],
            'PASSWORD': secrets['DB_PASSWORD'],
            'HOST': secrets['DB_HOST'],
            'PORT': secrets['DB_PORT'],
        }
    return settings._database_config

DATABASES = {
    'default': get_database_config()
}

2. Django Management Command for Secret Validation

Create management/commands/validate_secrets.py:

from django.core.management.base import BaseCommand
from utils.aws_secrets import SecretsManager
from django.conf import settings

class Command(BaseCommand):
    help = 'Validate that required secrets are available in AWS Secrets Manager'
    
    def handle(self, *args, **options):
        required_secrets = [
            'SECRET_KEY',
            'DB_NAME',
            'DB_USER',
            'DB_PASSWORD',
            'DB_HOST',
        ]
        
        secrets_manager = SecretsManager()
        secrets = secrets_manager.get_secret()
        
        missing = []
        for secret in required_secrets:
            if secret not in secrets:
                missing.append(secret)
                
        if missing:
            self.stdout.write(self.style.ERROR(
                f"Missing required secrets: {', '.join(missing)}"
            ))
            return
            
        self.stdout.write(self.style.SUCCESS(
            "All required secrets are available"
        ))

Advanced Integration Patterns

1. Multiple Environment Support

Extend the SecretsManager class to handle multiple environments:

class SecretsManager:
    def __init__(self, environment=None):
        self.client = boto3.client(
            'secretsmanager',
            region_name=getattr(settings, 'AWS_REGION', 'us-east-1')
        )
        self.environment = environment or os.getenv('DJANGO_ENV', 'development')
        
    def get_secret_name(self):
        """Determine secret name based on environment"""
        prefix_map = {
            'development': 'dev',
            'staging': 'stage',
            'production': 'prod'
        }
        prefix = prefix_map.get(self.environment, 'dev')
        return f"{prefix}/django/app"
        
    def get_secret(self, secret_name=None):
        secret_name = secret_name or self.get_secret_name()
        # Rest of the implementation...

2. Secret Caching with Refresh

from datetime import datetime, timedelta

class CachedSecretsManager(SecretsManager):
    def __init__(self, *args, **kwargs):
        self._cache = None
        self._last_refresh = None
        self.cache_ttl = timedelta(minutes=5)  # Refresh every 5 minutes
        super().__init__(*args, **kwargs)
        
    def get_secret(self, secret_name=None):
        now = datetime.now()
        if (self._cache is None or 
            self._last_refresh is None or
            (now - self._last_refresh) > self.cache_ttl):
            self._cache = super().get_secret(secret_name)
            self._last_refresh = now
        return self._cache

3. Django Middleware for Secret Rotation

Create middleware to handle secret rotation without app restart:

class SecretsRotationMiddleware:
    def __init__(self, get_response):
        self.get_response = get_response
        self.secrets_manager = CachedSecretsManager()
        
    def __call__(self, request):
        # Refresh secrets on every request (cache handles TTL)
        try:
            secrets = self.secrets_manager.get_secret()
            for key, value in secrets.items():
                os.environ[key] = str(value)
        except Exception as e:
            logging.error(f"Failed to refresh secrets: {e}")
            
        response = self.get_response(request)
        return response

Local Development Considerations

1. Local Secrets Fallback

Modify your settings.py to support local development:

# settings.py

# Try AWS secrets first
if os.getenv('AWS_SECRETS_NAME') or os.getenv('AWS_EXECUTION_ENV'):
    try:
        load_secrets_to_env()
    except Exception as e:
        print(f"Warning: Failed to load AWS secrets: {e}")

# Fallback to local .env file
if not os.getenv('SECRET_KEY'):
    from dotenv import load_dotenv
    load_dotenv()

# Final fallback for required settings
SECRET_KEY = os.getenv('SECRET_KEY', 'django-insecure-default-key-for-dev')

2. Local Testing with LocalStack

For local AWS services emulation:

  1. Install LocalStack:
pip install localstack
  1. Update your SecretsManager to support LocalStack:
class SecretsManager:
    def __init__(self, *args, **kwargs):
        self.use_localstack = os.getenv('USE_LOCALSTACK', 'false').lower() == 'true'
        if self.use_localstack:
            self.client = boto3.client(
                'secretsmanager',
                region_name=os.getenv('AWS_REGION', 'us-east-1'),
                endpoint_url=os.getenv('LOCALSTACK_ENDPOINT', 'http://localhost:4566'),
                aws_access_key_id='test',
                aws_secret_access_key='test'
            )
        else:
            self.client = boto3.client(
                'secretsmanager',
                region_name=os.getenv('AWS_REGION', 'us-east-1')
            )
        # Rest of initialization...

Security Best Practices

  1. Least Privilege Principle:
    • Restrict IAM roles to only necessary secrets
    • Use resource-level permissions in IAM policies
  2. Secret Naming Conventions:
    • Use paths like env/application/component (e.g., prod/django/database)
    • Avoid sensitive information in secret names
  3. Encryption:
    • Always enable encryption for secrets
    • Use AWS KMS customer-managed keys for additional control
  4. Audit and Monitoring:
    • Enable AWS CloudTrail for API calls monitoring
    • Set up CloudWatch alarms for unauthorized access attempts
  5. Secret Rotation:
    • Implement automatic rotation for credentials
    • Test rotation in staging before production

Deployment Strategies

1. AWS Elastic Beanstalk

  1. Add a .ebextensions/secrets.config file:
option_settings:
  aws:elasticbeanstalk:application:environment:
    AWS_SECRETS_NAME: prod/django/app
    AWS_REGION: us-east-1
  1. Ensure your instance profile has the necessary permissions

2. AWS ECS/Fargate

  1. Add environment variables to your task definition:
{
  "environment": [
    {
      "name": "AWS_SECRETS_NAME",
      "value": "prod/django/app"
    },
    {
      "name": "AWS_REGION",
      "value": "us-east-1"
    }
  ]
}
  1. Attach the IAM role to your task execution role

3. Kubernetes (EKS)

  1. Use AWS Secrets Manager CSI Driver:
apiVersion: secrets-store.csi.x-k8s.io/v1
kind: SecretProviderClass
metadata:
  name: django-secrets
spec:
  provider: aws
  parameters:
    objects: |
      - objectName: "prod/django/app"
        objectType: "secretsmanager"
  1. Mount secrets as volumes or environment variables

Monitoring and Rotation

1. Implementing Secret Rotation

  1. Create a Lambda function for rotation:
import boto3
import json

def lambda_handler(event, context):
    client = boto3.client('secretsmanager')
    
    # Get the current secret
    current_secret = client.get_secret_value(
        SecretId=event['SecretId']
    )
    
    # Generate new credentials (implementation depends on your service)
    new_credentials = generate_new_credentials()
    
    # Update the secret
    client.update_secret(
        SecretId=event['SecretId'],
        SecretString=json.dumps(new_credentials)
    )
    
    return {
        'statusCode': 200,
        'body': json.dumps('Secret rotation initiated')
    }
  1. Configure rotation in Secrets Manager with this Lambda function

2. Monitoring Setup

  1. CloudWatch Alarms for:
    • Failed secret access attempts
    • Rotation failures
    • Unauthorized access attempts
  2. SNS notifications for critical events

Troubleshooting

Common Issues and Solutions

  1. Permission Denied Errors:
    • Verify IAM role is attached to your EC2/ECS instance
    • Check policy permissions
    • Ensure the secret exists in the correct region
  2. Secret Not Found:
    • Verify secret name and region
    • Check for typos
    • Ensure the secret exists and has a value
  3. Performance Issues:
    • Implement caching as shown in advanced patterns
    • Consider increasing the cache TTL for less critical secrets
  4. Local Development Problems:
    • Ensure fallback to .env files works
    • Verify LocalStack configuration if used
    • Check AWS credentials aren't being unintentionally loaded locally
  5. Rotation Failures:
    • Test rotation in staging first
    • Ensure Lambda has proper permissions
    • Verify the Lambda can access all required services

Conclusion

Integrating AWS Secrets Manager with Django provides a secure and scalable way to manage your application's sensitive configuration. By following this guide, you've learned:

  1. How to set up AWS Secrets Manager and configure proper IAM permissions
  2. Multiple approaches to integrate secrets with your Django application
  3. Advanced patterns for caching, rotation, and environment-specific configurations
  4. Local development strategies and security best practices
  5. Deployment considerations for various AWS services
  6. Monitoring and troubleshooting techniques

This implementation provides a robust foundation that you can adapt to your specific requirements while maintaining security and operational best practices.

🔖Tags: Django
Share on Facebook
Share on Twitter

Leave a Reply

Your email address will not be published. Required fields are marked *