Finalizing the Production Stack and Deployment Considerations

Finalizing the Production Stack and Deployment Considerations

Welcome to the final chapter of our Docker Compose journey! So far, we’ve built a multi-service application, managed data, handled secrets, and implemented health checks. These are crucial steps, but moving from a development setup to a production-ready system requires a deeper look into operational hardening.

In this chapter, we will refine our Docker Compose stack to meet production standards. This involves configuring resource limits, enhancing logging, and performing security audits. By the end, you’ll have a more robust and observable application stack, ready for real-world deployment considerations. We’ll also discuss the boundaries of Docker Compose and where dedicated orchestration tools become necessary.

Project Overview: Hardening for Production

Our goal in this chapter is to transform our functional Docker Compose application into a more resilient, secure, and observable stack suitable for a single-host production environment. This involves applying a series of best practices that address common operational challenges.

By the end of this chapter, your Docker Compose application will feature:

• Resource Governance: CPU and memory limits applied to prevent resource exhaustion.
• Intelligent Logging: Configured log rotation to manage disk space and prepare for centralized logging.
• Enhanced Security: A read-only root filesystem for containers and an audit of your Docker environment using docker-bench-security.
• Deployment Awareness: An understanding of backup strategies, monitoring, and CI/CD integration for Docker Compose.

This final refinement will equip you with a robust foundation for deploying containerized applications, even if you eventually transition to more complex orchestration platforms.

Milestones: Production Hardening Checklist

To achieve our production-ready state, we’ll work through the following milestones:

1. Configure Resource Limits: Add CPU and memory limits to our services in docker-compose.yml.
2. Implement Log Rotation: Set up json-file logging driver options for local log file management.
3. Enable Read-Only Filesystems: Mark container filesystems as read-only for enhanced security.
4. Audit Docker Security: Use docker-bench-security to check the Docker host and containers against security benchmarks.
5. Review Deployment Strategy: Discuss considerations for deploying, updating, and backing up Docker Compose applications.

Architecture: Enhanced Production Stack

Transitioning an application to production means anticipating and mitigating failures, ensuring security, and gaining visibility into its operations. Our goal is to make our Docker Compose stack more resilient, secure, and easier to monitor.

While Docker Compose is excellent for defining and running multi-container applications on a single host, it’s not a full-fledged orchestrator like Kubernetes or Docker Swarm. Therefore, our “deployment considerations” will focus on preparing the stack for a single-host production environment or as a stepping stone to a larger orchestration system.

Here’s a high-level view of our enhanced production stack, showing how our local Docker Compose setup fits into a broader operational context:

This diagram illustrates how our Docker Compose stack (Web Application and Database services) resides on a Docker Host. External components like a Load Balancer, Monitoring System, and Log Aggregator interact with it, while a Security Scanner directly inspects the host and containers. This broader view emphasizes the surrounding systems needed for a truly production-grade setup.

Step-by-Step Implementation

Let’s apply these production-grade configurations to our docker-compose.yml file and the Docker environment. Remember that the current best practice for docker-compose.yml is to omit the explicit version field, as Docker Compose now adheres to the Compose Specification.

1. Setting Resource Limits

Uncontrolled resource consumption can lead to a “noisy neighbor” problem or even bring down the entire host. Docker Compose allows us to set CPU and memory limits for each service. This helps prevent a single runaway container from consuming all host resources.

Open your docker-compose.yml file (or docker-compose.production.yml if you created a separate file for production) and add a deploy section with resources.limits for your services.

File: docker-compose.yml

# docker-compose.yml

services:
  webapp:
    build:
      context: .
      dockerfile: Dockerfile
    ports:
      - "80:80"
    environment:
      # ... existing environment variables
    healthcheck:
      test: ["CMD-SHELL", "curl -f http://localhost/health || exit 1"]
      interval: 30s
      timeout: 10s
      retries: 3
    deploy: # This section applies resource limits
      resources:
        limits:
          cpus: '0.5' # Limit to 50% of one CPU core
          memory: 256M # Limit to 256 megabytes of RAM
    # ... other webapp configurations

  database:
    image: postgres:16-alpine # Using 16-alpine, latest as of 2026-05-22
    environment:
      POSTGRES_DB: ${DB_NAME}
      POSTGRES_USER: ${DB_USER}
      POSTGRES_PASSWORD: ${DB_PASSWORD}
    volumes:
      - db_data:/var/lib/postgresql/data
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U ${DB_USER} -d ${DB_NAME}"]
      interval: 10s
      timeout: 5s
      retries: 5
    deploy: # This section applies resource limits
      resources:
        limits:
          cpus: '0.25' # Limit to 25% of one CPU core
          memory: 512M # Limit to 512 megabytes of RAM
    # ... other database configurations

volumes:
  db_data:

Explanation of Decisions:

• deploy Key: While primarily used by Docker Swarm, Docker Compose respects the deploy.resources.limits configuration for standalone deployments.
• cpus: This limits the CPU cycles a container can consume. '0.5' means the container can use up to 50% of a single CPU core. This prevents a busy container from starving other processes on the host.
• memory: This sets a hard limit on the RAM available to the container. 256M for the web app and 512M for the database are initial estimates.
• Tradeoff: Setting limits too low can cause your application to perform poorly or crash due to resource starvation (e.g., Out Of Memory errors). It’s crucial to start with reasonable estimates and refine them based on actual application profiling under various load conditions.

2. Configuring Logging Drivers for Rotation

By default, Docker uses the json-file logging driver, which can lead to large log files filling up disk space over time. For production, you typically want to configure log rotation to manage file sizes or send logs to an external aggregator.

We’ll configure the json-file driver to rotate logs locally. This is a good intermediate step to prevent disk overflow before integrating with a full centralized logging solution (like ELK stack, Splunk, or cloud-native logging services).

File: docker-compose.yml

# docker-compose.yml

services:
  webapp:
    # ... existing webapp configurations
    logging: # Configure logging for the webapp service
      driver: "json-file"
      options:
        max-size: "10m" # Max size of the log file before rotation
        max-file: "5"   # Max number of log files to keep
    deploy:
      # ... existing deploy configurations

  database:
    # ... existing database configurations
    logging: # Configure logging for the database service
      driver: "json-file"
      options:
        max-size: "10m"
        max-file: "5"
    deploy:
      # ... existing deploy configurations

volumes:
  db_data:

Explanation of Decisions:

• logging.driver: Specifies which logging driver Docker should use. json-file is the default, but we’re explicitly configuring its options here. Docker supports various drivers, including syslog, journald, gelf (for Graylog), awslogs, and others for integration with external systems.
• max-size: When a log file reaches this size (e.g., 10m for 10 megabytes), Docker rotates it, creating a new log file.
• max-file: This defines the maximum number of rotated log files to keep (e.g., 5). Once this limit is reached, the oldest log file is deleted to make space for a new one.
• Real-world insight: While local log rotation helps prevent disk exhaustion, for true production observability, you’d centralize logs. This allows for easier searching, analysis, and correlation across multiple services and hosts.

3. Implementing a Read-Only Root Filesystem

For increased security, you can make a container’s root filesystem read-only. This is a powerful security control that prevents malicious actors or buggy processes from writing to arbitrary locations within the container’s filesystem (except for explicitly mounted volumes).

This requires careful planning: your application must be designed to write only to designated volumes, not to its own image layers or temporary internal paths.

File: docker-compose.yml

# docker-compose.yml

services:
  webapp:
    # ... existing webapp configurations
    read_only: true # Make the container's root filesystem read-only
    logging:
      # ... existing logging configurations
    deploy:
      # ... existing deploy configurations

  database:
    # ... existing database configurations
    read_only: true # Make the container's root filesystem read-only
    logging:
      # ... existing logging configurations
    deploy:
      # ... existing deploy configurations

volumes:
  db_data:

Explanation of Decisions:

• read_only: true: This setting ensures that the container cannot write to any part of its filesystem that isn’t explicitly mounted as a volume. This significantly reduces the attack surface and helps achieve immutability.
• Important: Test your application thoroughly with read_only: true. Many applications write temporary files, cache data, or logs to their internal filesystem by default. If your application attempts this, it will fail with a “Read-only file system” error. Ensure all necessary write operations are directed to Docker volumes. For our simple web app, it should work fine if all data is stored in the database. The database service already uses a volume for its data (db_data:/var/lib/postgresql/data), so its persistent writes will continue to function.

4. Auditing Docker Security with docker-bench-security

docker-bench-security is a script that checks your Docker host and containers against best practices defined in the CIS Docker Benchmark. It’s a critical tool for identifying potential security vulnerabilities in your Docker setup.

First, ensure you have git installed to clone the repository. Then, clone and run the tool.

# Clone the docker-bench-security repository
git clone https://github.com/docker/docker-bench-security.git

# Navigate into the directory
cd docker-bench-security

# Run the script. This script requires root privileges to perform comprehensive checks.
# It will analyze your Docker daemon, host configuration, and running containers.
sudo sh docker-bench-security.sh

Explanation of the Tool:

• CIS Docker Benchmark: This is a security benchmark provided by the Center for Internet Security (CIS) that offers a prescriptive guide for establishing a secure configuration posture for Docker.
• docker-bench-security.sh: This script automates checks against this benchmark.
• Output Interpretation: The script will output INFO, WARN, and PASS messages.
  • PASS: The check passed, indicating a secure configuration.
  • INFO: Informational message, often a recommendation that isn’t a direct security flaw but could be improved.
  • WARN: A potential security vulnerability or a configuration that doesn’t follow best practices. Always pay close attention to these and prioritize addressing them.
• Real-world insight: Regularly running docker-bench-security (e.g., as part of a CI/CD pipeline or scheduled job) on your Docker hosts is a crucial security practice. Addressing warnings promptly is key to maintaining a strong security posture.

Testing & Verification

After applying these production hardening changes, it’s essential to verify that everything works as expected and that our new configurations are active.

1. Restart your Docker Compose stack: First, bring down any existing services to ensure the new docker-compose.yml changes are applied. Then, bring them up in detached mode.

   docker compose down
   docker compose up -d

2. Verify Resource Limits: Use docker stats to see the real-time resource usage of your containers. You should observe that they stay within the limits you set, or at least don’t exceed them if under load.

   docker stats

   Look for the MEM USAGE / LIMIT and CPU % columns. The LIMIT column for memory should reflect your configuration (e.g., 256MiB for the webapp). For CPU, you’ll see a percentage, and if it consistently hits 50% for the webapp, it’s hitting its limit.

3. Verify Logging Configuration: Inspect the log files directly on your Docker host. Docker stores json-file logs in /var/lib/docker/containers/<container_id>/<container_id>-json.log.

   # Find your webapp container ID
   docker ps | grep webapp
   
   # Navigate to the logs directory (replace <your_webapp_container_id> with your actual ID)
   # Example path: /var/lib/docker/containers/a1b2c3d4e5f6g7h8i9j0k1l2m3n4o5p6q7r8s9t0/
   sudo ls -lh /var/lib/docker/containers/<your_webapp_container_id>/

   You should see multiple *.log files (e.g., *-json.log, *-json.log.1, etc.) if enough logs have been generated to trigger rotation. Their sizes should be capped around 10M.

4. Verify Read-Only Filesystem: Try to write a file inside one of your containers. This should fail if read_only: true is effective.

   # Get into your webapp container
   docker exec -it <your_webapp_container_id> sh
   
   # Try to create a file in the root filesystem
   touch /test.txt

   You should receive a “Read-only file system” error, confirming the security measure is active.

   # Exit the container
   exit

5. Review docker-bench-security Output: Re-run the docker-bench-security.sh script and carefully review its output. Specifically, look for any new WARN messages related to your container configurations. Addressing these warnings will further harden your setup.

Production Considerations

While our Docker Compose stack is significantly hardened, deploying to production involves more than just container configuration. These aspects are critical for reliable and maintainable systems.

Deployment Strategy

For single-host deployments, docker compose up -d is often sufficient for initial deployment. However, for updates, consider the following:

• Downtime: A simple docker compose down followed by docker compose up -d will cause downtime as old containers are stopped before new ones are started.
• Zero-Downtime Updates: Achieving true zero-downtime deployments with Docker Compose on a single host typically requires manual orchestration (e.g., running new containers on different ports, then switching a reverse proxy to the new containers) or moving to a full orchestrator. Platforms like Docker Swarm or Kubernetes offer built-in rolling updates and service discovery for seamless transitions.
• Automated Deployment: Integrate docker compose up -d into a CI/CD pipeline for consistent and repeatable deployments to your target host(s).

Backup and Recovery

Data loss is catastrophic. A robust backup and recovery strategy is non-negotiable.

• Volume Backups: Critical for persistent data stored in Docker volumes (like our db_data volume). You can back up volumes by running a temporary container that mounts the volume and copies its contents to a host path or cloud storage.

  # Example: Backup db_data volume to a local 'backups' directory
  # Ensure the database container is running so the volume is active.
  # Replace 'my_app_database-1' with your actual database container name.
  docker run --rm --volumes-from my_app_database-1 -v $(pwd)/backups:/backup alpine tar cvf /backup/db_backup_$(date +%F).tar /var/lib/postgresql/data

  This command creates a tar archive of your database data directory within a backups folder in your current directory.
• Configuration Backups: Always keep your docker-compose.yml, .env files, Dockerfiles, and any other configuration files under version control (e.g., Git). This ensures you can rebuild your environment from scratch.

Monitoring and Alerting

Visibility into your application’s health and performance is vital.

• Host-level Monitoring: Track key metrics of your Docker host itself: CPU utilization, memory usage, disk I/O, network throughput. Tools like node_exporter with Prometheus and Grafana are common for this.
• Container-level Monitoring: Collect specific metrics for individual containers, such as application request rates, error rates, latency, and resource consumption. Docker provides docker stats, but for production, solutions like cAdvisor, Prometheus, and Grafana offer more depth.
• Log Aggregation: Sending all container logs to a centralized system (e.g., ELK Stack, Grafana Loki, Splunk, cloud logging services like AWS CloudWatch or Google Cloud Logging) is crucial. This enables easier searching, analysis, and correlation of events across multiple services.
• Alerting: Configure alerts based on critical metrics (e.g., high CPU usage, low disk space, increased application error rates, health check failures) to notify your operations team proactively.

CI/CD Integration

Integrating your Docker Compose stack into a Continuous Integration/Continuous Deployment (CI/CD) pipeline automates the process of building, testing, and deploying your application.

1. Build Phase: The CI pipeline automatically builds Docker images for your application components whenever code changes are pushed.
2. Test Phase: Automated tests (unit, integration, end-to-end) are run against the newly built images to ensure code quality and functionality.
3. Deployment Phase: Upon successful testing, the CD pipeline uses docker compose up -d (or orchestrator commands) to deploy the new images to your staging or production environment. This ensures consistency and reduces human error.

Common Issues & Solutions

Even with careful planning, production deployments can encounter issues. Here are a few common problems related to the configurations we’ve implemented:

1. Container Crashes Due to Resource Limits:

   • Issue: Your container starts, then unexpectedly exits, often with an “Out Of Memory” (OOM) error, or experiences severe performance degradation due to CPU throttling.
   • Debugging: Check docker logs <container_name> for OOM errors. Use docker stats to monitor resource usage and see if containers are hitting their LIMIT frequently.
   • Solution: Increase cpus or memory limits in deploy.resources.limits. The best way to determine appropriate values is through application profiling and load testing under realistic conditions.

2. “Read-Only File System” Errors in Production:

   • Issue: After enabling read_only: true, your application fails to start or crashes because it attempts to write temporary files, cache data, or logs to its internal filesystem (which is now read-only).
   • Debugging: The error message Read-only file system will appear in docker logs <container_name>. You’ll need to identify the specific path the application is trying to write to.
   • Solution: Identify where the application needs to write. Mount specific paths as volumes (e.g., - /tmp:/tmp for temporary files, or - ./logs:/app/logs for application logs) or configure the application to write to an existing volume. For logs, ensure they are directed to stdout/stderr so Docker’s logging driver can capture them.

3. Ignoring docker-bench-security Warnings:

   • Issue: The security script outputs WARN messages (e.g., “Host is not configured to limit memory usage,” “Container is running as root”) but these are ignored, leaving potential vulnerabilities.
   • Debugging: Re-run sudo sh docker-bench-security.sh and carefully read each WARN message.
   • Solution: Treat WARN messages from docker-bench-security as actionable items. Research each warning, understand its implication for your specific environment, and implement the recommended fix. Regular security audits and continuous improvement are crucial for maintaining a strong security posture.

Summary & Next Step

Congratulations! You’ve successfully built and hardened a multi-service application stack using Docker and Docker Compose, incorporating many production best practices.

In this chapter, we:

• Configured resource limits for CPU and memory to prevent resource exhaustion and improve stability.
• Implemented local log rotation for better log file management, preventing disk space issues.
• Enhanced container security by making filesystems read-only, reducing the attack surface.
• Learned how to use docker-bench-security to audit our Docker environment against best practices.
• Discussed broader production considerations like deployment strategies, backups, monitoring, and CI/CD integration.

You now have a solid foundation for deploying containerized applications. While Docker Compose is excellent for single-host deployments and development, scaling beyond a single machine or requiring advanced orchestration features (like automatic load balancing, self-healing, rolling updates without manual intervention, or complex service discovery) typically necessitates moving to platforms like Docker Swarm or Kubernetes.

Your next step might be to explore these more advanced orchestration platforms to deploy your hardened Docker images at scale. Alternatively, you could delve deeper into specific areas like setting up a centralized logging solution (e.g., Loki with Grafana), implementing advanced monitoring with Prometheus, or integrating your stack into a full-fledged CI/CD pipeline using tools like GitHub Actions, GitLab CI, or Jenkins.

This page is AI-assisted and reviewed. It references official documentation and recognized resources where relevant.

References

• Docker Documentation: https://docs.docker.com/
• Compose Specification: https://docs.docker.com/compose/compose-file/
• docker-bench-security GitHub Repository: https://github.com/docker/docker-bench-security
• PostgreSQL Docker Image: https://hub.docker.com/_/postgres
• CIS Docker Benchmark: https://www.cisecurity.org/benchmark/docker