Orchestrating Services with Docker Compose

Modern applications rarely consist of a single, monolithic service. Instead, they are typically composed of multiple interconnected components: a web frontend, a backend API, a database, perhaps a caching layer, and other auxiliary services. Manually managing the lifecycle, networking, and configuration of these interconnected containers can quickly become complex, time-consuming, and prone to error.

This chapter introduces Docker Compose, a powerful command-line tool designed to simplify the definition and management of multi-container Docker applications. By using a single YAML file, you can declaratively define your entire application stack, ensuring consistency and reproducibility across development, testing, and even production environments.

By the end of this milestone, you will have a fully functional, two-service web application stack—a Python Flask web application and a PostgreSQL database—all orchestrated and managed by Docker Compose. This setup will demonstrate how services communicate securely over an internal network, laying the groundwork for more advanced configurations and production-ready deployments. You’ll be able to bring your entire application stack up and tear it down with simple, consistent commands.

Project Overview: A Multi-Service Web Application

In this chapter, we are building a foundational multi-service application stack. The goal is to deploy a simple web application that connects to a database, simulating a common real-world scenario. This setup will serve as the base for implementing further production-ready practices in subsequent chapters.

The core components we will integrate are:

Web Service: A lightweight Python Flask application that exposes an HTTP endpoint. Its primary function in this chapter is to demonstrate connectivity to the database.
Database Service: A PostgreSQL database instance responsible for persistent data storage.

These services will be defined in a docker-compose.yml file, which orchestrates their deployment, networking, and initial configuration.

Tech Stack: Docker Compose for Orchestration

Our primary tool for this chapter is Docker Compose.

Docker Engine: (Version unknown as of 2026-05-22). We assume you have a working Docker Engine installation (on Linux, macOS, or Windows with WSL2).
Docker Compose: Adheres to the Compose Specification (as of 2026-05-22). This specification defines the docker-compose.yml file format. Modern Docker Compose installations automatically follow this specification, meaning you no longer need to specify a version field within your docker-compose.yml file. This ensures forward compatibility and access to the latest features.
Python 3.11: For the web application.
Flask 2.3.3: The web framework.
PostgreSQL 16: The database server.

Build Plan: Orchestrating Your Services

To achieve our goal of a multi-service application, we will follow these steps:

Review existing application files: Ensure our Flask app and its Dockerfile are ready.
Define docker-compose.yml: Create the central configuration file for our services.
Configure web service: Detail how our Flask app will be built and exposed.
Configure db service: Set up the PostgreSQL database container.
Define custom network: Establish an isolated network for inter-service communication.
Add data volume definition: Prepare for persistent database storage (though detailed volume management is next chapter).
Deploy and verify: Bring up the stack and confirm connectivity.

Architecture: Multi-Service Interaction

Our application architecture is straightforward for this milestone, focusing on a client-server pattern with a database backend.

flowchart TD User -->|HTTP Request| Web_Service[Web Service] Web_Service -->|Database Query| Database_Service[Database Service]

Explanation:

A User sends an HTTP request to the Web Service (our Flask application).
The Web Service attempts to connect to and query the Database Service (PostgreSQL).
All communication between Web_Service and Database_Service happens over an internal Docker network, ensuring isolation.

Project Directory Structure

To keep our project organized, we’ll maintain the following structure. This assumes you’ve already set up the web/Dockerfile and web/app.py from previous chapters.

your-project/
├── docker-compose.yml       # Our orchestration file
├── web/
│   ├── Dockerfile           # Dockerfile for the web service
│   ├── app.py               # Python Flask application
│   └── requirements.txt     # Python dependencies
└── .env                     # (Will be added in a later chapter for secrets)

Step-by-Step Implementation: Defining `docker-compose.yml`

Let’s create our docker-compose.yml file and define our services.

1. Prepare Web Application Files

Ensure your web directory contains the necessary files. If you’re starting fresh, create these files:

your-project/web/requirements.txt:

Flask==2.3.3
psycopg2-binary==2.9.9

Decision: We use psycopg2-binary for simpler installation without requiring local PostgreSQL development headers, which is often preferred in containerized environments for faster builds.

your-project/web/app.py:

# your-project/web/app.py
import os
from flask import Flask
import psycopg2
from psycopg2 import OperationalError

app = Flask(__name__)

@app.route('/')
def hello():
    db_url = os.environ.get("DATABASE_URL")
    if not db_url:
        return "Error: DATABASE_URL environment variable not set.", 500

    try:
        # Attempt to connect to the database
        conn = psycopg2.connect(db_url)
        cur = conn.cursor()
        cur.execute("SELECT 1") # A simple query to check connection
        cur.close()
        conn.close()
        return "Hello from Flask! Connected to database successfully!"
    except OperationalError as e:
        # Specific error for database connection issues
        return f"Hello from Flask! Failed to connect to database: {e}", 500
    except Exception as e:
        # Catch any other unexpected errors
        return f"Hello from Flask! An unexpected error occurred: {e}", 500

if __name__ == '__main__':
    app.run(host='0.0.0.0', port=8000)

Explanation: This Flask app attempts to connect to a PostgreSQL database using the DATABASE_URL environment variable. It then performs a simple query to verify the connection. Error handling is included to provide clearer messages if the database connection fails.

your-project/web/Dockerfile:

# your-project/web/Dockerfile

# Stage 1: Build dependencies
FROM python:3.11-slim-bullseye AS builder

WORKDIR /app

# Install build dependencies required for psycopg2-binary
# These are only needed for the build stage, not the final runtime image
RUN apt-get update && apt-get install -y --no-install-recommends \
    build-essential \
    libpq-dev \
    gcc \
    && rm -rf /var/lib/apt/lists/*

COPY requirements.txt .
RUN pip install --no-cache-dir -r requirements.txt

# Stage 2: Create final image
FROM python:3.11-slim-bullseye

WORKDIR /app

# Copy only runtime dependencies from builder stage
COPY --from=builder /usr/local/lib/python3.11/site-packages /usr/local/lib/python3.11/site-packages
# Copy Flask executable if it's installed to a custom bin path (depends on pip env)
# For simplicity and robustness, ensure Flask is in site-packages and invoked via python -m flask
# If flask command is not found, you might need to adjust this or rely on python -m flask
# For standard installations, this line might not be strictly necessary as Flask is found via PATH in site-packages
# COPY --from=builder /usr/local/bin/flask /usr/local/bin/flask

COPY app.py .

# Expose the port the app runs on
EXPOSE 8000

# Set environment variables for the application
ENV FLASK_APP=app.py
ENV FLASK_RUN_HOST=0.0.0.0

# Command to run the application
# Use python -m flask run for better compatibility with different Flask installations
CMD ["python", "-m", "flask", "run", "--port=8000"]

Decision: This Dockerfile uses a multi-stage build. The builder stage includes build-essential and libpq-dev to compile psycopg2-binary. The final stage, python:3.11-slim-bullseye, is much smaller as it only copies the compiled Python packages and the application code, discarding unnecessary build tools. This is a crucial production practice for smaller, more secure images.
Modern Flask execution: Changed CMD ["flask", "run", ...] to CMD ["python", "-m", "flask", "run", ...] for better reliability in diverse Python environments and container setups.

2. Create the `docker-compose.yml` File

In the root of your your-project directory, create a new file named docker-compose.yml.

# your-project/docker-compose.yml
services:
  web:
    build: ./web
    ports:
      - "8000:8000"
    depends_on:
      - db
    environment:
      DATABASE_URL: postgresql://user:password@db:5432/mydatabase
    networks:
      - app_network

  db:
    image: postgres:16-alpine # Using a specific, lightweight version
    environment:
      POSTGRES_DB: mydatabase
      POSTGRES_USER: user
      POSTGRES_PASSWORD: password
    volumes:
      - db_data:/var/lib/postgresql/data # Persistent data volume
    networks:
      - app_network

networks:
  app_network:
    driver: bridge # Default driver, explicitly stated for clarity

volumes:
  db_data: # Define the named volume for database persistence

🧠 Important: Notice the absence of a version field at the top of the docker-compose.yml file. As of 2026-05-22, this is the recommended practice for adhering to the Compose Specification. Docker Compose automatically detects the latest specification when this field is omitted.

3. Understanding Each Section of `docker-compose.yml`

Let’s break down what each part of this configuration does.

`services` Block

This top-level key defines the individual containers that form your application stack. Each key under services (e.g., web, db) represents a service. Docker Compose will manage these as separate containers.

`web` Service Configuration

  web:
    build: ./web
    ports:
      - "8000:8000"
    depends_on:
      - db
    environment:
      DATABASE_URL: postgresql://user:password@db:5432/mydatabase
    networks:
      - app_network

build: ./web: This instruction tells Docker Compose to build the image for the web service using the Dockerfile located in the ./web directory, relative to docker-compose.yml.
ports: - "8000:8000": This maps port 8000 on your host machine to port 8000 inside the web container. This allows you to access your web application from your host machine’s browser at http://localhost:8000.
depends_on: - db: This specifies that the web service has a dependency on the db service. Docker Compose will start the db container before the web container.
- ⚠️ What can go wrong: While depends_on ensures startup order, it does not wait for the dependent service to be fully ready (e.g., the database accepting connections). This is a common source of “connection refused” errors during startup. We will implement robust health checks in a future chapter to address this.
environment:: This section sets environment variables inside the web container.
- DATABASE_URL: postgresql://user:password@db:5432/mydatabase: This URL configures the Flask application to connect to our PostgreSQL database. The hostname db is resolved by Docker Compose to the db service’s container IP within the app_network. The port 5432 is the default for PostgreSQL.
networks: - app_network: This assigns the web service to our custom app_network. Services on the same network can communicate with each other using their service names as hostnames.

`db` Service Configuration

  db:
    image: postgres:16-alpine # Using a specific, lightweight version
    environment:
      POSTGRES_DB: mydatabase
      POSTGRES_USER: user
      POSTGRES_PASSWORD: password
    volumes:
      - db_data:/var/lib/postgresql/data # Persistent data volume
    networks:
      - app_network

image: postgres:16-alpine: This specifies that the db service should use the official postgres Docker image, specifically version 16 based on the lightweight alpine Linux distribution. Using specific, lightweight base images is a good production practice to reduce image size and attack surface.
environment:: These variables are used by the PostgreSQL image to initialize the database when the container starts for the first time.
- POSTGRES_DB, POSTGRES_USER, POSTGRES_PASSWORD: These define the default database name, username, and password.
- ⚠️ What can go wrong: Hardcoding sensitive information like passwords directly in docker-compose.yml is a significant security risk in production. For demonstration purposes, we are doing it now, but we will cover proper secrets management using .env files and Docker secrets in upcoming chapters.
volumes: - db_data:/var/lib/postgresql/data: This line mounts a named volume called db_data to the /var/lib/postgresql/data directory inside the db container. This is crucial for data persistence. Without it, all your database data would be lost every time the db container is removed. We’ll explore volumes in detail in the next chapter.
networks: - app_network: Assigns the db service to the app_network, allowing it to communicate with the web service.

`networks` Block

This top-level key defines custom networks for your services.

networks:
  app_network:
    driver: bridge

app_network:: This is the name of our custom network.
driver: bridge: Specifies the network driver. bridge is the default and most common driver for single-host Docker setups. It creates a private, isolated internal network that services can join, providing secure communication channels.

`volumes` Block

This top-level key defines named volumes for data persistence.

volumes:
  db_data:

db_data:: This line defines a named volume called db_data. Docker manages the actual storage location on the host system. Named volumes are the preferred way to persist data in Docker as they are managed by Docker, easier to back up, and don’t tie your data to specific host paths.

Testing & Verification

Now that our docker-compose.yml file and application code are ready, let’s deploy our stack and verify its functionality.

Navigate to your project root: Open your terminal and ensure you are in the your-project directory, where docker-compose.yml is located.
```
cd your-project
```
Start the services: Use docker compose up to build (if necessary) and start all services defined in your docker-compose.yml file. The -d flag runs them in detached mode, meaning the containers will run in the background, freeing up your terminal.
```
docker compose up -d
```
- Explanation: This command orchestrates the entire startup process:
  - It first builds the web service image (if it doesn’t exist or if its Dockerfile or context has changed).
  - It then pulls the postgres:16-alpine image (if not already local).
  - It creates the app_network and the db_data volume.
  - Finally, it starts both the db and web containers in the defined order.

Verify service status: Check that your containers are running as expected.

docker compose ps

You should see output similar to this, indicating both services are running:

NAME                 COMMAND                  SERVICE             STATUS              PORTS
your-project-web-1   "python -m flask run…"   web                 running             0.0.0.0:8000->8000/tcp, :::8000->8000/tcp
your-project-db-1    "docker-entrypoint.s…"   db                  running             5432/tcp

Verification: Confirm that both the web and db services show a running status.

Inspect logs: Review the logs of your services to ensure they started without errors and are behaving as expected.
```
docker compose logs web
docker compose logs db
```
- Verification: You should see Flask startup messages for the web service and PostgreSQL initialization/startup messages for the db service. Look for any ERROR or FAIL messages.
Access the web application: Open your web browser and navigate to http://localhost:8000.
- Expected Output: You should see the message: Hello from Flask! Connected to database successfully!
- Verification: If you see this message, your Flask application successfully started, connected to the PostgreSQL database, and performed a simple query. This confirms inter-service communication and basic database functionality. If you receive an error, it indicates a problem with the database connection from the Flask app or the database itself.
Stop and remove services: When you’re finished experimenting, you can stop and remove all services, networks, and optionally volumes defined in your docker-compose.yml with:
```
docker compose down
```
- Explanation: This command gracefully stops the running containers, removes them, and also removes the custom network app_network. By default, docker compose down does not remove named volumes (like db_data), which is a safety measure to prevent accidental data loss.
- 🔥 Pro tip: To remove volumes as well (e.g., for a clean slate during development), you would explicitly use docker compose down --volumes. Be cautious with this command in production-like environments, as it will delete your persistent data.

Production Considerations

While Docker Compose is excellent for development and testing, some aspects require careful thought for production deployments:

Network Isolation: Docker Compose automatically creates a dedicated, isolated network for your services. This is a fundamental security practice, as it prevents your database from being directly accessible from the host network or the internet unless explicitly exposed via ports (which we deliberately did not do for db).
Environment Variables and Secrets: We currently hardcode database credentials in docker-compose.yml. This is unacceptable for production environments. Sensitive information should be managed using:
- .env files (for local development, handled in the next chapter).
- Docker Secrets (for Docker Swarm, more secure than .env files).
- Dedicated secrets management systems (e.g., HashiCorp Vault, AWS Secrets Manager, Azure Key Vault) for robust, large-scale deployments.
Resource Limits: For production, it’s critical to define CPU and memory limits for your containers using the deploy.resources.limits and deploy.resources.reservations keys in your docker-compose.yml (part of the Compose Specification). This prevents a single misbehaving service from consuming all host resources and impacting other services. We will cover this in a later chapter.
Logging: While docker compose logs is useful for development, production systems require centralized logging. You would configure your services to send logs to a dedicated logging solution (e.g., ELK stack, Splunk, cloud logging services) for aggregation, analysis, and alerting.
Health Checks: As noted, depends_on only guarantees startup order, not readiness. In production, implementing healthcheck configurations for your services is vital to ensure dependent services only connect when a service is truly ready to handle requests. This topic will be covered in a dedicated chapter.

Common Issues & Solutions

“Service ‘db’ failed to build” or “Error response from daemon: pull access denied”:
- Issue: Docker couldn’t pull the postgres:16-alpine image, or there’s a typo in the image name.
- Solution: Double-check the image name and tag in your docker-compose.yml. Ensure your machine has an active internet connection and that Docker Hub is accessible. A simple docker pull postgres:16-alpine can help diagnose connectivity issues.
Web app shows “Failed to connect to database”:
- Issue: The web service started before the db service was fully initialized and ready to accept connections, or the DATABASE_URL is incorrect.
- Solution:
  - Verify DATABASE_URL: Double-check that the DATABASE_URL in docker-compose.yml for the web service correctly references the db service name, user, password, and database defined for PostgreSQL.
  - Database Readiness: This is the most common cause. While depends_on helps, it’s not a readiness check. For now, try restarting just the web service after giving the db service a moment to fully start: docker compose restart web. In a future chapter, we will implement healthcheck configurations to properly handle service readiness.
  - Check db logs: Use docker compose logs db to ensure the PostgreSQL container started without errors and is listening on its port.
Port conflict: “Bind for 0.0.0.0:8000 failed: port is already allocated”:
- Issue: Another process on your host machine is already using port 8000. This could be a previously running container, another application, or even a system service.
- Solution:
  - Identify and stop the conflicting process: On Linux/macOS, you can use sudo lsof -i :8000 to find which process is using the port. Then, stop that process.
  - Change host port mapping: Modify the ports mapping in your docker-compose.yml for the web service to use a different host port (e.g., "8001:8000"). This maps host port 8001 to container port 8000.

Summary & Next Step

You have successfully orchestrated a multi-service application using Docker Compose! This chapter was a significant step in moving beyond single containers to managing a complete application stack.

What you’ve accomplished:

Defined a docker-compose.yml file: This central configuration file now defines your web service (a Python Flask application) and a db service (PostgreSQL).
Configured internal networking: Services communicate securely over a dedicated Docker network, enhancing isolation.
Mapped ports: Your web application is accessible from your host machine.
Understood key concepts: You now grasp how services, networks, volumes, build, ports, depends_on, and environment variables work together in Docker Compose.

You can now bring your entire application stack up and down with simple, consistent commands, greatly simplifying the management of your development environment.

In the next chapter, we will delve deeper into managing persistent data with Docker volumes. We’ll explore named volumes in more detail, understand their lifecycle, and ensure that your critical database data is never lost, even when containers are recreated, updated, or moved. This is a crucial aspect of building production-ready applications with Docker.

This page is AI-assisted and reviewed. It references official documentation and recognized resources where relevant.

References

Docker Documentation: https://docs.docker.com/
Compose Specification Versioning: https://github.com/jamesatdocker/docker-docs/blob/main/compose/compose-file/compose-versioning.md
PostgreSQL Docker Official Image: https://hub.docker.com/_/postgres
Flask Documentation: https://flask.palletsprojects.com/
Psycopg2 Documentation: https://www.psycopg.org/docs/

Orchestrating Services with Docker Compose

// table of contents