Chapter 7: Docker Compose - Orchestrating Multi-Container Applications

Introduction to Orchestrating Multi-Container Applications

Welcome back, future DevOps maestro! In our last chapter, we mastered the art of running single Docker containers and even crafted our own custom images using Dockerfile. That was a fantastic start, but in the real world, applications are rarely just one isolated container. Think about a typical web application: you’ll likely have a web server, a backend API, a database, maybe a cache, and more – all needing to talk to each other.

Managing these interconnected containers manually with individual docker run commands can quickly become a nightmare. How do you ensure they all start in the right order? How do they find each other on the network? How do you manage their configurations? This is where Docker Compose steps in, transforming chaos into harmony.

In this chapter, you’ll learn how to define and run multi-container Docker applications using a single configuration file. We’ll dive deep into the docker-compose.yml file, understand its structure, and use it to build, run, and manage complex application stacks with ease. By the end, you’ll be orchestrating your own containerized ecosystems, ready to tackle more sophisticated deployments!

Core Concepts: What is Docker Compose?

Imagine you’re a conductor, and each musician in your orchestra is a Docker container. Playing a single instrument (running one container) is easy enough. But to create a symphony (a complete application), you need to coordinate all the musicians, tell them when to start, how to play together, and where to sit. Docker Compose is your baton and sheet music for containerized applications.

Docker Compose is a tool for defining and running multi-container Docker applications. With Compose, you use a YAML file (typically docker-compose.yml) to configure your application’s services. Then, with a single command, you create and start all the services from your configuration.

Why Docker Compose?

1. Simplicity: Define your entire application stack in one file. No more juggling multiple docker run commands.
2. Reproducibility: Your application setup is codified, ensuring everyone on your team (or even future you!) can spin up the exact same environment.
3. Portability: The docker-compose.yml file can be shared and run on any system with Docker and Docker Compose installed.
4. Isolation: Each service runs in its own container, providing process isolation and resource management.
5. Networking: Compose automatically creates a network for your services, allowing them to communicate with each other using their service names as hostnames.

Let’s visualize this orchestration:

The docker-compose.yml File: Your Application Blueprint

The heart of Docker Compose is the docker-compose.yml file. This YAML file describes the services that make up your application, including:

• version: The Compose file format version. As of 2026, 3.8 or 3.9 are common stable choices, offering robust features.
• services: This is where you define each individual container (or “service”) in your application. Each service entry specifies:
  • image: The Docker image to use (e.g., redis:latest).
  • build: Path to a Dockerfile if you’re building a custom image for this service.
  • ports: How to map host ports to container ports.
  • environment: Environment variables to pass into the container.
  • volumes: Mount host paths or named volumes into the container for persistent data.
  • networks: Which networks the service should connect to.
  • depends_on: To express dependencies between services, ensuring they start in a particular order (though this doesn’t guarantee a service is ready, just started).
• networks: Define custom networks for your services, allowing for better isolation and organization.
• volumes: Define named volumes for persistent data storage, separate from the container’s lifecycle.

Docker Compose Commands

The primary command you’ll use is docker compose (note: without the hyphen!). The older docker-compose command is largely deprecated in favor of the docker compose CLI plugin, which is typically bundled with Docker Desktop or installed separately. Always use docker compose for modern workflows.

• docker compose up: Builds, (re)creates, starts, and attaches to containers for all services.
  • Add -d for “detached” mode, running containers in the background.
• docker compose down: Stops and removes containers, networks, volumes, and images created by up.
• docker compose ps: Lists all services running in the current Compose project.
• docker compose logs: Displays log output from services.
• docker compose build: Builds or rebuilds services.

Ready to put this into practice? Let’s build a simple multi-container application!

Step-by-Step Implementation: A Flask App with Redis

We’re going to create a simple Python Flask web application that counts how many times you’ve visited it, storing the count in a Redis database. This will involve two services: our Flask app and the Redis database.

1. Project Setup

First, let’s create a new directory for our project.

mkdir my-flask-redis-app
cd my-flask-redis-app

2. The Flask Application (app.py)

Inside my-flask-redis-app, create a file named app.py and add the following Python code. This simple app will connect to Redis, increment a counter, and display it.

# app.py
import time
import redis
from flask import Flask

app = Flask(__name__)
# Connect to Redis. The hostname 'redis' will be resolved by Docker Compose's internal DNS.
cache = redis.Redis(host='redis', port=6379)

def get_hit_count():
    retries = 5
    while True:
        try:
            # Increment the counter and return its new value
            return cache.incr('hits')
        except redis.exceptions.ConnectionError as exc:
            if retries == 0:
                raise exc
            retries -= 1
            time.sleep(0.5)

@app.route('/')
def hello():
    count = get_hit_count()
    return f"Hello from Docker! I have been seen {count} times.\n"

if __name__ == "__main__":
    app.run(host="0.0.0.0", port=5000, debug=True)

Explanation:

• We import Flask for our web app and redis to interact with the Redis database.
• app = Flask(__name__) initializes our Flask application.
• cache = redis.Redis(host='redis', port=6379) is crucial! Notice host='redis'. Docker Compose creates a network where services can find each other by their service name. So, our Flask app can simply refer to the Redis container as redis.
• get_hit_count() attempts to increment a counter in Redis. It includes basic retry logic because the Redis container might not be fully ready immediately when the Flask app starts.
• @app.route('/') defines our main web route.
• app.run(host="0.0.0.0", port=5000, debug=True) makes the Flask app accessible from outside the container on port 5000.

3. Flask Application Dependencies (requirements.txt)

Our Flask app needs Flask and redis-py. Create requirements.txt in the same directory:

# requirements.txt
Flask==3.0.3
redis==5.0.1

Explanation:

• We specify the exact versions for Flask and redis to ensure consistent builds. As of early 2026, these are stable and widely used versions.

4. Dockerfile for the Flask Application

Next, we need a Dockerfile to build our Flask application’s image. Create Dockerfile in the my-flask-redis-app directory:

# Dockerfile
# Use an official Python runtime as a parent image
FROM python:3.9-slim-buster

# Set the working directory in the container
WORKDIR /app

# Copy the requirements file into the container at /app
COPY requirements.txt .

# Install any needed packages specified in requirements.txt
RUN pip install --no-cache-dir -r requirements.txt

# Copy the rest of the application code into the container at /app
COPY . .

# Make port 5000 available to the world outside this container
EXPOSE 5000

# Run app.py when the container launches
CMD ["python", "app.py"]

Explanation:

• FROM python:3.9-slim-buster: We start with a lightweight Python 3.9 image.
• WORKDIR /app: Sets /app as the default directory for subsequent commands.
• COPY requirements.txt .: Copies our dependency list.
• RUN pip install --no-cache-dir -r requirements.txt: Installs Flask and Redis client.
• COPY . .: Copies app.py and any other files.
• EXPOSE 5000: Informs Docker that the container listens on port 5000.
• CMD ["python", "app.py"]: Specifies the command to run when the container starts.

5. Creating docker-compose.yml

Now for the main event! Create a file named docker-compose.yml in your my-flask-redis-app directory. We’ll build this incrementally.

Start with the basic structure:

# docker-compose.yml
version: '3.8' # Using Compose file format version 3.8

services:
  # This is where we'll define our individual services

Explanation:

• version: '3.8': Declares the Compose file format version. Version 3.8 (or higher like 3.9) is a good, stable choice for modern Docker Compose features as of 2026.

Now, let’s add our web service (the Flask app):

# docker-compose.yml
version: '3.8'

services:
  web: # This is the name of our service
    build: . # Tells Compose to build an image using the Dockerfile in the current directory
    ports:
      - "5000:5000" # Map host port 5000 to container port 5000
    environment:
      FLASK_ENV: development # Set a Flask environment variable

Explanation:

• web:: Defines a service named web. This name will also be used as its hostname on the Docker network.
• build: .: Instead of pulling a pre-existing image, Compose will look for a Dockerfile in the current directory (.) and build an image for this service.
• ports: - "5000:5000": This maps port 5000 on your host machine to port 5000 inside the web container. This allows you to access the Flask app from your browser.
• environment: FLASK_ENV: development: Sets an environment variable inside the container.

Next, add the redis service:

# docker-compose.yml
version: '3.8'

services:
  web:
    build: .
    ports:
      - "5000:5000"
    environment:
      FLASK_ENV: development
    depends_on: # Ensure Redis starts before the web app
      - redis
    # We don't need to specify a network here; Docker Compose creates a default one.

  redis: # This is the name of our Redis service
    image: "redis:7.2.4-alpine" # Use the official Redis image, version 7.2.4 (stable as of 2026)
    # We don't expose Redis ports to the host by default for security,
    # as only the 'web' service needs to talk to it internally.
    # volumes:
    #   - redis-data:/data # Optional: for persistent Redis data (more on this later)

# Optional: Define networks and volumes if needed (we'll stick to defaults for now)
# volumes:
#   redis-data:

Explanation:

• redis:: Defines a service named redis.
• image: "redis:7.2.4-alpine": We’re using the official Redis Docker image. 7.2.4-alpine is a good choice for stability and smaller image size.
• depends_on: - redis: This tells Docker Compose that the web service depends on the redis service. Compose will try to start redis before web. Important: This only ensures startup order, not that Redis is fully ready to accept connections. Our get_hit_count function in app.py handles this with retries.

6. Running Your Multi-Container Application

Now that our docker-compose.yml is complete, let’s bring our application to life! Make sure you are in the my-flask-redis-app directory.

docker compose up -d

Explanation:

• docker compose up: Reads your docker-compose.yml file.
• -d: Runs the containers in “detached” mode, meaning they run in the background and don’t tie up your terminal.

You should see output similar to this (image names/hashes will vary):

[+] Running 3/3
 ✔ Network my-flask-redis-app_default  Created                                                    0.0s
 ✔ Container my-flask-redis-app-redis-1  Started                                                  0.0s
 ✔ Container my-flask-redis-app-web-1  Started                                                    0.0s

7. Verifying the Application

Open your web browser and navigate to http://localhost:5000.

You should see: Hello from Docker! I have been seen 1 times.

Refresh the page a few times, and the count should increment! This proves your Flask app is running, connecting to the Redis container, and successfully incrementing the counter.

To see the running containers:

docker compose ps

Output will look something like this:

NAME                    COMMAND                  SERVICE             STATUS              PORTS
my-flask-redis-app-redis-1   "docker-entrypoint.s…"   redis               running             6379/tcp
my-flask-redis-app-web-1     "python app.py"          web                 running             0.0.0.0:5000->5000/tcp

To view the logs of your services:

docker compose logs web
docker compose logs redis

8. Stopping and Cleaning Up

When you’re done, you can stop and remove all the services and the network created by Compose:

docker compose down

Explanation:

• docker compose down: Gracefully stops the containers, then removes the containers and the network created by docker compose up. If you had defined named volumes, you’d need docker compose down --volumes to remove them as well.

You should see:

[+] Running 3/3
 ✔ Container my-flask-redis-app-web-1  Removed                                                    0.1s
 ✔ Container my-flask-redis-app-redis-1  Removed                                                  0.1s
 ✔ Network my-flask-redis-app_default  Removed                                                    0.0s

Congratulations! You’ve successfully orchestrated a multi-container application using Docker Compose.

Mini-Challenge: Add a Persistent Volume to Redis

Currently, if you run docker compose down and then docker compose up -d again, your hit count resets. This is because Redis stores data inside its container, and when the container is removed, the data is lost.

Your Challenge: Modify the docker-compose.yml file to add a named volume for the Redis service, ensuring that the hit count persists even after you stop and restart your application.

Hint:

1. You’ll need to define a volumes section at the top level of docker-compose.yml.
2. Then, within the redis service definition, add a volumes entry to mount this named volume to Redis’s data directory (which is /data by default in the official Redis image).

What to Observe/Learn: After implementing the change, try the following:

1. Run docker compose up -d.
2. Refresh your browser a few times at http://localhost:5000 to increment the count.
3. Run docker compose down.
4. Run docker compose up -d again.
5. Check http://localhost:5000. Did the count persist? If so, great job! You’ve mastered persistent data with volumes.

# docker-compose.yml (with persistent volume)
version: '3.8'
services:
web:
build: .
ports:
- “5000:5000”
environment:
FLASK_ENV: development
depends_on:
- redis
redis:
image: “redis:7.2.4-alpine”
volumes:
- redis_data:/data # Mount the named volume ‘redis_data’ to Redis’s /data directory
volumes: # Define the named volume
redis_data:

Common Pitfalls & Troubleshooting

Working with Docker Compose is powerful, but you might encounter a few common issues:

1. YAML Syntax Errors: YAML is sensitive to indentation. A single incorrect space can lead to errors like ERROR: yaml.scanner.ScannerError: while scanning a simple key.
   • Troubleshooting: Use a YAML linter (many IDEs have them) or online YAML validators. Pay close attention to indentation.
2. Service Startup Order vs. Readiness: depends_on only ensures that a service’s container is started before its dependencies. It doesn’t guarantee the service inside the container is fully initialized and ready to accept connections (e.g., a database might take longer to boot up).
   • Troubleshooting: Implement retry logic in your application code (like our Flask app’s get_hit_count). For more robust solutions, consider health checks (defined in docker-compose.yml with healthcheck) or wait-for-it scripts in your CMD or entrypoint.
3. Port Conflicts: If you try to map a container port to a host port that’s already in use (e.g., another application is using host port 5000), docker compose up will fail with an error like Error starting userland proxy: listen tcp 0.0.0.0:5000: bind: address already in use.
   • Troubleshooting: Change the host port mapping in your docker-compose.yml (e.g., "5001:5000") or stop the application currently using that port.
4. Environment Variable Issues: Forgetting to pass crucial environment variables (like database connection strings or API keys) can lead to application failures inside the container.
   • Troubleshooting: Use docker compose logs <service_name> to check for application errors. Use docker compose exec <service_name> env to inspect the environment variables inside a running container.

Summary

You’ve come a long way in this chapter! Let’s recap the key takeaways:

• Docker Compose simplifies the definition and management of multi-container Docker applications.
• The docker-compose.yml file is the blueprint, written in YAML, defining services, networks, and volumes.
• services define individual containers, specifying build or image, ports, environment, and depends_on.
• image vs build: Use image for pre-built images (like redis), and build with a Dockerfile for custom application images.
• Networking is automatically handled by Compose, allowing services to communicate using their service names.
• volumes are essential for persisting data beyond the container’s lifecycle.
• The modern command for interacting with Compose is docker compose (without the hyphen).
• Key commands: docker compose up -d (start in detached mode), docker compose down (stop and remove), docker compose ps (list services), docker compose logs (view logs).
• You learned to implement a Flask web app with a Redis database, demonstrating inter-service communication and persistent storage.

You’re now equipped to manage complex, multi-service applications with confidence! This skill is fundamental in modern DevOps practices, providing a solid foundation for deploying applications consistently across different environments.

In the next chapter, we’ll delve deeper into Docker Image Optimization and Best Practices, learning how to build smaller, faster, and more secure Docker images, which is crucial for efficient deployments.

References

• Docker Compose Overview
• Compose file version 3 reference
• Docker CLI compose command reference
• Official Redis Docker Image

This page is AI-assisted and reviewed. It references official documentation and recognized resources where relevant.