Chapter 9: Advanced Kubernetes - Scaling, Configuration & Secrets

Welcome back, future DevOps maestro! In our previous Kubernetes adventures, you mastered the fundamentals: deploying applications with Pods, making them accessible with Services, and managing their lifecycle with Deployments. You’ve got a solid foundation, but real-world applications demand more – they need to be dynamic, adaptable, and secure.

This chapter is your gateway to making your Kubernetes applications truly production-ready. We’ll explore how to automatically scale your applications to handle varying loads, how to manage application configurations cleanly and efficiently, and critically, how to protect sensitive information like API keys and database credentials. By the end of this chapter, you’ll be able to build more resilient, flexible, and secure applications on Kubernetes.

Ready to level up your Kubernetes skills? Let’s dive in!

Scaling Applications Automatically with Horizontal Pod Autoscaler (HPA)

Imagine your application experiences a sudden surge in user traffic. Without intervention, it might slow down, or worse, crash! Manually adding more Pods is reactive and slow. This is where Horizontal Pod Autoscaler (HPA) comes to the rescue.

What is HPA? HPA is a Kubernetes API resource that automatically scales the number of Pods in a Deployment, ReplicaSet, or StatefulSet based on observed CPU utilization or other custom metrics. “Horizontal” means it increases or decreases the number of Pod replicas, as opposed to “vertical” scaling, which involves increasing the resources (CPU, RAM) of existing Pods.

Why is HPA important?

• Cost Efficiency: Only use resources when needed. Scale up during peak times, scale down during low usage.
• Reliability: Maintain performance and availability even under fluctuating loads.
• Automation: Reduces manual intervention, letting Kubernetes manage resource allocation.

How HPA Works (The Basics)

1. You define an HPA object, specifying the target Deployment (or ReplicaSet/StatefulSet), the desired minimum and maximum number of Pods, and the metric to watch (e.g., target average CPU utilization of 50%).
2. Kubernetes’ controller-manager continuously monitors the specified metrics from the metrics-server (which collects data from Kubelets).
3. If the average metric value (e.g., CPU usage) exceeds the target, HPA calculates how many more Pods are needed to bring the average down to the target. It then updates the target Deployment’s replica count.
4. If the average metric value falls below the target, HPA scales down the number of Pods.

Let’s visualize this flow:

Prerequisite: Metrics Server For HPA to work with CPU/Memory metrics, your Kubernetes cluster needs the metrics-server installed. Most managed Kubernetes services (like GKE, EKS, AKS) come with it pre-installed. For local clusters (like Minikube or Kind), you might need to install it.

To check if metrics-server is running, you can use:

kubectl get apiservice v1beta1.metrics.k8s.io

If it’s available, you’ll see a status like Available: True. If not, you might need to install it. For Minikube, you can enable it with minikube addons enable metrics-server.

Externalizing Configuration with ConfigMaps

Hardcoding configuration values directly into your application code or Docker images is a recipe for disaster. What if a database password changes? Or an API endpoint? You’d have to rebuild and redeploy your entire application! This is where ConfigMaps shine.

What are ConfigMaps? ConfigMaps are Kubernetes API objects used to store non-sensitive configuration data as key-value pairs. They allow you to decouple configuration artifacts from image content, keeping your application images generic and reusable. Think of them as a way to inject configuration settings into your Pods from outside.

Why use ConfigMaps?

• Separation of Concerns: Keep configuration separate from your application code.
• Flexibility: Change configuration without rebuilding Docker images or redeploying code.
• Environment Agnostic: Use the same Docker image across different environments (dev, staging, prod) by simply swapping ConfigMaps.

How ConfigMaps work: You create a ConfigMap object, then reference it in your Pod or Deployment definition. Kubernetes can inject ConfigMap data into your Pods in two main ways:

1. As Environment Variables: Each key-value pair in the ConfigMap becomes an environment variable in your container.
2. As Mounted Files: The ConfigMap data can be mounted as files within a volume inside your container. Each key becomes a filename, and its value becomes the file’s content.

Managing Sensitive Data with Secrets

While ConfigMaps are great for general configuration, they are not suitable for sensitive information like passwords, API keys, or TLS certificates. For that, Kubernetes provides Secrets.

What are Secrets? Secrets are Kubernetes objects designed to store sensitive data. They are similar to ConfigMaps but provide some additional security-focused features. By default, Secrets are stored in etcd (Kubernetes’ backing store) as base64-encoded strings.

Why use Secrets?

• Security for Sensitive Data: Designed for credentials, tokens, and keys.
• Access Control: Kubernetes can restrict access to Secrets based on RBAC (Role-Based Access Control) policies.
• Decoupling: Keep sensitive data separate from your application code.

Important Note on Security: It’s crucial to understand that base64 encoding is not encryption. It merely obfuscates the data, making it unreadable at a glance. Anyone with access to your cluster can easily decode a base64-encoded Secret. For true encryption of Secrets at rest, you need to configure Kubernetes with “encryption at rest” for etcd or use external Secret management solutions (like HashiCorp Vault, AWS Secrets Manager, Azure Key Vault, Google Secret Manager) that integrate with Kubernetes. However, for basic usage, Kubernetes Secrets provide a dedicated mechanism to handle sensitive data separately.

How Secrets work: Like ConfigMaps, Secrets can be injected into Pods in two primary ways:

1. As Environment Variables: Each key-value pair (after decoding) becomes an environment variable.
2. As Mounted Files: The decoded Secret data is mounted as files within a volume inside your container.

Let’s get our hands dirty with some practical examples!

Step-by-Step Implementation

We’ll use a simple Nginx web server for our examples.

1. Setting up Horizontal Pod Autoscaler (HPA)

First, let’s deploy a simple Nginx application.

Step 1.1: Create an Nginx Deployment Create a file named nginx-deployment.yaml:

# nginx-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: nginx-hpa-demo
  labels:
    app: nginx-hpa
spec:
  replicas: 1 # Start with 1 replica
  selector:
    matchLabels:
      app: nginx-hpa
  template:
    metadata:
      labels:
        app: nginx-hpa
    spec:
      containers:
      - name: nginx
        image: nginx:1.25.3 # Using a recent stable Nginx version
        ports:
        - containerPort: 80
        resources: # Define resource limits for HPA to work effectively
          requests:
            cpu: "100m" # Request 100 millicpu (0.1 CPU core)
          limits:
            cpu: "200m" # Limit to 200 millicpu (0.2 CPU core)
---
apiVersion: v1
kind: Service
metadata:
  name: nginx-hpa-service
spec:
  selector:
    app: nginx-hpa
  ports:
    - protocol: TCP
      port: 80
      targetPort: 80
  type: ClusterIP # We'll access it internally for now

Explanation:

• We define a Deployment named nginx-hpa-demo with a single replica initially.
• The nginx:1.25.3 image is specified.
• Crucially, we define resources.requests.cpu and resources.limits.cpu. HPA relies on these requests to calculate CPU utilization. Without them, HPA won’t know the baseline for your Pod’s CPU.
• A Service exposes the Nginx Pods internally.

Apply this deployment:

kubectl apply -f nginx-deployment.yaml

Verify your deployment and service:

kubectl get deployments,pods,services -l app=nginx-hpa

You should see one Nginx Pod running.

Step 1.2: Create the Horizontal Pod Autoscaler Now, let’s define the HPA. Create a file named nginx-hpa.yaml:

# nginx-hpa.yaml
apiVersion: autoscaling/v2 # Use v2 for more advanced features, v1 is also common
kind: HorizontalPodAutoscaler
metadata:
  name: nginx-cpu-hpa
spec:
  scaleTargetRef: # Which resource to scale
    apiVersion: apps/v1
    kind: Deployment
    name: nginx-hpa-demo
  minReplicas: 1 # Minimum number of Pods
  maxReplicas: 5 # Maximum number of Pods
  metrics: # What metrics to watch
  - type: Resource
    resource:
      name: cpu
      target:
        type: Utilization
        averageUtilization: 50 # Target 50% average CPU utilization

Explanation:

• apiVersion: autoscaling/v2: We’re using the v2 API version for HPA, which supports more metric types. v1 is simpler but less flexible.
• scaleTargetRef: Points to our nginx-hpa-demo Deployment.
• minReplicas: 1, maxReplicas: 5: The HPA will ensure there’s always at least 1 Pod and never more than 5.
• metrics: We’re targeting cpu resource utilization.
• averageUtilization: 50: The HPA will try to keep the average CPU utilization of all Pods at 50% of their requested CPU. If it goes above, it scales up; if below, it scales down.

Apply the HPA:

kubectl apply -f nginx-hpa.yaml

Check the HPA status:

kubectl get hpa

You’ll see output similar to this (initially, CPU will be <unknown> or low):

NAME            REFERENCE                 TARGETS         MINPODS   MAXPODS   REPLICAS   AGE
nginx-cpu-hpa   Deployment/nginx-hpa-demo   <unknown>/50%   1         5         1          10s

Step 1.3: Generate Load and Observe Scaling To see the HPA in action, we need to generate some CPU load on our Nginx Pod. We can do this by running a busybox Pod that continuously makes requests to our Nginx service.

Create a temporary busybox Pod to generate load:

kubectl run -it --rm load-generator --image=busybox:1.36.1 -- /bin/sh -c "while true; do wget -q -O- nginx-hpa-service; done"

(Note: busybox:1.36.1 is a recent stable version.)

This command starts a busybox Pod, and inside it, a loop continuously sends requests to nginx-hpa-service. Keep this terminal open.

In a separate terminal, continuously monitor your HPA:

watch kubectl get hpa nginx-cpu-hpa

You should start to see the TARGETS column increase as the CPU load on the Nginx Pod rises. After a minute or two, the REPLICAS column should increase from 1 to 2, then 3, and so on, up to 5, as the HPA scales out to meet the 50% CPU target.

Once you’ve observed the scaling up, stop the load-generator Pod by pressing Ctrl+C in its terminal. Then, watch the HPA again. After a few minutes of no load, you’ll see the REPLICAS slowly scale back down to 1. Kubernetes has a default cooldown period before scaling down to prevent “flapping” (rapid scaling up and down).

Clean up HPA and Deployment:

kubectl delete -f nginx-hpa.yaml
kubectl delete -f nginx-deployment.yaml

2. Externalizing Configuration with ConfigMaps

Let’s create an application that uses a custom greeting message from a ConfigMap.

Step 2.1: Create a ConfigMap Create a file named my-configmap.yaml:

# my-configmap.yaml
apiVersion: v1
kind: ConfigMap
metadata:
  name: app-config
data:
  APP_MESSAGE: "Hello from Kubernetes ConfigMap!"
  ENVIRONMENT: "Development"
  API_URL: "http://api.example.com/v1"

Explanation:

• data: This section holds our key-value pairs. APP_MESSAGE, ENVIRONMENT, and API_URL are the keys, and their respective strings are the values.

Apply the ConfigMap:

kubectl apply -f my-configmap.yaml

Verify the ConfigMap:

kubectl get configmap app-config -o yaml

You’ll see its content.

Step 2.2: Deploy an Application Using ConfigMap as Environment Variables We’ll use a simple alpine/git image to demonstrate, which can easily echo environment variables. In a real application, your code would read these variables.

Create a file named configmap-env-deployment.yaml:

# configmap-env-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: configmap-env-demo
  labels:
    app: configmap-env
spec:
  replicas: 1
  selector:
    matchLabels:
      app: configmap-env
  template:
    metadata:
      labels:
        app: configmap-env
    spec:
      containers:
      - name: my-app-container
        image: alpine/git:2.43.0 # Using a recent stable alpine/git image
        command: ["/bin/sh", "-c"]
        args:
          - echo "Starting application with config:" &&
            echo "Message: $APP_MESSAGE" &&
            echo "Environment: $ENVIRONMENT" &&
            echo "API URL: $API_URL" &&
            sleep 3600 # Keep the container running for inspection
        envFrom: # Inject all key-value pairs from the ConfigMap
        - configMapRef:
            name: app-config

Explanation:

• image: alpine/git:2.43.0: A lightweight image to run our sh command.
• command and args: This sets up a shell script that echoes the environment variables we expect to receive and then sleeps to keep the container alive.
• envFrom: This is the magic! Instead of listing each env variable separately, envFrom.configMapRef injects all key-value pairs from the app-config ConfigMap directly as environment variables into the container.

Apply the deployment:

kubectl apply -f configmap-env-deployment.yaml

Check the logs of the Pod to see the injected environment variables:

POD_NAME=$(kubectl get pods -l app=configmap-env -o jsonpath='{.items[0].metadata.name}')
kubectl logs $POD_NAME

You should see output similar to:

Starting application with config:
Message: Hello from Kubernetes ConfigMap!
Environment: Development
API URL: http://api.example.com/v1

Success! Your application received its configuration from the ConfigMap.

Step 2.3: Deploy an Application Using ConfigMap as Mounted Files Sometimes, applications expect configuration in files (e.g., .conf files, .properties files). ConfigMaps can also provide this.

Create a file named configmap-file-deployment.yaml:

# configmap-file-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: configmap-file-demo
  labels:
    app: configmap-file
spec:
  replicas: 1
  selector:
    matchLabels:
      app: configmap-file
  template:
    metadata:
      labels:
        app: configmap-file
    spec:
      containers:
      - name: my-app-container
        image: alpine/git:2.43.0
        command: ["/bin/sh", "-c"]
        args:
          - echo "Checking mounted config files:" &&
            cat /etc/config/APP_MESSAGE &&
            cat /etc/config/ENVIRONMENT &&
            sleep 3600
        volumeMounts:
        - name: config-volume
          mountPath: /etc/config # Mount the ConfigMap here
      volumes:
      - name: config-volume
        configMap:
          name: app-config

Explanation:

• volumeMounts: We define a volumeMount named config-volume that will be mounted at /etc/config inside the container.
• volumes: We define a volume named config-volume that sources its data from the app-config ConfigMap. Each key in the ConfigMap (APP_MESSAGE, ENVIRONMENT) will become a separate file in /etc/config with its value as the file content.

Apply the deployment:

kubectl apply -f configmap-file-deployment.yaml

Check the logs of the Pod:

POD_NAME=$(kubectl get pods -l app=configmap-file -o jsonpath='{.items[0].metadata.name}')
kubectl logs $POD_NAME

You should see:

Checking mounted config files:
Hello from Kubernetes ConfigMap!
Development

Excellent! The configuration was successfully injected as files.

Clean up ConfigMap and Deployments:

kubectl delete -f my-configmap.yaml
kubectl delete -f configmap-env-deployment.yaml
kubectl delete -f configmap-file-deployment.yaml

3. Managing Sensitive Data with Secrets

Now, let’s work with Secrets. We’ll simulate storing a database password.

Step 3.1: Create a Secret You can create Secrets from literal values or files. Let’s do both.

Method A: From Literal (for simple key-value pairs)

kubectl create secret generic db-credentials \
  --from-literal=DB_USERNAME=admin \
  --from-literal=DB_PASSWORD=supersecurepassword123

Explanation:

• kubectl create secret generic: This command creates a generic Secret.
• db-credentials: The name of our Secret.
• --from-literal: Specifies a key-value pair to include in the Secret.

Method B: From File (better for multi-line data or actual files) Let’s create a dummy password file:

echo "another_secret_key" > api_key.txt

Now create the Secret from this file:

kubectl create secret generic api-key-secret --from-file=api_key.txt

Explanation:

• --from-file=api_key.txt: The filename becomes the key in the Secret, and the file’s content becomes the value.

Verify the Secrets (and see the base64 encoding):

kubectl get secret db-credentials -o yaml
kubectl get secret api-key-secret -o yaml

You’ll notice the data values are base64 encoded. To decode them (for educational purposes, never do this in production without strong justification):

kubectl get secret db-credentials -o jsonpath='{.data.DB_PASSWORD}' | base64 --decode
kubectl get secret api-key-secret -o jsonpath='{.data.api_key\.txt}' | base64 --decode

Step 3.2: Deploy an Application Using Secrets as Environment Variables We’ll use our db-credentials Secret.

Create a file named secret-env-deployment.yaml:

# secret-env-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: secret-env-demo
  labels:
    app: secret-env
spec:
  replicas: 1
  selector:
    matchLabels:
      app: secret-env
  template:
    metadata:
      labels:
        app: secret-env
    spec:
      containers:
      - name: my-app-container
        image: alpine/git:2.43.0
        command: ["/bin/sh", "-c"]
        args:
          - echo "Starting application with secrets:" &&
            echo "DB Username: $DB_USERNAME" &&
            echo "DB Password: $DB_PASSWORD" &&
            sleep 3600
        env: # Use 'env' for specific key-value pairs
        - name: DB_USERNAME
          valueFrom:
            secretKeyRef:
              name: db-credentials
              key: DB_USERNAME
        - name: DB_PASSWORD
          valueFrom:
            secretKeyRef:
              name: db-credentials
              key: DB_PASSWORD

Explanation:

• env: We specify individual environment variables.
• valueFrom.secretKeyRef: This tells Kubernetes to fetch the value for this environment variable from a specific key within a named secret. This is generally preferred over envFrom for Secrets, as it gives you finer control over which specific secret keys are exposed and how they are named in the environment.

Apply the deployment:

kubectl apply -f secret-env-deployment.yaml

Check the logs of the Pod:

POD_NAME=$(kubectl get pods -l app=secret-env -o jsonpath='{.items[0].metadata.name}')
kubectl logs $POD_NAME

You should see your decoded username and password.

Step 3.3: Deploy an Application Using Secrets as Mounted Files Now, let’s use the api-key-secret and mount it as a file.

Create a file named secret-file-deployment.yaml:

# secret-file-deployment.yaml
apiVersion: apps/v1
kind: Deployment
metadata:
  name: secret-file-demo
  labels:
    app: secret-file
spec:
  replicas: 1
  selector:
    matchLabels:
      app: secret-file
  template:
    metadata:
      labels:
        app: secret-file
    spec:
      containers:
      - name: my-app-container
        image: alpine/git:2.43.0
        command: ["/bin/sh", "-c"]
        args:
          - echo "Checking mounted secret files:" &&
            cat /etc/secrets/api_key.txt &&
            sleep 3600
        volumeMounts:
        - name: secret-volume
          mountPath: /etc/secrets
      volumes:
      - name: secret-volume
        secret:
          secretName: api-key-secret

Explanation:

• volumeMounts: Mounts the secret-volume to /etc/secrets.
• volumes.secret: This sources the volume from api-key-secret. The api_key.txt key from the Secret will appear as a file named api_key.txt in the /etc/secrets directory.

Apply the deployment:

kubectl apply -f secret-file-deployment.yaml

Check the logs of the Pod:

POD_NAME=$(kubectl get pods -l app=secret-file -o jsonpath='{.items[0].metadata.name}')
kubectl logs $POD_NAME

You should see:

Checking mounted secret files:
another_secret_key

Fantastic! You’ve successfully managed sensitive data using Kubernetes Secrets.

Clean up Secrets and Deployments:

kubectl delete secret db-credentials api-key-secret
kubectl delete -f secret-env-deployment.yaml
kubectl delete -f secret-file-deployment.yaml
rm api_key.txt # Clean up the local file too

Mini-Challenge: Advanced Configuration with an Echo Server

Let’s combine what you’ve learned! Your challenge is to create a simple web server (you can use Nginx or a custom image that prints env vars) that:

1. Reads a custom welcome message from a ConfigMap (e.g., WELCOME_MESSAGE).
2. Reads an API token from a Secret (e.g., AUTH_TOKEN).
3. Automatically scales between 1 and 3 replicas based on CPU utilization, targeting 60%.

Challenge Steps:

1. Create a ConfigMap named echo-config with a WELCOME_MESSAGE key.
2. Create a Secret named echo-secret with an AUTH_TOKEN key.
3. Create a Deployment for an Nginx server (or a simple custom image that prints environment variables) that:
   • Uses resources.requests.cpu and limits.cpu.
   • Injects WELCOME_MESSAGE from echo-config as an environment variable.
   • Injects AUTH_TOKEN from echo-secret as an environment variable.
   • (Optional but encouraged) Modify the Nginx configuration (via another ConfigMap mounted as a file) to display the WELCOME_MESSAGE on its default page.
4. Create a Service for this deployment.
5. Create an HPA for this deployment, targeting 60% CPU utilization.
6. Generate load to observe the scaling.
7. Verify the environment variables/files are correctly injected by checking Pod logs.

Hint:

• For Nginx, you can mount a ConfigMap to /etc/nginx/conf.d/default.conf to override its default configuration and include environment variables. Remember to set envsubst '$WELCOME_MESSAGE $AUTH_TOKEN;' < /etc/nginx/conf.d/default.conf.template > /etc/nginx/conf.d/default.conf in your container’s entrypoint if you want to use env variables in the Nginx config.
• Alternatively, use a simpler image like busybox or alpine/git to just echo the variables.

What to Observe/Learn:

• How to combine multiple configuration methods (HPA, ConfigMap, Secret) in a single application.
• The lifecycle of scaling up and down based on load.
• The difference in how ConfigMaps and Secrets are defined and referenced.

Common Pitfalls & Troubleshooting

1. HPA not scaling:

   • Missing Metrics Server: kubectl get apiservice v1beta1.metrics.k8s.io should show Available: True. If not, install it.
   • No Resource Requests/Limits: Pods must have resources.requests.cpu defined for HPA to calculate utilization percentage.
   • Insufficient Load: Are you generating enough load to consistently push CPU above the target utilization?
   • Cooldown Periods: HPA has default scaleUp and scaleDown stabilization windows. Scaling isn’t instantaneous.
   • Incorrect scaleTargetRef: Ensure the HPA correctly points to your Deployment’s apiVersion, kind, and name.

2. ConfigMap/Secret not found or not mounted correctly:

   • Typos in Names: Double-check the name of your ConfigMap/Secret and the name in configMapRef/secretKeyRef/volume.
   • Incorrect mountPath: If mounting as files, ensure the mountPath is where your application expects them.
   • envFrom vs. env: Remember envFrom injects all key-value pairs, while env with valueFrom allows you to pick specific keys and rename environment variables. For Secrets, valueFrom is generally safer.
   • Missing key in secretKeyRef: If you’re referencing a specific key from a Secret, ensure that key actually exists in the Secret.

3. Misunderstanding Secret Security:

   • Base64 is NOT Encryption: Never assume your Secrets are encrypted by default in Kubernetes. They are merely encoded. Always follow best practices for Secret management, including using encryption at rest for etcd or external Secret managers for production.
   • Don’t echo Secrets in logs: While we did this for demonstration, in a real application, avoid printing sensitive information to standard output or logs.

Summary

Congratulations! You’ve successfully navigated some of the most critical aspects of advanced Kubernetes application management:

• Horizontal Pod Autoscaler (HPA) empowers your applications to scale dynamically, ensuring resilience and cost efficiency by automatically adjusting the number of Pod replicas based on CPU or custom metrics.
• ConfigMaps provide a clean and flexible way to externalize non-sensitive application configuration, allowing you to easily manage settings without modifying your application code or Docker images.
• Secrets offer a dedicated mechanism for handling sensitive data like passwords and API keys, injecting them into your Pods securely as environment variables or mounted files, while also understanding the nuances of base64 encoding versus true encryption.

By mastering these tools, you’re well on your way to building robust, scalable, and secure applications on Kubernetes. You’re thinking like a true DevOps professional!

What’s Next? In the next chapter, we’ll shift our focus to Web Server Setup and Management. We’ll learn how to deploy and configure Nginx and Apache, understand the critical differences between HTTP and HTTPS, and delve into the world of SSL/TLS certificates for securing your web traffic. This will prepare you for making your Kubernetes-hosted applications accessible and secure to the outside world.

References

1. Kubernetes Official Documentation - Horizontal Pod Autoscaler: https://kubernetes.io/docs/tasks/run-application/horizontal-pod-autoscale/
2. Kubernetes Official Documentation - ConfigMaps: https://kubernetes.io/docs/tasks/configure-pod-container/configure-a-pod-configmap/
3. Kubernetes Official Documentation - Secrets: https://kubernetes.io/docs/concepts/configuration/secret/
4. Kubernetes Official Documentation - kubectl create secret: https://kubernetes.io/docs/reference/kubectl/cheatsheet/#secrets
5. Mermaid.js Flowchart Syntax: https://mermaid.js.org/syntax/flowchart.html

This page is AI-assisted and reviewed. It references official documentation and recognized resources where relevant.