Configuration and Secrets Management: ConfigMaps, Secrets, and Volume Mount Mechanics

A security team rotates a database password in production due to a compliance policy update.

The DevOps team updates the corresponding Kubernetes Secret:

kubectl create secret generic db-credentials --from-literal=password=NewSecurePassword123 --dry-run=client -o yaml | kubectl apply -f -

The Secret updates successfully in etcd.

However, for the next 12 hours, application Pods continue failing database connections with ACCESS DENIED errors.

When the team inspects the running container processes, they discover the application was reading the password from an Environment Variable (env):

env:
- name: DB_PASSWORD
  valueFrom:
    secretKeyRef:
      name: db-credentials
      key: password

Because environment variables are injected into a Linux process at execve(2) startup time, updating a Secret or ConfigMap does NOT update environment variables of running containers until the Pods are restarted!

When the team refactors the deployment to consume configuration via a Volume Mount:

volumeMounts:
- name: secret-volume
  mountPath: /etc/secrets
  readOnly: true

The underlying file /etc/secrets/password is updated atomically in real time, allowing the application to reload the new credentials without a single container restart.

How do ConfigMaps and Secrets project data into Pods using Environment Variables versus Volume Mounts?

1. ConfigMaps vs Secrets

Kubernetes decouples application configuration code from environment settings using two dedicated resource objects:

+-------------------------------------------------------------------+
| 1. ConfigMap (Non-Confidential Configuration)                     |
|    - Database URLs, log levels, feature flags, Nginx configs      |
|    - Stored as plaintext key-value pairs in etcd                  |
+-------------------------------------------------------------------+

+-------------------------------------------------------------------+
| 2. Secret (Confidential Sensitive Payloads)                       |
|    - API tokens, TLS private keys, database passwords             |
|    - Stored as Base64 encoded byte arrays                         |
|    - Backed by memory-based tmpfs when mounted into Pods          |
+-------------------------------------------------------------------+

The Base64 Misconception:

Base64 encoding (echo "my-password" | base64) is encoding, NOT encryption. Anyone with get secret permissions can decode Base64 strings instantly.

To secure production Secrets:

Enable Encryption at Rest in etcd (using KMS provider plugins like AWS KMS or HashiCorp Vault).
Enforce strict RBAC Policies limiting who can read Secret resources.

2. Delivery Mechanism 1: Environment Variables (`env`)

Injecting configuration via environment variables is simple and widely supported by application frameworks:

spec:
  containers:
  - name: web-app
    image: my-app:v1.0
    env:
    - name: LOG_LEVEL
      valueFrom:
        configMapKeyRef:
          name: app-config
          key: log_level

Mechanics & Trade-offs:

Static Injection: The Kubelet reads the value from etcd when constructing the PodSpec and passes it to the container runtime via CRI CreateContainer().
No Live Reloading: Modifying the underlying ConfigMap in etcd has zero effect on running containers. You must execute a rolling restart (kubectl rollout restart deployment) to inject updated values into new Pod instances.

3. Delivery Mechanism 2: Volume Mounts (`volumeMounts`)

Mounting a ConfigMap or Secret as a volume projects keys as individual files inside a container directory path:

spec:
  containers:
  - name: web-app
    image: my-app:v1.0
    volumeMounts:
    - name: config-vol
      mountPath: /etc/config
      readOnly: true
  volumes:
  - name: config-vol
    configMap:
      name: app-config

Inside the container, directory /etc/config contains files named after the keys:

$ ls -l /etc/config
log_level -> ..data/log_level
db_url    -> ..data/db_url

How Atomic Symlink Live-Reloading Works

When you update a ConfigMap mounted as a volume, Kubelet updates the files inside the container automatically without restarting the Pod.

To ensure that an application never reads a partial, corrupt file mid-update, Kubelet uses a 3-tier Symlink Swapping System:

[ Projected Volume Directory: /etc/config ]
  ├── log_level --------------------------+
  ├── db_url -----------------------------|
  └── ..data --------------------------+  |
                                       |  |
  +------------------------------------+  |
  |                                       |
  v                                       v
[ Timestamp Symlink: ..2026_09_08_18_41 ] <-- Swapped atomically!
  ├── log_level ("DEBUG")
  └── db_url ("jdbc:postgresql://...")

Kubelet creates a new timestamp directory containing the updated file values (..2026_09_08_18_41).
Kubelet updates the ..data symlink to point to the new timestamp directory in a single atomic OS symlink swap (symlink(2)).
Applications watching /etc/config via inotify or periodic file polling observe the new configuration instantly!

Environment Variables vs Volume Mounts Matrix

Feature / Behavior	Environment Variables (`env`)	Volume Mounts (`volumeMounts`)
Container Injection Method	CRI process environment array	Projected virtual filesystem mount
Live Reload Capability	❌ Static (Requires Pod restart)	✅ Dynamic (Atomic symlink updates)
Memory Backing for Secrets	Process memory table	Memory-backed `tmpfs` (Never written to disk)
File Structure Support	Single string values	Multi-line files (Nginx `.conf`, JSON, PEM certs)
Subpath Mount Limitation	N/A	Subpath mounts (`subPath`) disable live-reloading

Summary & Next Steps

ConfigMaps and Secrets decouple configuration code from execution environments:

ConfigMaps store non-confidential configuration; Secrets store sensitive Base64 payloads.
Environment Variable Injection (env) provides static configuration set at container startup, requiring Pod restarts to update.
Volume Mount Projections (volumeMounts) use atomic symlink swapping (..data) to update container files in real time without restarting Pods.
Secrets Mounted as Volumes are backed by memory-based tmpfs filesystems to prevent leaking credentials to disk.

In the next article, we transition to Module 6 and explore Kubernetes Scheduler Internals: Filtering (Predicates), Scoring (Priorities), and Affinities.

References & Further Reading

CNCF SIG Autoscaling. Horizontal Pod Autoscaler Algorithm Specification. CNCF Docs.
CNCF KEDA Project. KEDA Event-driven Autoscaling Specifications. KEDA Docs.
CNCF SIG Instrumentation. Prometheus Adapter for Kubernetes Metrics APIs. CNCF GitHub.