Container Observation and Troubleshooting: Logging Drivers, Health Checks, and Diagnostics

At 03:00 AM, a high-traffic microservice application enters a deadlocked state.

Inside the container, all HTTP worker threads are frozen, waiting on a database connection pool lock. Every incoming request times out with 500 Internal Server Error.

Yet, when the automated monitoring system queries Docker:

docker ps

The output displays:

CONTAINER ID   IMAGE          COMMAND                  CREATED        STATUS
a1b2c3d4e5f6   api-server:v1  "node server.js"         2 days ago     Up 2 days

Because the Node.js process (PID 1) is still running, Docker reports the container status as Up 2 days.

For three hours, load balancers continue routing real customer traffic to the deadlocked container because Docker has no built-in way of knowing that the process inside PID 1 is functionally dead.

When the engineering team adds a HEALTHCHECK Instruction:

HEALTHCHECK --interval=10s --timeout=3s --retries=3 \
  CMD curl -f http://localhost:8080/health || exit 1

Within 30 seconds of entering a deadlock, Docker marks the container status as unhealthy, and the container orchestrator automatically replaces the deadlocked container instance.

How do Health Checks, Logging Drivers, and Diagnostic Inspection Tools provide visibility into running container environments?

1. Container Health Checks (`HEALTHCHECK`)

A running process (PID 1) does not equal a healthy application.

The HEALTHCHECK instruction tells the Docker daemon how to test if the application process inside the container is actually functioning correctly.

                  [ HEALTHCHECK Execution Loop (Every 10s) ]
                                      |
                                      v
                  Executes: `curl -f http://localhost:8080/health`
                                      |
                       +--------------+--------------+
                       |                             |
            Exit Code 0 (Success)         Exit Code 1 (Failure)
                       |                             |
                       v                             v
               [ Status: HEALTHY ]            Increment Failure Counter
                                                     |
                                         Failures >= Retries (3)?
                                                     |
                                        +------------+------------+
                                        | YES                     | NO
                                        v                         v
                               [ Status: UNHEALTHY ]      [ Retain State ]

Syntax and Parameters

HEALTHCHECK --interval=10s --timeout=3s --start-period=15s --retries=3 \
  CMD wget --no-verbose --tries=1 --spider http://localhost:8080/health || exit 1

--interval: Frequency between health check probe executions (default: 30s).
--timeout: Maximum time allowed for a single probe command to complete (default: 30s).
--start-period: Grace period initialization window during app startup before failures count toward retries.
--retries: Number of consecutive probe failures required to transition container status from healthy to unhealthy.

2. Container Logging Architecture & Log Rotation

When an application inside a container writes to standard output (stdout) or standard error (stderr), Docker captures those stream bytes.

By default, Docker uses the json-file Logging Driver, writing log lines to a JSON file on the host machine:

/var/lib/docker/containers/<container_id>/<container_id>-json.log

The Unbounded Log File Incident

If log rotation is not configured, high-logging production applications will continuously write log entries until host disk space hits 100%, causing host system crashes.

Configuring Global Log Rotation (`daemon.json`)

Configure log file rotation rules in /etc/docker/daemon.json:

{
  "log-driver": "json-file",
  "log-opts": {
    "max-size": "10m",
    "max-file": "3"
  }
}

This configuration caps individual log files at 10 Megabytes and retains a maximum of 3 rotated backup files, guaranteeing that logging overhead never exceeds 30MB per container.

3. The Production Diagnostic Toolkit

When troubleshooting container issues, use the 4-step diagnostic workflow:

[ 1. docker ps / stats ] ---> [ 2. docker logs ] ---> [ 3. docker inspect ] ---> [ 4. docker exec ]
  Check CPU/Memory/Health       Read stdout/stderr      Inspect JSON Config       Debug inside NetNS

A. Resource Metrics (`docker stats`)

Provides a real-time terminal stream of CPU, memory, network I/O, and disk I/O metrics across active containers:

docker stats --no-stream

CONTAINER ID   NAME       CPU %     MEM USAGE / LIMIT     MEM %     NET I/O
a1b2c3d4e5f6   api-web    98.4%     510MiB / 512MiB       99.6%     1.2MB / 45MB

Diagnosis: api-web is thrashing near its 512MB RAM cap, causing high CPU allocation during garbage collection.

B. Metadata Inspection (`docker inspect`)

Returns the complete JSON configuration, network IP bindings, volume mounts, and exit state of a container:

docker inspect --format='{{.State.ExitCode}} - {{.State.Error}}' my-container

C. Live Debugging Shell (`docker exec`)

Launches a new debug process inside an existing container’s active namespaces:

docker exec -it my-container /bin/sh

If the container image does not contain a shell (e.g., Google Distroless or scratch), use Ephemeral Debug Containers or inspect network namespaces from the host using nsenter:

# Debug network namespace using host binaries:
CONTAINER_PID=$(docker inspect --format '{{.State.Pid}}' my-container)
sudo nsenter -t $CONTAINER_PID -n ip addr

Diagnostic Tool Selection Matrix

Troubleshooting Task	Command / Tool	Primary Metric / Output
Check Health Status	`docker ps`	Displays `healthy`, `unhealthy`, or `starting` state
Diagnose Crashes / Errors	`docker logs --tail 100 <id>`	Captures `stdout`/`stderr` application stack traces
Inspect OOM / Exit Codes	`docker inspect <id>`	Reads `ExitCode`, `OOMKilled`, and `FinishedAt` timestamps
Check Memory/CPU Leaks	`docker stats`	Live CPU %, Memory %, and I/O byte counters
Inspect Virtual Mounts	`docker inspect -f '{{.Mounts}}'`	Verifies volume destinations and bind mount paths

Summary & Next Steps

Container observation requires multi-layered diagnostic tooling:

HEALTHCHECK Instructions test functional application readiness beyond basic PID 1 process execution.
Log Rotation (max-size=10m) prevents container log files from exhausting host disk capacity.
docker stats & docker inspect reveal real-time cgroup resource consumption and exact exit error codes.
nsenter enables low-level kernel namespace debugging for minimal distroless containers.

In the final article of this master series—Post 20: Building a Custom Container Runtime Engine in Java—we synthesize everything we’ve learned by building a functional container engine CLI!

References & Further Reading

Cloud Native Computing Foundation. Kubernetes Container Runtime Interface (CRI) gRPC Protocol Definitions. CNCF.
Kubernetes Authors. (2020). Don’t Panic: Kubernetes and Docker shim Deprecation. Kubernetes Blog.
containerd Authors. containerd CRI Plugin Architecture. containerd Docs.