Dockerfile Instructions and Layer Mechanics: How Build Caching Works Under the Hood

A Node.js web development team notices their CI/CD build pipeline takes 12 minutes to finish on every single commit.

Even if a developer changes only a single comment line in a source file, the build engine spends 11 minutes downloading 400 megabytes of npm dependencies.

An engineer restructures the Dockerfile by moving a single line:

# BEFORE (Takes 12 Minutes on every commit):
COPY . .
RUN npm install

# AFTER (Takes 6 Seconds on every commit):
COPY package.json package-lock.json ./
RUN npm install
COPY . .

Build time drops from 12 minutes to 6 seconds.

Why did moving COPY . . below RUN npm install produce a 120x build speedup?

The cause is the Docker Build Cache Invalidation Algorithm.

1. How the Build Cache Engine Operates

When BuildKit or the Docker Engine evaluates a Dockerfile, it iterates through instructions sequentially from top to bottom.

For each instruction, the engine checks its local Build Cache to determine if an identical layer was previously built:

[ Dockerfile Instruction ]
           |
           v
  Matches Parent Layer Cache?
           |
   +-------+-------+
   | YES           | NO
   v               v
Check Instruction   *** CACHE MISS ***
Match Condition     1. Execute instruction
   |                2. Build new layer
   +-------+        3. INVALIDATE CACHE for ALL trailing layers!
   | YES   | NO
   v       v
[USE CACHE] [CACHE MISS]

Cache Matching Rules by Instruction Type

Static Command Instructions (RUN, ENV, EXPOSE):
- The engine checks if the text string of the instruction matches an existing cached layer derived from the same parent layer.
- Example: RUN apt-get update matches the cached layer if the command string is identical.
File Transfer Instructions (COPY, ADD):
- The engine does not rely on file modification timestamps or command string text alone.
- It computes a Checksum (SHA-256) over the contents of every file being copied from the host.
- If a single character inside a copied file changes, the checksum changes, triggering a Cache Miss.

2. The Cache Invalidation Cascade Effect

The fundamental rule of Docker build caching is: Once a single instruction incurs a Cache Miss, the build engine invalidates the cache for EVERY subsequent instruction in the Dockerfile!

Consider the inefficient Dockerfile:

FROM node:18-alpine

# Step 1: Copies ALL source code files (including index.js, package.json, etc.)
COPY . /app
WORKDIR /app

# Step 2: Installs dependencies
RUN npm install

# Step 3: Builds bundle
RUN npm run build

What Happens on a Code Edit:

You edit a single line in index.js.
COPY . /app calculates the SHA-256 checksum of host directory files.
Because index.js changed, COPY . /app incurs a Cache Miss.
The engine invalidates the cache for COPY . /app and executes it.
The Cascade: Because the parent layer for RUN npm install changed, the engine bypasses the cache for RUN npm install, forcing a full 11-minute dependency download!

3. Optimizing Dockerfile Instruction Ordering

To maximize build cache hits, order Dockerfile instructions by Frequency of Change, placing slow, stable steps at the top and fast, volatile steps at the bottom.

Frequency of Change              Dockerfile Instruction Placement
-----------------------------------------------------------------------
SLOwest / STABLE                 FROM ubuntu:22.04
  |                              RUN apt-get update && apt-get install -y nodejs
  |                              COPY package.json package-lock.json ./
  |                              RUN npm install
  v                              COPY . . (Fast changing source code!)
FASTEST / VOLATILE               CMD ["npm", "start"]

The Optimized Dependency Layer Pattern

FROM node:18-alpine
WORKDIR /app

# 1. Copy ONLY dependency manifest files first
COPY package.json package-lock.json ./

# 2. Run dependency installation (Cached as long as package.json is unchanged!)
RUN npm install

# 3. Copy changing application source code AFTER dependency installation
COPY . .

# 4. Build application
RUN npm run build

When you edit index.js, COPY package.json ... and RUN npm install hit the cache instantly. BuildKit skips downloading dependencies and proceeds directly to copying source files, completing the build in seconds!

4. Exec Form vs Shell Form Mechanics

Dockerfile instructions like RUN, CMD, and ENTRYPOINT can be written in two distinct syntactic forms:

Shell Form

CMD node server.js

The engine executes the command via an underlying shell wrapper: /bin/sh -c "node server.js".
Issue: The shell process /bin/sh becomes PID 1 inside the container. It does not forward POSIX system signals (SIGTERM, SIGINT) to node, preventing graceful container shutdowns!

Exec Form (JSON Array Syntax)

CMD ["node", "server.js"]

The engine executes execve("/usr/local/bin/node", ["server.js"]) directly without a shell wrapper.
Advantage: node becomes PID 1 inside the container, receiving SIGTERM signals directly and allowing graceful connection shutdowns.

Best Practice: Always use Exec Form (["executable", "param1"]) for CMD and ENTRYPOINT instructions!

Summary & Next Steps

Understanding build caching rules transforms container deployment speed and CI pipeline efficiency:

BuildKit matches layers using instruction strings and file content SHA-256 checksums.
Cache Invalidation Cascades invalidate all trailing layers as soon as a single step misses cache.
Instruction Ordering should place stable dependencies before volatile application source code.
Exec Form (["cmd", "arg"]) ensures container binaries run as PID 1 without shell signal trapping issues.

In the next article, we examine Multi-Stage Build Architectures & Minimal Base Image Optimization.

References & Further Reading

Linux Kernel Organization. Overlay Filesystem Specification (overlayfs.rst). Linux Kernel Docs.
Docker Inc. About Storage Drivers and Overlay2 Architecture. Docker Docs.
Bovet, D. P., & Cesati, M. (2005). Understanding the Linux Kernel (3rd Edition). O’Reilly Media.