Storage Drivers Deep Dive: OverlayFS Layering and Copy-On-Write (CoW) Performance Overhead

A high-throughput telemetry service processes incoming IoT device logs inside a Docker container.

The application appends incoming metrics to a single 2 Gigabyte file on disk: /var/log/telemetry.dat.

As system throughput increases to 5,000 writes per second, host storage metrics show severe degradation:

CPU I/O Wait (iowait) spikes to 68%.
Disk write throughput drops from 400 MB/sec down to 12 MB/sec.
Application request latencies balloon from 2 milliseconds to 850 milliseconds.

Surprisingly, host storage hardware is a high-speed Enterprise NVMe SSD.

When an engineer reconfigures the log path to write to a Docker Volume instead of the container’s internal filesystem:

docker run -d -v telemetry-data:/var/log telemetry-service

CPU iowait collapses back to 0.2%, and write latency drops back to 1.5 milliseconds.

Why did writing to a file inside the container filesystem trigger severe I/O degradation while writing to a Docker volume ran at native NVMe speed?

The cause is the Copy-on-Write (CoW) Overhead of the OverlayFS Storage Driver.

1. How OverlayFS Works

OverlayFS is a modern Linux union filesystem driver that merges multiple directories into a single unified mount view.

When Docker launches a container using OverlayFS (specifically overlay2), it constructs four core directory locations inside /var/lib/docker/overlay2/:

+-------------------------------------------------------------------+
| 1. mergedir (/var/lib/docker/overlay2/<id>/merged)                |
|    The unified mount directory presented to container processes   |
+-------------------------------------------------------------------+
                                  ^
                                  |  Kernel Union Mount
             +--------------------+--------------------+
             |                                         |
+------------------------------------+   +------------------------------------+
| 2. upperdir                        |   | 3. lowerdir                        |
|    Container Writable Layer        |   |    Read-Only Image Layers          |
|    (Stores new & modified files)   |   |    (/layer3:/layer2:/layer1)       |
+------------------------------------+   +------------------------------------+
                                         | 4. workdir                         |
                                         |    Internal kernel atomic operations|
                                         +------------------------------------+

lowerdir: A list of read-only image layer directories separated by colons (/dir3:/dir2:/dir1).
upperdir: The single read-write container layer directory where new or modified files are stored.
mergedir: The unified virtual mount point. When a process inside the container views /, it sees mergedir.
workdir: An internal workspace directory used by the Linux kernel to execute atomic operations (like file renames).

2. File Access Mechanics in OverlayFS

When a process inside a container interacts with a file path inside mergedir, OverlayFS evaluates file operations based on where the file resides:

A. Reading a File (`open(..., O_RDONLY)`)

Case 1 (File exists in upperdir): Read directly from upperdir.
Case 2 (File exists ONLY in lowerdir): Read directly from lowerdir.
Performance: Zero overhead! Reading files inherited from read-only image layers runs at native disk speed.

B. Modifying an Existing File (`open(..., O_WRONLY)`) $\to$ Copy-on-Write (CoW)

What happens when a container opens a 2 Gigabyte file (/var/log/telemetry.dat) for modification when that file exists only in a read-only lower image layer (lowerdir)?

Because lowerdir is strictly immutable, the Linux kernel cannot modify the file in-place on disk.

Instead, the kernel executes a Copy-on-Write (CoW) operation:

Step 1: Container process opens /var/log/telemetry.dat for WRITE.
        |
        v
Step 2: OverlayFS intercepts syscall, detects file is in read-only lowerdir.
        |
        v
Step 3: *** COPY-ON-WRITE LATENCY ***
        Kernel reads the ENTIRE 2GB file from lowerdir...
        ...and writes all 2GB bytes into upperdir!
        |
        v
Step 4: The write operation proceeds against the new file copy in upperdir.

The Performance Penalty:

If an application modifies a single 4-byte integer inside a 2GB file inherited from an image layer, OverlayFS must copy all 2,147,483,648 bytes from lowerdir up to upperdir before the 4-byte write can complete!

This explains why our telemetry service suffered severe CPU iowait spikes when writing to large files in the container filesystem.

C. Deleting a File (`unlink(...)`)

When a container process deletes a file inherited from lowerdir:

OverlayFS does not delete the file from lowerdir.
It creates a Character Device Whiteout File (major/minor number 0/0) in upperdir named /.wh.<filename>.
The whiteout marker masks the lower file, making it disappear from mergedir.

3. Bypassing Copy-on-Write with Docker Volumes

Why did switching to a Docker Volume eliminate Copy-on-Write overhead entirely?

A Docker Volume is not managed by OverlayFS.

When you attach a volume (-v my-vol:/var/log), Docker mounts a host directory (e.g., /var/lib/docker/volumes/my-vol/_data) directly into the container’s mount namespace using a standard Linux bind mount:

[ Container Mount Namespace ]
/var/log  ===================>  Direct Bind Mount to Host NVMe Disk
                                (/var/lib/docker/volumes/my-vol/_data)
                                (Bypasses OverlayFS CoW entirely!)

Because volume reads and writes bypass upperdir and lowerdir completely, file I/O runs at 100% native host storage speed without Copy-on-Write penalties!

Storage Layer Comparison Matrix

Storage Location	Managed By	Copy-on-Write (CoW) Penalty?	Data Persistence	Best Use Case
Container Writable Layer	OverlayFS (`upperdir`)	Yes (High overhead for large file edits)	Ephemeral (Deleted on `docker rm`)	Small temporary configuration edits, ephemeral runtime files
Named Docker Volumes	Host VFS (Bind Mount)	No (Native host storage speed)	Persistent (Survives container removal)	Databases (Postgres, MySQL), application logs, uploaded media
Host Bind Mounts	Host VFS (Bind Mount)	No (Native host storage speed)	Persistent (Direct host path access)	Local developer code mounting (`-v $(pwd):/app`)
tmpfs Mounts	Host RAM	No (RAM execution speed)	Ephemeral (Erased on container stop)	High-security secrets, temporary in-memory caches

Summary & Next Steps

OverlayFS union mounts balance storage sharing against file modification overhead:

OverlayFS merges lowerdir (read-only layers) and upperdir (writable layer) into mergedir.
Copy-on-Write (CoW) copies entire files from lowerdir up to upperdir before executing initial write operations, introducing severe latency for large files.
Whiteout files (.wh.filename) mask deleted lower layer files.
Docker Volumes bypass OverlayFS entirely, providing native host storage performance for I/O-intensive workloads.

In the next article, we inspect Docker Storage Options: Named Volumes, Bind Mounts, and tmpfs Mount Mechanics.

References & Further Reading

CNCF Prometheus Project. cAdvisor Integration & Container Metrics Guidelines. Prometheus Docs.
Wiggins, A. (2012). The Twelve-Factor App: Factor XI Logs (Treat logs as event streams). 12factor.net.
Docker Inc. Configure Logging Drivers. Docker Docs.