Stage 1: Solved, not run
Everyone learns the Dockerfile as a shell script: a list of commands, executed top to bottom, one after another. That model was correct, and both the builder it describes and the one that replaced it are in the same binary on your machine today.
Here is a file with three stages, two of which nothing refers to. On the legacy builder:
$ DOCKER_BUILDKIT=0 docker build -t lab/dag:classic .
Step 1/6 : FROM scratch AS neverused
Step 2/6 : COPY marker-A /A
---> a5e1e5a962d9
...
Step 6/6 : COPY marker-C /CSix steps, all executed, two intermediate images left behind. Now the same file on the default builder, measured on Engine 29.4.3:
$ docker build -t lab/dag:buildkit .
#4 [final 1/1] COPY marker-C /C
#4 DONE 0.0sOne vertex. BuildKit reads your file into a graph, works out what the requested output depends on, and solves for that. The unreferenced stages are not merely quiet — they never ran. The proof is to break one deliberately: change the unreferenced stage’s COPY to name a file that does not exist anywhere, and the ordinary build still succeeds, while --target neverused fails immediately with "/does-not-exist-anywhere": not found.
Two consequences arrive straight away. Stages that are referenced can run in parallel, so the order of numbered steps in your output is a scheduling artefact rather than a reading of your file. And .dockerignore has quietly changed jobs.
The build context is not uploaded
Same directory, same 64 MB file that no COPY names, three builds:
| Builder and Dockerfile | Transferred |
|---|---|
BuildKit, COPY app.txt /app.txt | 40 B |
BuildKit, COPY . /app | 67.13 MB |
| legacy builder, either file | 67.11 MB |
Forty bytes against sixty-seven megabytes, from one word in the Dockerfile. BuildKit transfers what the graph references and nothing else. So the advice that .dockerignore is how you keep your builds fast is describing a builder most readers stopped using in February 2023, when BuildKit became the default on Linux in Engine 23.0. Its remaining job — keeping things out of the image — is real, sharper than the size job ever was, and belongs to stage 1 of the page on what ships: COPY . /app puts your .git directory in the image, and every secret that repository has ever held with it.
And a green step is not a step that worked
This is the one that ships broken images. A RUN that fetches an installer over a pipe, on a base image with no curl:
#6 [3/3] RUN curl -fsSL https://no.such.host.invalid/install.sh | sh
#6 0.245 /bin/sh: 1: curl: not found
#6 DONE 0.3s
$ echo $?
0The step reports DONE. The build exits 0. The image ships without the software it exists to carry, and nothing anywhere says so. The mechanism is not exotic and not a BuildKit behaviour: /bin/sh -c reports the exit status of the last command in a pipeline, and pipefail is not on. RUN false | true behaves identically.
The error was printed. It was printed on a step that then reported success, in collapsed output, on a build nobody was watching — which is why the first line of the checklist at the end of this page is to run the build once with --progress=plain and actually read it.
Verify: do not read the log. Run the thing the step was supposed to install, inside the finished image — docker run --rm <image> sh -c 'command -v curl && curl --version'. A green build log is a statement about the builder. This is a statement about the image.
Stage 2: The base image is a shell you did not choose
Resolve the hinge’s silence first. If docker history --format '{{.CreatedBy}}' <image> | grep '^USER' printed nothing, you did not write a USER line, and the empty field is something you inherited. That is worth knowing on its own, and it is also a sample of a larger fact: FROM selects a set of defaults, and every one of them belongs to somebody else. Here are the ones that break builds.
The shell
Stage 1’s curl | sh failure has a well-known remedy, and on a Debian-family base it does not work:
SHELL ["/bin/sh", "-o", "pipefail", "-c"]
#5 0.144 /bin/sh: 0: Illegal option -o pipefail/bin/sh on Debian and Ubuntu is dash, and dash has no pipefail. SHELL ["/bin/bash", "-o", "pipefail", "-c"] works where bash exists — which is not everywhere, and on a busybox-based image you should test it rather than assume it. The general point is the one to keep: your RUN lines are interpreted by a program the base image chose, and shell-form RUN never says which.
The dynamic linker
Go from a full base to FROM scratch and the build gets faster, smaller and green. Then:
$ docker run --rm lab/scratch:cgo
exec /probe: no such file or directory
$ docker run --rm lab/scratch:cgo ls -l /probe
ls: not found # there is no ls either; the file is thereThe file that does not exist is not the one named in the error. A CGO-linked Go binary is dynamically linked, and the missing file is /lib64/ld-linux-x86-64.so.2 — the interpreter the kernel is asked to load first. The error names the binary because that is the path execve was given. Build with CGO_ENABLED=0, or copy the interpreter and the libraries in, or use a base that has them.
The certificate store, and the passwd file
Fix the linking and the same image fails differently, this time only in production, and only on the code path that talks to something:
x509: certificate signed by unknown authorityNothing is wrong with the certificate. FROM scratch has no /etc/ssl/certs/ca-certificates.crt, so there is no trust anchor to check it against — the system trust store is a file, and an empty image does not have it. The same applies to /etc/passwd: with no passwd file, a numeric USER has nothing to resolve to, whoami fails, HOME is /, and any library that looks up its own user fails with it. And to /etc/nsswitch.conf, and to the zoneinfo database, each of which fails in its own idiom.
None of these is a reason not to use a minimal base. They are the reason to know that a minimal base is a list of absences, and every absence is something the previous base was quietly supplying.
Verify: docker run --rm <image> sh -c 'ls /etc/ssl/certs/ca-certificates.crt /etc/passwd; id'. If there is no shell to run it with, that is the answer to stage 7’s question as well, and you have met it early.
Stage 3: Layers, and the eleven instructions that make none
“One instruction, one layer” is repeated everywhere and it is not close. A file with sixteen instructions, built and then counted:
$ docker image inspect lab/census:1 --format '{{len .RootFS.Layers}}'
5| Produces a layer | Produces only a config field |
|---|---|
FROM, RUN, COPY, ADD, WORKDIR | ARG, ENV, LABEL, EXPOSE, VOLUME, USER, STOPSIGNAL, SHELL, HEALTHCHECK, ENTRYPOINT, CMD |
Eleven of the sixteen wrote nothing to a filesystem. They wrote to the image’s config blob — a small JSON document that travels beside the layers, is not a layer, and cannot be squashed. That is where ENV lands, and it is why a credential set with ENV or passed as --build-arg survives every multi-stage build and every rm; the page on what ships covers that in full and this one does not repeat it.
Better still: two of the instructions that did produce a layer produced an empty one. RUN true, and a WORKDIR on a directory that already existed, each got a real slot in the image — and the same empty blob ends up occupying three of the five:
sha256:4f4fb700ef54… 32 B # RUN true
sha256:4f4fb700ef54… 32 B # WORKDIR /srv
sha256:4f4fb700ef54… 32 B # and once moreOne blob, three times: the empty gzipped tar, thirty-two bytes, occupying three separate entries in rootfs.diff_ids. It costs nothing to ship and it is a good demonstration that a layer is a diff rather than a step.
And the size column is not layer bytes
Here is the tool everyone is told to reach for, against the same image:
| Layer | docker history says | compressed blob | uncompressed tar |
|---|---|---|---|
RUN true / WORKDIR | 4.1 kB | 32 B | 1024 B |
COPY f.txt /srv/f.txt | 12.3 kB | 132 B | 2560 B |
4096 and 12288 bytes: snapshot blocks, not layer bytes. Under the containerd image store, which is the default on new installations from Engine 29, that column answers a storage question and not a shipping question. A legacy-built image reports 8.19 kB — two blocks — for a two-byte file. There is also a {{.Size}} field on docker image inspect; it returns a number that matches neither the sum of the compressed blobs nor the uncompressed content, and this page will not tell you what it counts, because it could not establish that.
Which is where that symptom-table row at the top of this page came from. docker history is a reasonable way to read your own instructions back; it is not a size tool, and it is not a cache tool at all.
Verify: docker image inspect <image> --format '{{len .RootFS.Layers}}' against the number of instructions in your file. If they match, either you wrote only RUN and COPY, or you are looking at a legacy build.
Stage 4: The cache is a graph
What a cache key is made of — the instruction text for a RUN, the content and mode of the files for a COPY — is settled, measured and published in stage 7 of the page on what ships, and this page does not re-derive it. What is left is the question that row in 552 got wrong: when a key misses, what does the miss propagate to?
The answer is “whatever depends on it”. Inside a single stage that is everything below it, because each instruction builds on the one before — which is where the file-order intuition comes from and why it survives. Across stages it is not, and almost every real Dockerfile now has more than one. Here is a build with an assets stage written at line 2 and a build stage at lines 6 and 7, invalidated from each end in turn:
# change the input to the stage written FIRST
#8 [build 2/3] COPY src/main.go . CACHED
#9 [build 3/3] RUN go build CACHED
#11 [assets 2/3] COPY assets/logo.svg . 0.1s <- only this missed
# change the input to the stage written LAST
#7 [assets 2/3] COPY assets/logo.svg . CACHED
#9 [build 3/3] RUN go build 0.4s <- only this missedIt goes both ways, and neither direction cascades. Line order is not dependency order; the builder solved a graph and the file was only ever a description of it. On a cold build it even numbers [assets 1/3] ahead of [build 2/3], which is written first.
The practical form of this is that “put the things that change least at the top” is a rule about dependencies that happens to be expressible as a rule about position. Within one stage, COPY the lockfile, install, then COPY the source: the install is above the source and therefore does not depend on it. Split the same work into two stages and the dependency is stated directly, the position stops mattering, and the two halves rebuild independently — which is the whole reason multi-stage builds are faster as well as smaller.
What --no-cache does not clear
A RUN --mount=type=cache directory is not part of the layer cache. It is a separate, mutable record, and it survives:
$ docker build --no-cache . && docker build --no-cache .
$ docker buildx du --verbose | grep -A2 cachemount
Type: exec.cachemount
Mutable: truedocker builder prune -af clears them. And this is the mechanism behind the largest single size result on this page. Installing jq three ways, compressed layer bytes, on a Debian-family base built here with debootstrap rather than pulled:
| How | Compressed layer | Against plain |
|---|---|---|
plain apt-get install | 4,789,555 B | — |
&& rm -rf /var/lib/apt/lists/* | 4,070,636 B | −15 % |
--mount=type=cache on /var/cache/apt and /var/lib/apt/lists | 504,824 B | −89 % |
The folk remedy addresses the package lists and never touches /var/cache/apt/archives, which held 28 MB of .deb files on this base. Read those numbers as a mechanism, not as a target — the official Debian and Ubuntu images ship an apt.conf.d hook that deletes the downloaded packages for you, which narrows the first gap considerably, and this base has no such hook. The ordering holds; the ratio is base-dependent.
One more thing that has moved: BuildKit leaves no dangling images at all. docker image prune was the right advice for a builder that created an intermediate image per step, and the build cache it has replaced them with is invisible to docker images and untouched by that command. docker builder prune is the one that reclaims it, and docker buildx history is where a finished build’s own record now lives.
Verify: build twice with no changes and confirm every step says CACHED; then change one input and read which steps missed. If the misses match the dependency and not the file order, you have understood this stage.
Stage 5: The identity, and who has to resolve it
This is the hinge’s stage. USER looks like it sets the user your container runs as. It writes a string into the config blob, and every failure below is somebody else failing to resolve that string.
Start with the case that is easiest to write and hardest to see. The Dockerfile says USER appuser and nothing ever created that account in the image:
$ docker build -f Dockerfile.missing -t lab/user:missing .
BUILD SUCCEEDED
$ docker run --rm lab/user:missing id
docker: Error response from daemon: unable to find user appuser:
no matching entries in passwd fileThe build validated nothing. It wrote the string down. The daemon is the thing that has to turn it into a UID, and it does so at docker run, by reading /etc/passwd inside the image — so this fails on every machine, forever, and it failed for the first time somewhere that is not your laptop.
Create the account and the string resolves, which introduces the opposite problem: the number is now real and nobody has written it down. useradd picked it. Your Dockerfile does not contain it. The bind mount in stage 7 is about to find out what it is.
Write the number yourself — USER 10001:10001 — and a third party appears. Numeric works everywhere and resolves to nothing: with no matching passwd entry, getpwuid has no answer, whoami fails, HOME is /, and any library that looks up its own user fails alongside them. The honest form is both: create the account and pin the number, so the name resolves inside the image and the number is a fact you chose rather than one useradd chose.
There is a fourth party, and it is the one that turns this into a production incident rather than a build error. A Kubernetes kubelet asked to enforce runAsNonRoot cannot prove a name is non-root without resolving it, and refuses: container has runAsNonRoot and image has non-numeric user (appuser), cannot verify user is non-root. That behaviour is reported widely and was not reproduced here; Kubernetes, Honestly is where admission and the kubelet’s checks live. It is the second branch of the hinge arriving weeks later on somebody else’s cluster.
Two more strings the build writes and does not check
USER is a default and nothing enforces it. docker run --user 0:0 overrides it and writes happily to files the image made root-owned; a compose user: key or a Kubernetes runAsUser does the same thing more quietly. If the image must not run as root, something outside the image has to say so.
And --platform is a label. Build with --platform linux/arm64 over a COPY of an x86-64 binary and the build succeeds, the config says linux/arm64, and the container runs on amd64 with a warning:
$ docker image inspect lab/plat:arm64 --format '{{.Os}}/{{.Architecture}}'
linux/arm64
$ file probe
probe: ELF 64-bit LSB executable, x86-64--platform selects base image variants and sets a field. Nothing validates that field against the bytes you brought. Cross-building means a compiler that targets the other architecture, or emulation; the flag on its own means neither.
Verify: run the hinge, then docker run --rm <image> id and compare. A UID you cannot account for, or an error from the daemon, are the two outcomes this stage exists to produce before anyone else finds them.
Stage 6: What the build never checked
There is a linter built into the builder, and its default is the interesting part. On a Dockerfile with a secret in an ENV, a shell-form ENTRYPOINT and a legacy key-value LABEL:
$ docker build --check .
SecretsUsedInArgOrEnv, JSONArgsRecommended, LegacyKeyValueFormat
$ echo $?
1
$ docker build -t lab/warn:1 .
... the same three warnings printed ...
$ echo $?
0The same three findings, printed either way, and the exit code differs. CI runs the second command. The warnings scroll past in a job that goes green, and the image gets pushed. Two ways to close it: run docker build --check . as its own step, or put # check=error=true as the first line of the Dockerfile, which makes the ordinary build exit 1.
Worth knowing what it is: a Dockerfile linter, built in since buildx 0.15, replacing the third-party tool most guides still recommend. It reads your file. It has no opinion about your image — it cannot know that curl was missing in stage 1, that appuser does not exist in stage 5, that the binary you copied is the wrong architecture, or that the interpreter it needs is absent. Everything this page has shown you passes --check.
One of its rules, JSONArgsRecommended, is worth taking seriously for reasons this page will not restate: 552 explains what shell-form CMD does to signal handling and to your process’s claim on PID 1, and why docker stop then takes exactly ten seconds.
Verify: docker build --check .; echo $?. If it is 1 and your pipeline is green, your pipeline is not running this command.
Stage 7: The handover
Every page on this subject ends at docker push. But the image is not for you, and the last thing that happens to it is that somebody else has to run it — with a volume, on a host, under a scheduler. Everything that person needs is a number, and the config blob does not carry it.
$ docker run --rm -v "$PWD/data:/data" lab/user:1 sh -c 'id -u; id -un; touch /data/x'
10001
app
touch: cannot touch '/data/x': Permission deniedThe builder chose 10001 and told nobody. Ask the image what it will hand over and it answers with a name and two nulls:
$ docker image inspect lab/user:1 \
--format 'user={{.Config.User}} volumes={{json .Config.Volumes}} ports={{json .Config.ExposedPorts}}'
user=app volumes=null ports=nullThe host filesystem has never heard of app, and it never will: a bind mount carries UID numbers across the boundary and nothing translates them. Moving a service into a container documents the far end of exactly this and prescribes the diagnostic — docker run --rm <image> id — as the first thing an adopter should run.
Which is where the page turns around on itself. Take the small, non-root, FROM scratch image that the six stages above have been teaching you to build, and run that diagnostic against it:
$ docker run --rm lab/scratch:static id -u
hello from /probeThere is no id, there is no shell, and the ENTRYPOINT swallowed the argument and ran the application instead. A good build produces an image that defeats this site’s own published procedure for adopting one — and it does so silently, by printing something plausible.
The fix is not to put a shell back. It is to stop making the next person interrogate the image, and publish the number instead: pin the UID in the Dockerfile rather than letting useradd pick it, create the passwd entry so a name resolves as well, and write the number in the same place you write the ports and the volumes. An image that has to be reverse-engineered before it can be mounted is finished from the builder’s point of view and not from anybody else’s.
Verify: docker run --rm <image> id, and if it cannot run, docker image inspect --format '{{.Config.User}}' — then check that whatever it prints appears somewhere a person will read before they mount a volume.
A worked diagnosis: the image that had not contained the application for three weeks
A service starts returning 404 on every route after a routine deploy. The rollback works, so the team assumes a bad release and looks at the application. Nothing in the diff explains it. The pipeline is green and has been green for months; the release build is green; docker build --check is clean; the image was pushed and the digest in the deployment matches the one CI produced.
Run the hinge against the released image:
$ docker image inspect registry.internal/svc:2.31.0 --format '{{printf "%q" .Config.User}}'
""
$ docker history --format '{{.CreatedBy}}' registry.internal/svc:2.31.0 | grep '^USER'
$Empty, and no USER instruction of theirs — so this is stage 2, an inherited default, and the base image is worth looking at. The hinge is pointing at the right neighbourhood and naming the wrong stage, which is worth saying out loud: it is a localiser, not an oracle. The base had changed — someone closed a size ticket three weeks earlier by moving from the full image to the slim one — but the mechanism that turned that into a broken release is stage 1’s.
Two commands finish it. First, against the image rather than the log:
$ docker run --rm registry.internal/svc:2.31.0 sh -c 'command -v curl; ls /opt/app/dist | head -1'
$ # nothing. no curl, and no dist.And then the build, re-run with the output uncollapsed:
$ docker build --progress=plain --no-cache . 2>&1 | grep -B1 -A1 'not found'
#9 0.201 /bin/sh: 1: curl: not found
#9 DONE 0.2sThe line had been printed on every build for three weeks, on a step that then reported DONE, in collapsed output, on a job that went green. The slim base had no curl; the asset build step fetched nothing, installed nothing, and exited 0 because /bin/sh -c reports the last command in a pipeline. Every release since had shipped an image with no front end in it. The rollback worked because the digest it rolled back to was built before the base changed — which is why the deploy looked like the cause.
Three things generalise. The hinge did its work by naming a party rather than a cause, and it was one stage off — which is the normal case and not a failure of it. A pipeline that gates on the build’s exit code is gating on the builder’s opinion of itself. And the sentence this whole page turns on: every green step in a build log is a statement about the builder; none of it is a statement about the image. The only way to learn something about an image is to ask the image.
Symptoms, and which stage they belong to
Numbered by the spine at the top of the page, which is also the order of the sections.
| Symptom | Stage | What is actually true |
|---|---|---|
An error was printed and the step still reported DONE | 1 | /bin/sh -c reports the last command in a pipeline |
| An instruction in your Dockerfile never ran | 1 | Unreferenced stages are not in the solved graph |
| The build transfers hundreds of megabytes every time | 1 | COPY . /app — the graph references the whole directory |
The image is missing software a RUN was supposed to install | 1 | The step exited 0 and nothing checked the result |
Illegal option -o pipefail | 2 | /bin/sh is dash on a Debian-family base |
exec <binary>: no such file or directory, for a file that is present | 2 | The missing file is the dynamic linker, not the binary |
x509: certificate signed by unknown authority from a minimal image | 2 | No CA bundle in the image to check against |
whoami fails and HOME is / | 2 | A numeric UID with no matching /etc/passwd entry |
| The layer count does not match the instruction count | 3 | Eleven instruction types write only config |
docker history sizes do not add up to anything | 3 | It reports snapshot blocks, not layer bytes |
| A cache miss did not invalidate the steps below it | 4 | Invalidation follows the graph, not the file |
--no-cache and the build is still suspiciously fast | 4 | Cache mounts are separate records and survive it |
docker image prune reclaims nothing after a build | 4 | BuildKit leaves no dangling images; use builder prune |
unable to find user <name>: no matching entries in passwd file | 5 | The daemon resolves USER at run time, from the image’s passwd |
| A pod is refused for having a non-numeric user | 5 | The kubelet cannot prove a name is non-root |
| The container runs as root although the Dockerfile says otherwise | 5 | USER is a default; --user and runAsUser override it |
| An image labelled arm64 runs on amd64 with a warning | 5 | --platform sets a field; nothing validates it |
CI is green and docker build --check exits 1 | 6 | An ordinary build prints the same findings and exits 0 |
| A bind mount is Permission denied and the UID is one nobody chose | 7 | useradd picked it and the image never published it |
docker run --rm <image> id prints something that is not an id | 7 | No shell, and ENTRYPOINT swallowed the argument |
Advice that has expired
| Still repeated | What changed |
|---|---|
“docker build uploads the context to the daemon” | BuildKit transfers only what the graph references — 40 B against the legacy builder’s 67.11 MB on the same directory. BuildKit became the default on Linux in Engine 23.0, February 2023 |
| “The Dockerfile runs top to bottom” | Unreferenced stages never execute, and referenced ones run in parallel |
| “One instruction, one layer” | Sixteen instructions produced five layers here; eleven instruction types write config only |
“Check docker history for layer sizes” | It reports snapshot blocks — 4.1 kB for a 32-byte layer — and no cache status at all |
“docker image prune cleans up after builds” | BuildKit leaves no dangling images; its cache needs docker builder prune |
“rm -rf /var/lib/apt/lists/* keeps the layer small” | RUN --mount=type=cache reaches /var/cache/apt as well, which is where the .deb files are |
| “Lint your Dockerfile with a third-party tool” | docker build --check has been built in since buildx 0.15 |
“docker build and docker buildx build are different commands” | In Engine 29 docker build is buildx, and an ordinary build emits an index plus a provenance attestation |
Everything measured on this page was measured on Engine 29.4.3 with buildx 0.33.0, BuildKit 0.29.0 and the containerd image store active. Where a number depends on the base image, the base is named.
Before you call it done
- Build once with
--progress=plainand read it. EveryRUNthat fetches something is a candidate for stage 1. - Run the hinge, and resolve its silence with the
docker history … | grep '^USER'follow-up. docker run --rm <image> id— and if it cannot run, know that and say so where an adopter will see it.- Run the thing you installed, inside the finished image, rather than trusting the step that installed it.
docker build --check .; echo $?as its own pipeline step, or# check=error=trueat the top of the file.- Pin the UID rather than letting
useraddchoose it, create the passwd entry, and publish the number. - Then read what ships in your container image before you push, because none of the above is about what the image publishes.
How to tell whether a page about building images is worth reading
Almost everything above comes out of one sentence:
The corpus reads a Dockerfile as a shell script that builds an image — so a line that ran is a change that happened, and a build that finished is an image that works.
Watch it generate the rest. A script’s lines all run, so every stage in your file executed — and two of them never did. A script’s input is the directory it sits in, so the directory is uploaded — and 40 B moved where 67 MB used to. A script’s lines each do one thing, so instructions and layers correspond — sixteen produced five, three of which were the same empty blob. A script’s output is the record of what happened, so the build log is evidence about the image — and a step that could not find curl printed DONE. A script is executed by a shell you chose, so pipefail is available — and the base image chose dash. And the structural one: a script’s job is finished when the script ends, so every page on this subject stops at docker push, which is where the page after this one has to start.
The absences worth noticing. No --progress=plain anywhere means the author has only ever seen collapsed output, and has therefore never read the stdout of a step that succeeded. No mention of CACHED means they have never watched a second build. A layer count that equals the instruction count means they have never run docker history on their own output. rm -rf /var/lib/apt/lists/* with no mention of --mount=type=cache dates the page before February 2023. USER appuser with no numeric UID means they have never run the image under anything that enforces runAsNonRoot. And FROM scratch or distroless recommended with no mention of CA certificates or /etc/passwd means the image has never made an outbound request or looked up its own user.
Related reading
- What ships in your container image — what happens to the artefact after the build: the config blob, the tag, the three digests, and when a digest stops resolving.
- Containers, All the Way Down — what happens when one runs.
- Docker basics — the commands, if you have not met them.
- Moving a service into a container — the far end of stage 7: adopting somebody else’s image, UID first.
- Kubernetes, Honestly — where
runAsNonRootis enforced and what refuses the pod. - The Life of a Package — what
apt-get installis doing inside thatRUN.