A build goes green, the image is pushed, the deploy succeeds, and every route returns 404. The pipeline has been green for months; docker build --check is clean; the digest in the deployment matches the one CI produced. Inside the image there is no front end, and there has not been one for three weeks, because a RUN that fetched the asset toolchain printed curl: not found on a base that had quietly moved to its slim variant, and then reported DONE, because /bin/sh -c reports the exit status of the last command in a pipeline. Every green step in a build log is a statement about the builder. None of it is a statement about the image. The builder can tell you what it did; only the image can tell you what you have, and this page is seven ways of asking it.
This page builds an image and then checks it. Every stage ends with a Verify line, and every one of those lines is a command run against the finished image rather than a line read out of the build log. What happens to the image after you push it is What ships in your container image‘s; what happens when one runs is Containers, All the Way Down‘s. Everything measured here was measured on Engine 29.4.3 with buildx 0.33.0, BuildKit 0.29.0 and the containerd image store active; where a number depends on the base image, the base is named.
The seven stages, and the one command that says which one you are in.
- Solved, not run — what the builder actually did with your file, and what a green step proves.
- The base image is a shell you did not choose — and a passwd file, and a certificate store.
- Layers, and the eleven instructions that make none.
- The cache is a graph — which is why a miss does not cascade down the file.
- The identity, and who has to resolve it —
USER, and why a name is not a number. - What the build never checked — and the exit code that hides it.
- The handover — and why the image you have just built defeats this site’s own procedure for adopting one.
The diagnostic hinge: ask what identity your image will run as, and who has to resolve it.
docker image inspect <the image you just built> --format '{{printf "%q" .Config.User}}'One quoted token comes back, and you are reading which of three kinds of thing it is — empty, a name, or a number — not its value. USER does not set a UID. It writes a string into the image’s config blob, and something that is neither the builder nor you has to turn that string into a running process. Which something, and whether it can, is the branch:
| What comes back | Whose fact it is | Stage |
|---|---|---|
"" | the base image publisher’s — either they set a USER and you inherited it, or they set none and you inherited the runtime’s default, which is root | 2, and see the silence below |
"app", and the name is in the image’s /etc/passwd | the base image publisher’s again: they wrote that passwd entry and chose the number behind it | 5, then 7 |
"app", and the name is not in the image’s /etc/passwd | the daemon’s: it refuses the container before your program starts, and the build that produced this image reported success | 5 — and why the build let it through is 1 and 6 |
"10001:10001" — numeric, with no matching passwd entry | nobody’s. getpwuid has nothing to answer with: HOME is /, whoami fails, and any library that looks up its own user fails with it | 2 and 7 |
a name, and your deployment sets --user, user: or runAsUser | whoever runs it — USER is advisory and has already been overridden | 5, and 7 for what to publish about it |
The silence, named, because it is the case this page exists for. A first Dockerfile very often has no USER line at all — and then the hinge prints "", which is exactly what a base image that set none also prints. It cannot tell those apart. One more command can, and resolving it is the whole of stage 2:
docker history --format '{{.CreatedBy}}' <image> | grep '^USER'Nothing printed means no USER instruction of yours is in this image, so the empty field is not a gap in your Dockerfile — it is a default you inherited, and stage 2 is about the other things you inherited alongside it. A line printed means you wrote one and it resolved to nothing, which is the third row of the table.
Stage 1: Solved, not run
Everyone learns the Dockerfile as a shell script: commands executed top to bottom, one after another. That model was correct, and the builder it describes and the one that replaced it are in the same binary on your machine today.
Here is a file with three stages, two of which nothing refers to. On the legacy builder:
$ DOCKER_BUILDKIT=0 docker build -t lab/dag:classic .
Step 1/6 : FROM scratch AS neverused
Step 2/6 : COPY marker-A /A
---> a5e1e5a962d9
...
Step 6/6 : COPY marker-C /CSix steps, all executed, two intermediate images left behind. Now the same file on the default builder, measured on Engine 29.4.3:
$ docker build -t lab/dag:buildkit .
#4 [final 1/1] COPY marker-C /C
#4 DONE 0.0sOne vertex. BuildKit reads your file into a graph, works out what the requested output depends on, and solves for that. The unreferenced stages never ran: change the unreferenced stage’s COPY to name a file that exists nowhere and the ordinary build still succeeds, while --target neverused fails at once with "/does-not-exist-anywhere": not found.
Two consequences arrive straight away. Stages that are referenced can run in parallel, so the order of numbered steps in your output is a scheduling artefact rather than a reading of your file. And .dockerignore has quietly changed jobs.
The build context is not uploaded
Same directory, same 64 MB file that no COPY names, three builds:
| Builder and Dockerfile | Transferred |
|---|---|
BuildKit, COPY app.txt /app.txt | 40 B |
BuildKit, COPY . /app | 67.13 MB |
| legacy builder, either file | 67.11 MB |
Forty bytes against sixty-seven megabytes, from one word in the Dockerfile. BuildKit transfers what the graph references and nothing else, so “.dockerignore keeps builds fast” describes a builder most readers stopped using when BuildKit became the default on Linux in Engine 23.0, February 2023. Its remaining job is keeping things out of the image, and that one is sharper than the size job ever was: COPY . /app ships your .git directory and every secret it has ever held (stage 1 of the page on what ships).
And a green step is not a step that worked
This is the one that ships broken images. A RUN that fetches an installer over a pipe, on a base image with no curl:
#6 [3/3] RUN curl -fsSL https://no.such.host.invalid/install.sh | sh
#6 0.245 /bin/sh: 1: curl: not found
#6 DONE 0.3s
$ echo $?
0The step reports DONE. The build exits 0. The image ships without the software it exists to carry, and nothing anywhere says so. The mechanism is not exotic and not BuildKit’s: /bin/sh -c reports the exit status of the last command in a pipeline, and pipefail is not on. RUN false | true behaves identically.
The error was printed, on a step that then reported success, in collapsed output, on a build nobody was watching. That is why the first line of the checklist at the end of this page is to run the build once with --progress=plain and read it.
Verify: do not read the log. Run the thing the step was supposed to install, inside the finished image: docker run --rm <image> sh -c 'command -v curl && curl --version'. A green build log is a statement about the builder; this is a statement about the image.
Stage 2: The base image is a shell you did not choose
Resolve the hinge’s silence first. If docker history --format '{{.CreatedBy}}' <image> | grep '^USER' printed nothing, you did not write a USER line and the empty field is something you inherited. FROM selects a set of defaults, and every one of them belongs to somebody else.
The shell
Stage 1’s curl | sh failure has a well-known remedy, and on a Debian-family base it does not work:
SHELL ["/bin/sh", "-o", "pipefail", "-c"]
#5 0.144 /bin/sh: 0: Illegal option -o pipefail/bin/sh on Debian and Ubuntu is dash, and dash has no pipefail. SHELL ["/bin/bash", "-o", "pipefail", "-c"] works where bash exists, which is not everywhere. The point to keep: your RUN lines are interpreted by a program the base image chose, and shell-form RUN never says which.
The dynamic linker
Go from a full base to FROM scratch and the build gets faster, smaller and green. Then:
$ docker run --rm lab/scratch:cgo
exec /probe: no such file or directory
$ docker run --rm lab/scratch:cgo ls -l /probe
ls: not found # there is no ls either; the file is thereThe file that does not exist is not the one named in the error. A CGO-linked Go binary is dynamically linked, and the missing file is /lib64/ld-linux-x86-64.so.2, the interpreter the kernel loads first; the error names the binary because that is the path execve was given. Build with CGO_ENABLED=0, copy the interpreter and the libraries in, or use a base that has them.
The certificate store, and the passwd file
Fix the linking and the same image fails differently, this time only in production, and only on the code path that talks to something:
x509: certificate signed by unknown authorityNothing is wrong with the certificate. FROM scratch has no /etc/ssl/certs/ca-certificates.crt, so there is no trust anchor to check it against: the system trust store is a file, and an empty image does not have it. The same goes for /etc/passwd — a numeric USER has nothing to resolve to, whoami fails, HOME is /, and any library that looks up its own user fails with it — and for /etc/nsswitch.conf and the zoneinfo database, each failing in its own idiom.
None of these is a reason not to use a minimal base. They are the reason to know that a minimal base is a list of absences, and every absence is something the previous base was quietly supplying.
Verify: docker run --rm <image> sh -c 'ls /etc/ssl/certs/ca-certificates.crt /etc/passwd; id'. If there is no shell to run it with, that is the answer to stage 7’s question as well, and you have met it early.
Stage 3: Layers, and the eleven instructions that make none
“One instruction, one layer” is repeated everywhere and it is not close. A file with sixteen instructions, built and then counted:
$ docker image inspect lab/census:1 --format '{{len .RootFS.Layers}}'
5| Produces a layer | Produces only a config field |
|---|---|
FROM, RUN, COPY, ADD, WORKDIR | ARG, ENV, LABEL, EXPOSE, VOLUME, USER, STOPSIGNAL, SHELL, HEALTHCHECK, ENTRYPOINT, CMD |
Eleven of the sixteen wrote nothing to a filesystem. They wrote to the image’s config blob, a small JSON document that travels beside the layers, is not a layer, and cannot be squashed. That is where ENV lands, and why a credential set with ENV or passed as --build-arg survives every multi-stage build and every rm.
Two of the instructions that did produce a layer produced an empty one. RUN true, and a WORKDIR on a directory that already existed, each got a real slot in the image, and the same empty blob occupies three of the five:
sha256:4f4fb700ef54… 32 B # RUN true
sha256:4f4fb700ef54… 32 B # WORKDIR /srv
sha256:4f4fb700ef54… 32 B # and once moreOne blob, three times: the empty gzipped tar, thirty-two bytes, in three separate entries in rootfs.diff_ids. A layer is a diff, not a step.
And the size column is not layer bytes
Here is the tool everyone is told to reach for, against the same image:
| Layer | docker history says | compressed blob | uncompressed tar |
|---|---|---|---|
RUN true / WORKDIR | 4.1 kB | 32 B | 1024 B |
COPY f.txt /srv/f.txt | 12.3 kB | 132 B | 2560 B |
4096 and 12288 bytes: snapshot blocks, not layer bytes. Under the containerd image store, the default on new installations from Engine 29, that column answers a storage question and not a shipping question; a legacy-built image reports 8.19 kB, two blocks, for a two-byte file. docker history is a reasonable way to read your own instructions back; it is not a size tool, and it is not a cache tool at all.
Verify: docker image inspect <image> --format '{{len .RootFS.Layers}}' against the number of instructions in your file. If they match, either you wrote only RUN and COPY, or you are looking at a legacy build.
Stage 4: The cache is a graph
What a cache key is made of — the instruction text for a RUN, the content and mode of the files for a COPY — is measured in stage 7 of the page on what ships, and this page does not re-derive it. What is left is the question almost everyone gets wrong: when a key misses, what does the miss propagate to?
The answer is “whatever depends on it”. Inside a single stage that is everything below it, because each instruction builds on the one before, which is where the file-order intuition comes from and why it survives. Across stages it is not. Here is a build with an assets stage written at line 2 and a build stage at lines 6 and 7, invalidated from each end in turn:
# change the input to the stage written FIRST
#8 [build 2/3] COPY src/main.go . CACHED
#9 [build 3/3] RUN go build CACHED
#11 [assets 2/3] COPY assets/logo.svg . 0.1s <- only this missed
# change the input to the stage written LAST
#7 [assets 2/3] COPY assets/logo.svg . CACHED
#9 [build 3/3] RUN go build 0.4s <- only this missedIt goes both ways, and neither direction cascades. Line order is not dependency order; the builder solved a graph and the file was only ever a description of it. On a cold build it numbers [assets 1/3] ahead of [build 2/3], which is written first.
The practical form: “put the things that change least at the top” is a rule about dependencies that happens to be expressible as a rule about position. Within one stage, COPY the lockfile, install, then COPY the source: the install does not depend on the source. Split the same work into two stages and the dependency is stated directly, position stops mattering, and the two halves rebuild independently, which is why multi-stage builds are faster as well as smaller.
What --no-cache does not clear
A RUN --mount=type=cache directory is not part of the layer cache. It is a separate, mutable record, and it survives:
$ docker build --no-cache . && docker build --no-cache .
$ docker buildx du --verbose | grep -A2 cachemount
Type: exec.cachemount
Mutable: truedocker builder prune -af clears them. And this is the mechanism behind the largest single size result on this page. Installing jq three ways, compressed layer bytes, on a Debian-family base built here with debootstrap rather than pulled:
| How | Compressed layer | Against plain |
|---|---|---|
plain apt-get install | 4,789,555 B | — |
&& rm -rf /var/lib/apt/lists/* | 4,070,636 B | −15 % |
--mount=type=cache on /var/cache/apt and /var/lib/apt/lists | 504,824 B | −89 % |
The folk remedy addresses the package lists and never touches /var/cache/apt/archives, which held 28 MB of .deb files on this base. Read those numbers as a mechanism, not as a target: the official Debian and Ubuntu images ship an apt.conf.d hook that deletes the downloaded packages for you, which narrows the first gap considerably. The ordering holds; the ratio is base-dependent.
One more thing that has moved: BuildKit leaves no dangling images at all. docker image prune was the right advice for a builder that created an intermediate image per step; the build cache that replaced them is invisible to docker images and untouched by that command. docker builder prune reclaims it.
Verify: build twice with no changes and confirm every step says CACHED; then change one input and read which steps missed. If the misses match the dependency and not the file order, you have understood this stage.
Stage 5: The identity, and who has to resolve it
This is the hinge’s stage. USER looks like it sets the user your container runs as. It writes a string into the config blob, and every failure below is somebody else failing to resolve that string.
Start with the case that is easiest to write and hardest to see. The Dockerfile says USER appuser and nothing ever created that account in the image:
$ docker build -f Dockerfile.missing -t lab/user:missing .
BUILD SUCCEEDED
$ docker run --rm lab/user:missing id
docker: Error response from daemon: unable to find user appuser:
no matching entries in passwd fileThe build validated nothing. It wrote the string down. The daemon has to turn it into a UID, and does so at docker run by reading /etc/passwd inside the image, so this fails on every machine, forever, and it failed for the first time somewhere that is not your laptop.
Create the account and the string resolves, which introduces the opposite problem: the number is now real and nobody has written it down. useradd picked it; your Dockerfile does not contain it; the bind mount in stage 7 is about to find out what it is.
Write the number yourself — USER 10001:10001 — and a third party appears. Numeric works everywhere and resolves to nothing: with no matching passwd entry, getpwuid has no answer, whoami fails, HOME is /, and any library that looks up its own user fails alongside them. The honest form is both: create the account and pin the number, so the name resolves inside the image and the number is one you chose rather than one useradd chose.
There is a fourth party, and it turns this into a production incident rather than a build error. A Kubernetes kubelet asked to enforce runAsNonRoot cannot prove a name is non-root without resolving it, and refuses: container has runAsNonRoot and image has non-numeric user (appuser), cannot verify user is non-root. That behaviour is widely reported and was not reproduced here; Kubernetes, Honestly is where the kubelet’s checks live. It is the second branch of the hinge arriving weeks later on somebody else’s cluster.
Two more strings the build writes and does not check
USER is a default and nothing enforces it. docker run --user 0:0 overrides it and writes happily to files the image made root-owned; a compose user: key or a Kubernetes runAsUser does the same more quietly. If the image must not run as root, something outside the image has to say so.
And --platform is a label. Build with --platform linux/arm64 over a COPY of an x86-64 binary and the build succeeds, the config says linux/arm64, and the container runs on amd64 with a warning:
$ docker image inspect lab/plat:arm64 --format '{{.Os}}/{{.Architecture}}'
linux/arm64
$ file probe
probe: ELF 64-bit LSB executable, x86-64--platform selects base image variants and sets a field. Nothing validates that field against the bytes you brought. Cross-building means a compiler that targets the other architecture, or emulation; the flag on its own means neither.
Verify: run the hinge, then docker run --rm <image> id and compare. A UID you cannot account for, or an error from the daemon, are the two outcomes this stage exists to produce before anyone else finds them.
Stage 6: What the build never checked
There is a linter built into the builder, and its default is the interesting part. On a Dockerfile with a secret in an ENV, a shell-form ENTRYPOINT and a legacy key-value LABEL:
$ docker build --check .
SecretsUsedInArgOrEnv, JSONArgsRecommended, LegacyKeyValueFormat
$ echo $?
1
$ docker build -t lab/warn:1 .
... the same three warnings printed ...
$ echo $?
0The same three findings, printed either way, and the exit code differs. CI runs the second command; the warnings scroll past in a job that goes green, and the image gets pushed. Two ways to close it: run docker build --check . as its own step, or put # check=error=true as the first line of the Dockerfile, which makes the ordinary build exit 1.
It is a Dockerfile linter, built in since buildx 0.15, replacing the third-party tool most guides still recommend. It reads your file. It has no opinion about your image: it cannot know that curl was missing in stage 1, that appuser does not exist in stage 5, that the binary you copied is the wrong architecture, or that its interpreter is absent. Everything this page has shown you passes --check.
One of its rules, JSONArgsRecommended, is worth taking seriously: Containers, All the Way Down explains what shell-form CMD does to signal handling and to your process’s claim on PID 1, and why docker stop then takes exactly ten seconds.
Verify: docker build --check .; echo $?. If it is 1 and your pipeline is green, your pipeline is not running this command.
Stage 7: The handover
The image is not for you. The last thing that happens to it is that somebody else has to run it — with a volume, on a host, under a scheduler — and everything that person needs is a number the config blob does not carry.
$ docker run --rm -v "$PWD/data:/data" lab/user:1 sh -c 'id -u; id -un; touch /data/x'
10001
app
touch: cannot touch '/data/x': Permission deniedThe builder chose 10001 and told nobody. Ask the image what it will hand over and it answers with a name and two nulls:
$ docker image inspect lab/user:1 \
--format 'user={{.Config.User}} volumes={{json .Config.Volumes}} ports={{json .Config.ExposedPorts}}'
user=app volumes=null ports=nullThe host filesystem has never heard of app, and it never will: a bind mount carries UID numbers across the boundary and nothing translates them. Moving a service into a container documents the far end of exactly this and prescribes docker run --rm <image> id as the first thing an adopter should run.
Which is where the page turns around on itself. Take the small, non-root, FROM scratch image that the six stages above have been teaching you to build, and run that diagnostic against it:
$ docker run --rm lab/scratch:static id -u
hello from /probeThere is no id, there is no shell, and the ENTRYPOINT swallowed the argument and ran the application instead. A good build produces an image that defeats this site’s own published procedure for adopting one — and it does so silently, by printing something plausible.
The fix is not to put a shell back. It is to stop making the next person interrogate the image and publish the number instead: pin the UID rather than letting useradd pick it, create the passwd entry so a name resolves as well, and write the number where you write the ports and the volumes. An image that has to be reverse-engineered before it can be mounted is finished from the builder’s point of view and nobody else’s.
Verify: docker run --rm <image> id, and if it cannot run, docker image inspect --format '{{.Config.User}}' — then check that whatever it prints appears somewhere a person will read before they mount a volume.
A worked diagnosis: the image that had not contained the application for three weeks
Back to the deploy from the top of the page: 404 on every route, a rollback that works — so the team assumes a bad release and looks at the application, where nothing in the diff explains it — a pipeline green for months, a clean docker build --check, and a digest in the deployment that matches the one CI produced.
Run the hinge against the released image:
$ docker image inspect registry.internal/svc:2.31.0 --format '{{printf "%q" .Config.User}}'
""
$ docker history --format '{{.CreatedBy}}' registry.internal/svc:2.31.0 | grep '^USER'
$Empty, and no USER instruction of theirs — so this is stage 2, an inherited default, and the base image is worth looking at. The hinge is pointing at the right neighbourhood and naming the wrong stage. The base had changed — someone closed a size ticket three weeks earlier by moving from the full image to the slim one — but the mechanism that turned that into a broken release is stage 1’s.
Two commands finish it. First, against the image rather than the log:
$ docker run --rm registry.internal/svc:2.31.0 sh -c 'command -v curl; ls /opt/app/dist | head -1'
$ # nothing. no curl, and no dist.And then the build, re-run with the output uncollapsed:
$ docker build --progress=plain --no-cache . 2>&1 | grep -B1 -A1 'not found'
#9 0.201 /bin/sh: 1: curl: not found
#9 DONE 0.2sThe line had been printed on every build for three weeks, on a step that then reported DONE. The slim base had no curl; the asset build step fetched nothing, installed nothing, and exited 0 because /bin/sh -c reports the last command in a pipeline. Every release since had shipped an image with no front end. The rollback worked because the digest it rolled back to was built before the base changed, which is why the deploy looked like the cause.
Three things generalise. The hinge did its work by naming a party rather than a cause, and it was one stage off, which is the normal case. A pipeline that gates on the build’s exit code is gating on the builder’s opinion of itself. And the sentence this page turns on: every green step in a build log is a statement about the builder; none of it is a statement about the image. The only way to learn something about an image is to ask the image.
Symptoms, and which stage they belong to
| Symptom | Stage | What is actually true |
|---|---|---|
An error was printed and the step still reported DONE | 1 | /bin/sh -c reports the last command in a pipeline |
| An instruction in your Dockerfile never ran | 1 | Unreferenced stages are not in the solved graph |
| The build transfers hundreds of megabytes every time | 1 | COPY . /app — the graph references the whole directory |
The image is missing software a RUN was supposed to install | 1 | The step exited 0 and nothing checked the result |
Illegal option -o pipefail | 2 | /bin/sh is dash on a Debian-family base |
exec <binary>: no such file or directory, for a file that is present | 2 | The missing file is the dynamic linker, not the binary |
x509: certificate signed by unknown authority from a minimal image | 2 | No CA bundle in the image to check against |
whoami fails and HOME is / | 2 | A numeric UID with no matching /etc/passwd entry |
| The layer count does not match the instruction count | 3 | Eleven instruction types write only config |
docker history sizes do not add up to anything | 3 | It reports snapshot blocks, not layer bytes |
| A cache miss did not invalidate the steps below it | 4 | Invalidation follows the graph, not the file |
--no-cache and the build is still suspiciously fast | 4 | Cache mounts are separate records and survive it |
docker image prune reclaims nothing after a build | 4 | BuildKit leaves no dangling images; use builder prune |
unable to find user <name>: no matching entries in passwd file | 5 | The daemon resolves USER at run time, from the image’s passwd |
| A pod is refused for having a non-numeric user | 5 | The kubelet cannot prove a name is non-root |
| The container runs as root although the Dockerfile says otherwise | 5 | USER is a default; --user and runAsUser override it |
| An image labelled arm64 runs on amd64 with a warning | 5 | --platform sets a field; nothing validates it |
CI is green and docker build --check exits 1 | 6 | An ordinary build prints the same findings and exits 0 |
| A bind mount is Permission denied and the UID is one nobody chose | 7 | useradd picked it and the image never published it |
docker run --rm <image> id prints something that is not an id | 7 | No shell, and ENTRYPOINT swallowed the argument |
Before you call it done
- Build once with
--progress=plainand read it. EveryRUNthat fetches something is a candidate for stage 1. - Run the hinge, and resolve its silence with the
docker history … | grep '^USER'follow-up. docker run --rm <image> id— and if it cannot run, know that and say so where an adopter will see it.- Run the thing you installed, inside the finished image, rather than trusting the step that installed it.
docker build --check .; echo $?as its own pipeline step, or# check=error=trueat the top of the file.- Pin the UID rather than letting
useraddchoose it, create the passwd entry, and publish the number. - Then read what ships in your container image before you push, because none of the above is about what the image publishes.
Related reading
- What ships in your container image — what happens to the artefact after the build: the config blob, the tag, the three digests, and when a digest stops resolving.
- Containers, All the Way Down — what happens when one runs.
- Docker basics — the commands, if you have not met them.
- Moving a service into a container — the far end of stage 7: adopting somebody else’s image, UID first.
- Kubernetes, Honestly — where
runAsNonRootis enforced and what refuses the pod. - The Life of a Package — what
apt-get installis doing inside thatRUN.