Containers, All the Way Down: What docker run Actually Does

There is no container() system call. There is no container object in the kernel, no container ID the kernel knows about, and no single boundary you can point at. A running container is an ordinary Linux process that had nine separate things done to it before it started, most of which have nothing to do with each other and any of which can be the reason yours is not working.

That is not a pedantic framing. It is the only framing that makes container problems tractable. “The container will not start” is not a diagnosis, because five of those nine stages happen before the kernel has been asked to isolate anything at all — they are HTTP requests, JSON documents, tar archives and mount options. Three more are process attributes you can read straight out of /proc. The last is an ordinary execve.

This page follows a single docker run the whole way down. If you have not read containers explained, start there — it covers what containers are for and when to use them. This one is about the machinery, and it assumes you already accept the premise.

docker inspect -f '{{.State.Pid}} {{.State.ExitCode}} {{.State.Error}}' mycontainer

docker inspect -f '{{.State.Pid}} {{.State.ExitCode}} {{.State.Error}}' mycontainer

Which programs are actually involved

Before the stages, the cast. Typing docker run on a default Linux install sets four separate programs in motion:

The shim is the piece people are surprised by. Its job is to be the parent that stays: it routes the container’s standard streams through FIFOs, mounts and unmounts the rootfs, acts as a sub-reaper for orphaned processes, and — the point of the design — survives a containerd restart, with containerd reconnecting to it over a socket afterwards. That is why you can restart the Docker daemon without killing every container on the box.

The shim’s name is generated, not chosen. containerd takes the runtime name io.containerd.runc.v2, keeps the last two components, replaces dots with hyphens and prepends containerd-shim. Since containerd 2.0 the v2 shim is the only one — io.containerd.runtime.v1.linux and io.containerd.runc.v1 were both removed.

Podman deletes the first three rows. There is no daemon and no supervisor: podman run does the work itself in the calling process, leaves a small conmon per container in place of the shim, and calls the runtime directly. That difference is the reason Podman integrates with systemd naturally and Docker does not.

Two live 2026 caveats. Docker Engine 29.7 added an experimental embedded-containerd mode that runs containerd inside dockerd rather than beside it, which collapses two of those rows into one. And on Red Hat, the runtime is not runc: RHEL 10 removed runc outright and ships crun, with existing installs needing podman system migrate --new-runtime=crun. Podman’s documented search order puts crun first everywhere, so on Fedora and RHEL that is what you are running.

# Everything a "container" is, read off an ordinary process
PID=$(docker inspect -f '{{.State.Pid}}' mycontainer)

ls -l /proc/$PID/ns/           # the namespaces it is in
cat /proc/$PID/cgroup          # the cgroup it is in
grep Cap /proc/$PID/status     # its capability sets
grep Seccomp /proc/$PID/status # 0 disabled, 2 filtered

# And from the other direction
lsns -p $PID
sudo nsenter -t $PID -a /bin/sh

# Everything a "container" is, read off an ordinary process
PID=$(docker inspect -f '{{.State.Pid}}' mycontainer)

ls -l /proc/$PID/ns/           # the namespaces it is in
cat /proc/$PID/cgroup          # the cgroup it is in
grep Cap /proc/$PID/status     # its capability sets
grep Seccomp /proc/$PID/status # 0 disabled, 2 filtered

# And from the other direction
lsns -p $PID
sudo nsenter -t $PID -a /bin/sh

Stage 1 — The CLI sends an HTTP request

docker run nginx does no container work. It serialises your flags into JSON and POSTs them to /var/run/docker.sock, a Unix socket speaking HTTP. Everything after that happens in a daemon running as root.

# Talk to the socket yourself
curl --unix-socket /var/run/docker.sock http://localhost/version
curl --unix-socket /var/run/docker.sock http://localhost/containers/json

# Which is why this is enough to own the machine
curl --unix-socket /var/run/docker.sock http://localhost/images/json

# Talk to the socket yourself
curl --unix-socket /var/run/docker.sock http://localhost/version
curl --unix-socket /var/run/docker.sock http://localhost/containers/json

# Which is why this is enough to own the machine
curl --unix-socket /var/run/docker.sock http://localhost/images/json

Two consequences follow immediately, and they are the reason this stage is worth naming at all.

Membership of the docker group is equivalent to root. Not “close to”, not “a risk” — equivalent. Anyone who can write to that socket can ask the daemon to run a container that bind-mounts / with --privileged, and the daemon is root. This is the single most consequential fact about Docker’s architecture, and it is not a bug: it is what a root daemon with an unauthenticated local API means. Permissions and privilege covers why no amount of capability trimming inside the container changes it.

Relative paths are resolved by the daemon, not by you. A bind mount source is interpreted in the daemon’s filesystem view. On a normal Linux host those coincide; over a remote context or with Docker Desktop, where the engine runs inside a VM, they do not, and this is where “the mount is empty” comes from.

Docker Engine 29.0 raised the minimum API version to v1.44, which means clients older than v25.0 no longer work against a current daemon. If an old CI image suddenly cannot talk to your engine, that is why.

Stage 2 — A name becomes a digest

nginx is shorthand for docker.io/library/nginx:latest. Resolving it is a sequence of ordinary HTTPS GETs against a registry API, and every step is inspectable.

1. The tag is fetched, returning an index (application/vnd.oci.image.index.v1+json) — a list of descriptors, one per platform.
2. The descriptor matching your architecture is followed to a manifest (...image.manifest.v1+json).
3. The manifest names exactly one config blob and an ordered list of layer blobs.
4. The config blob carries the entrypoint, environment, working directory, and rootfs.diff_ids.

# Look at all of it without pulling anything
skopeo inspect --raw docker://docker.io/library/nginx:latest | jq .
skopeo inspect --raw docker://alpine:latest | jq '.manifests[].platform'

# Or with the tools you already have
docker manifest inspect nginx:latest
docker image inspect nginx:latest --format '{{json .RootFS.Layers}}' | jq

# Look at all of it without pulling anything
skopeo inspect --raw docker://docker.io/library/nginx:latest | jq .
skopeo inspect --raw docker://alpine:latest | jq '.manifests[].platform'

# Or with the tools you already have
docker manifest inspect nginx:latest
docker image inspect nginx:latest --format '{{json .RootFS.Layers}}' | jq

What “the digest of an image” refers to is the thing most often got wrong. The sha256:... in docker pull image@sha256:... is the digest of the top-level JSON document — the manifest, or for a multi-architecture image the index. It is not a hash of the filesystem and not a hash of the layers. That is precisely why one digest can be correct on both amd64 and arm64: the index is identical, and it points at different manifests.

There is a second digest distinction that bites when you compare things: a layer’s descriptor digest is the hash of the compressed blob, while its diff_id in the config is the hash of the uncompressed tar. Same layer, two hashes of two different byte streams. If two images “contain the same layer” but show different digests, compare diff_ids before concluding anything.

The OCI Image Specification reached 1.1.0 on 15 February 2024 and is at 1.1.1 as of April 2025. That release is why registries can now hold things that are not images at all: artifactType makes Helm charts and Wasm modules first-class, subject links an artifact to an image it describes, and the Referrers API (GET /v2/<name>/referrers/<digest>) is how signatures and SBOMs are discovered. If you last read the spec when “OCI artifacts” were a proposal, they are standard now.

Stage 3 — What a layer actually is

A layer is a tar archive of the difference between one filesystem state and the next, usually gzip-compressed. There is nothing clever in it. The only non-obvious part is how a layer says “this file is gone”, because tar has no way to record a deletion.

The specification answers it with naming conventions:

Two rules make this safe: a whiteout only ever applies to lower layers, never to entries in its own, and the whiteout file itself must be hidden after it has been applied — you never see a .wh. file in a running container.

On disk, the same deletion looks completely different. OverlayFS represents a whiteout as a character device with device number 0/0, or alternatively as a zero-length regular file carrying the trusted.overlay.whiteout extended attribute. An opaque directory carries trusted.overlay.opaque set to y. The .wh. filename exists only inside the archive; unpacking translates it. Knowing that the two representations are the same thing saves a lot of confusion when you go looking in the storage directory.

# Pull an image to a directory and look inside the layers
skopeo copy docker://alpine:latest dir:/tmp/alpine
ls /tmp/alpine
tar tzf /tmp/alpine/<blob-digest> | grep '\.wh\.'

# Find whiteouts in unpacked storage
sudo find /var/lib/docker/overlay2 -type c -size 0 2>/dev/null | head

# Pull an image to a directory and look inside the layers
skopeo copy docker://alpine:latest dir:/tmp/alpine
ls /tmp/alpine
tar tzf /tmp/alpine/<blob-digest> | grep '\.wh\.'

# Find whiteouts in unpacked storage
sudo find /var/lib/docker/overlay2 -type c -size 0 2>/dev/null | head

Three layer media types are current and none is deprecated: plain tar, tar+gzip, and tar+zstd. zstd is a fully standard layer format, containerd handles it, and zstd:chunked — the format Podman and CRI-O use for lazy pulling — produces valid zstd layers that any zstd-capable runtime can consume; only the partial-pull optimisation is implementation-specific. The nondistributable media type family is deprecated and should not be used in anything new. Whether a particular registry accepts zstd pushes is a per-registry question and worth testing rather than assuming.

This is also where the practical advice about image size comes from. A layer that deletes a file does not make the image smaller, because the earlier layer containing it is still shipped. Removing a build dependency in a later RUN adds a whiteout and adds bytes. That is what multi-stage builds exist to solve, and containerising a service works through it. Size is only half of the consequence, though. That earlier layer is also still readable — by anyone who can pull the image, with one command and no privileges. A secret fetched in one RUN and deleted in the next is published permanently, and rebuilding does not unpublish it; the same goes for anything set with ENV or passed as --build-arg, which are not in a layer at all and which no squash or multi-stage build can reach. What ships in your container image extracts a key back out of a registry and covers the rest.

Stage 4 — Assembling a root filesystem

Unpacked layers are stacked into one directory tree by OverlayFS, which takes three kinds of directory:

# See the real mount for a running container
findmnt -t overlay
mount | grep overlay

# Everything the container has written since it started
docker inspect -f '{{.GraphDriver.Data.UpperDir}}' mycontainer
docker diff mycontainer

# See the real mount for a running container
findmnt -t overlay
mount | grep overlay

# Everything the container has written since it started
docker inspect -f '{{.GraphDriver.Data.UpperDir}}' mycontainer
docker diff mycontainer

docker diff is underused. It lists exactly what is in the upper directory — every file the container has added, changed or whited out — which is the fastest way to find out that an application is writing gigabytes into its own writable layer instead of the volume you gave it.

The performance rule that follows from the design: modifying an existing file copies it up in full first. Overlay works at file granularity, not block granularity, so appending one line to a 2 GB file inherited from an image copies 2 GB before the append happens. Databases and log files belong on volumes for this reason and not only for durability. The life of a write covers what happens below the mount once the copy-up is done.

Unprivileged users could not mount overlay filesystems for most of its history, which is why fuse-overlayfs existed. That changed: unprivileged overlay inside a user namespace works on kernel 5.12 and newer per Podman’s own documentation, using the userxattr mount option, which switches overlay from the trusted.overlay. extended attribute namespace to user.overlay. — one option, and the whole rootless storage story changes.

And overlay is no longer the only answer, which is genuinely new. containerd 2.3 — the current LTS — ships a native EROFS snapshotter: layers are converted from tar into EROFS images rather than unpacked file by file, each layer gets its own fsync instead of a whole-filesystem syncfs, and per-layer fs-verity and dm-verity become possible. It needs Linux 5.4, and file-backed mounts without loop devices need 6.12. Separately, composefs is close to stable as a bootc storage backend and ships as a Technology Preview in RHEL 10. If you last looked at container storage when the choice was overlay2 versus devicemapper, that landscape has moved.

Stage 5 — The bundle, the shim and the runtime

What containerd hands the runtime is not an image and not a container. It is a bundle: a directory containing the assembled rootfs and a single config.json conforming to the OCI Runtime Specification, currently v1.3.0 (4 November 2025). That file is the contract, and it is worth reading once.

# Generate a default bundle config and read it
mkdir -p /tmp/bundle/rootfs && cd /tmp/bundle && runc spec
jq '.process.capabilities, .linux.namespaces, .linux.seccomp.defaultAction' config.json

# The generated config for a live container
sudo cat /run/containerd/io.containerd.runtime.v2.task/moby/$ID/config.json | jq .

# List containers behind Docker's back
sudo ctr --namespace moby containers list

# Generate a default bundle config and read it
mkdir -p /tmp/bundle/rootfs && cd /tmp/bundle && runc spec
jq '.process.capabilities, .linux.namespaces, .linux.seccomp.defaultAction' config.json

# The generated config for a live container
sudo cat /run/containerd/io.containerd.runtime.v2.task/moby/$ID/config.json | jq .

# List containers behind Docker's back
sudo ctr --namespace moby containers list

The specification mandates a split that explains a lot of otherwise confusing behaviour. create must build the entire runtime environment but must not run your program; everything in config.json except process.args is applied. start then runs it. So runc create does all the expensive work — namespaces, cgroups, mounts, rootfs — and parks a helper called runc init inside the finished container, blocked on a pipe. runc start writes one byte to that pipe. runc init finishes its last few steps and execves your entrypoint, replacing itself.

That is why PID 1 in your container is your process and not a runtime helper, and why docker create followed later by docker start is fast: the second half is trivial.

Which runtime runs is a genuine question in 2026, not a formality. runc remains Docker’s default. crun, written in C rather than Go, is the default on Fedora and the only option on RHEL 10. youki, in Rust, is a CNCF Sandbox project whose README now describes production adoption — but it is still 0.x with no 1.0, so treat it accordingly. All three read the same config.json, which is the entire point of having a specification.

Stage 6 — Namespaces

Now the kernel gets involved. A namespace makes one global resource appear private to a set of processes. There are eight kinds, and there have been eight since Linux 5.6 — nothing has been added in any 6.x or 7.x kernel, so if you learned this list a few years ago it is still complete.

They are independent, and a default container does not use all of them. --network host means the network namespace is not created; --pid host means the PID namespace is not; a rootful Docker container by default does not use a user namespace at all, which is why root inside it is root outside it. There is no all-or-nothing switch.

The time namespace is the one to be careful about: it virtualises CLOCK_MONOTONIC and CLOCK_BOOTTIME and their variants, and explicitly does not virtualise CLOCK_REALTIME — the kernel documentation says virtualisation of that clock was avoided for complexity and overhead. You cannot give a container a different wall-clock date this way.

Two ordering facts explain most hand-rolled namespace confusion. CLONE_NEWPID and CLONE_NEWTIME do not move the caller — only its subsequently created children — which is why unshare --pid needs --fork and why /proc/PID/ns/pid_for_children exists as a separate entry. And creating a user namespace together with any other namespace in a single unshare() call does not require CAP_SYS_ADMIN, because the new user namespace grants a full capability set within itself. That one sentence in unshare(2) is the mechanism that makes rootless containers possible at all.

# Build a container's isolation by hand, one flag at a time
sudo unshare --uts --pid --fork --mount-proc /bin/sh
hostname isolated
ps aux            # only what is inside

# Namespaces on the system, and who is in them
lsns
lsns -t net

# Two processes in the same namespace show the same inode
readlink /proc/self/ns/net
readlink /proc/1/ns/net

# Build a container's isolation by hand, one flag at a time
sudo unshare --uts --pid --fork --mount-proc /bin/sh
hostname isolated
ps aux            # only what is inside

# Namespaces on the system, and who is in them
lsns
lsns -t net

# Two processes in the same namespace show the same inode
readlink /proc/self/ns/net
readlink /proc/1/ns/net

Joining a namespace is setns(2), which is what nsenter and docker exec do. Since Linux 5.8 it accepts a pidfd from pidfd_open(2), which lets you OR several CLONE_NEW* constants together and join multiple namespaces in one atomic call rather than one file descriptor at a time — the modern detail most write-ups predate. Joining a PID namespace still affects only children, never the caller.

The network namespace is where a container’s packets begin their journey; the life of a packet follows one out of a veth pair, through the host’s tables and onto the wire, and there is no point repeating it here.

Stage 7 — The cgroup

Namespaces control what a process can see. Cgroups control what it can use. They are completely separate mechanisms that people routinely blur together, and the practical consequence is direct: a container with no limits set has none. It can consume the whole machine. Isolation is not limitation.

docker run --memory=512m --cpus=1.5 --pids-limit=100 myimage

# Where that actually landed
systemd-cgls
cat /sys/fs/cgroup/system.slice/docker-$ID.scope/memory.max
cat /sys/fs/cgroup/system.slice/docker-$ID.scope/cpu.max
cat /sys/fs/cgroup/system.slice/docker-$ID.scope/memory.events

docker run --memory=512m --cpus=1.5 --pids-limit=100 myimage

# Where that actually landed
systemd-cgls
cat /sys/fs/cgroup/system.slice/docker-$ID.scope/memory.max
cat /sys/fs/cgroup/system.slice/docker-$ID.scope/cpu.max
cat /sys/fs/cgroup/system.slice/docker-$ID.scope/memory.events

memory.events is the file to reach for when something is being killed: its oom_kill counter is the only unambiguous evidence that the cgroup’s limit, rather than the host running out of memory, is what ended your process.

memory.high and memory.max are not two names for the same idea. max is a hard limit: reach it, fail to reclaim, and the OOM killer runs inside the cgroup. high is a throttle: exceeding it puts the cgroup under escalating reclaim pressure and slows the allocator down, and going over it never invokes the OOM killer — the kernel documentation says the limit may even be breached under extreme conditions. Docker’s --memory sets max. If you want a container to be slowed rather than shot, high is the file you want, and you will be setting it through systemd or by hand.

The state of cgroup v1 has three different answers and blurring them produces confident wrong statements.

The practical version: since systemd 258 there is effectively no current distribution booting legacy or hybrid, so if you meet cgroup v1 in 2026 you are on an enterprise 8-series host. I have not been able to confirm that any v1 controller has actually been deleted from the kernel as of 7.2 — deprecation warnings and removal proposals are documented; a completed removal is not.

The controller list a few years of writing will have missed is dmem — device memory accounting, effectively GPU VRAM, merged in Linux 6.14. It is a controller, not a namespace, and it is the interesting recent addition in this area.

Pressure information is on by default: cpu.pressure, memory.pressure, io.pressure and irq.pressure exist per cgroup, with a cgroup.pressure control file defaulting to 1. Reading memory.pressure on a struggling container tells you whether it is short of memory long before anything is killed.

Stage 8 — Closing the sandbox

This is the stage with the most steps and the most surprises. Reading runc’s own initialisation code, the order inside the container process is roughly: mounts and pivot_root; console setup; remounting things read-only; hostname; AppArmor profile; sysctls; masked and read-only paths; then the sync with the parent that makes runc create return; then seccomp; then capabilities, setgid, setuid, chdir and no_new_privs; then execve.

The root switch is pivot_root(2), not chroot(2), and the difference is the point. chroot changes a process’s idea of / while leaving the old root mounted and reachable — a process holding an open directory file descriptor from outside can fchdir() back to it, which is the classic escape. pivot_root moves the old root to a location that can then be unmounted, after which it is genuinely gone rather than merely un-navigated.

Its constraints explain several “invalid argument” failures when people build containers by hand: the new root must itself be a mount point and cannot be /; the old-root location must be at or under the new root; and the parent mount must not be shared, which is why a runtime makes the container’s mounts private first. It also historically failed outright when the current root was an initramfs — from power-on to a login prompt covers why, and why Linux 7.0’s nullfs changed it.

Seccomp is always installed before execve. Where it sits relative to dropping capabilities is conditional, and this is the detail almost every article gets wrong in one direction or the other. runc’s own comment says it plainly: without no_new_privs, installing a seccomp filter is a privileged operation, so it must happen before capabilities are dropped; with no_new_privs set it is deferred until as late as possible, so that as few syscalls as possible run under the filter before your program starts.

Docker’s default seccomp profile is an allowlist with a default action of SCMP_ACT_ERRNO, and Docker’s documentation describes it as blocking “around 44” syscalls out of 300-plus. Treat 44 as a floor rather than a figure — there is an open documentation bug noting the published table no longer matches the shipped profile, and it has grown since:

One clause more, added after this page was written: which errno is the whole story. SCMP_ACT_ERRNO carries a number of the policy author’s choosing, and that number decides whether a program that never asked for the blocked call keeps working. Block clone3 and answer ENOSYS, and glibc falls back to clone and the program runs. Block the same call and answer EPERM, and every pthread_create in the container fails — because a C library reads ENOSYS as “not here, try the old way” and everything else as a hard refusal. Docker’s own default profile changed to ENOSYS in 20.10.10, October 2021; hand-written and vendored profiles still carry the old one, which is why containers that have run for years break on a base-image bump with nothing in the changelog. The Life of a System Call owns that mechanism and the diagnosis it produces.

• Docker 25.0.0 blocked io_uring_setup, io_uring_enter and io_uring_register, citing container-escape concerns. containerd removed io_uring from its default allowlist in 2.0.
• Docker 29.0.0 blocked AF_ALG sockets and the socketcall(2) multiplexer — with the documented side effect that this can break 32-bit programs.

# What this container actually holds
docker run --rm alpine grep CapEff /proc/self/status
docker run --rm --cap-drop=ALL alpine grep CapEff /proc/self/status

# Decode it
capsh --decode=00000000a80425fb

# The sane default for anything you wrote yourself
docker run --cap-drop=ALL --cap-add=NET_BIND_SERVICE \
  --security-opt no-new-privileges myimage

# What this container actually holds
docker run --rm alpine grep CapEff /proc/self/status
docker run --rm --cap-drop=ALL alpine grep CapEff /proc/self/status

# Decode it
capsh --decode=00000000a80425fb

# The sane default for anything you wrote yourself
docker run --cap-drop=ALL --cap-add=NET_BIND_SERVICE \
  --security-opt no-new-privileges myimage

Stage 9 — execve, and being PID 1

runc init calls execve and ceases to exist. Your program is now the only thing in the PID namespace, and it is PID 1 — a role it was almost certainly not written for. How a program runs covers what execve does to an address space; what matters here is what PID 1 means.

PID 1 does not get default signal handlers. For every other process, a signal with no installed handler takes the kernel’s default action — SIGTERM terminates. For PID 1 the kernel suppresses that: a signal with no explicit handler is ignored. This is a deliberate protection for the real init and it applies inside every PID namespace.

The consequence is the ten-second pause everyone has seen and few have explained. docker stop sends SIGTERM; a shell script or a language runtime that installs no handler ignores it; ten seconds later Docker sends SIGKILL. Your process did not “take a while to shut down” — it never received a signal it was willing to act on. Signals goes through the rules.

PID 1 also inherits every orphaned process in the namespace and is responsible for reaping them. A program that never calls wait() accumulates zombies until the PID limit stops it.

# Shell form: your process is a child of sh, and sh ignores SIGTERM
CMD npm start

# Exec form: your process IS PID 1 and receives signals
CMD ["npm", "start"]

# Or let Docker supply a real init that forwards and reaps
docker run --init myimage

# Shell form: your process is a child of sh, and sh ignores SIGTERM
CMD npm start

# Exec form: your process IS PID 1 and receives signals
CMD ["npm", "start"]

# Or let Docker supply a real init that forwards and reaps
docker run --init myimage

The shell versus exec form of CMD is not a style preference. Shell form wraps your program in /bin/sh -c, which becomes PID 1, ignores SIGTERM, and does not forward it. The shell, in depth explains what else that wrapper does to your arguments on the way past.

When the process exits, its status is the container’s exit status, and the shim reports it and goes away. docker inspect -f '{{.State.ExitCode}}' reads it back. 137 is 128 + 9, meaning SIGKILL — usually the OOM killer or the end of a docker stop timeout, and memory.events is how you tell which.

Rootless changes stages 6, 7 and 8

Rootless containers are the same nine stages with one extra namespace created first. A user namespace maps your UID to root inside it, and every subsequent privileged-looking operation is performed with capabilities that are real inside that namespace and meaningless outside it.

Mapping more than a single ID needs help, because writing a multi-ID mapping to /proc/self/uid_map is privileged. That help is the setuid pair newuidmap and newgidmap, which consult /etc/subuid and /etc/subgid:

# A user needs at least 65536 subordinate IDs
grep $USER /etc/subuid /etc/subgid
# kevin:100000:65536

# What the mapping ended up as
podman unshare cat /proc/self/uid_map
podman info --format '{{.Host.IDMappings}}'

# After editing subuid/subgid, this is mandatory
podman system migrate

# A user needs at least 65536 subordinate IDs
grep $USER /etc/subuid /etc/subgid
# kevin:100000:65536

# What the mapping ended up as
podman unshare cat /proc/self/uid_map
podman info --format '{{.Host.IDMappings}}'

# After editing subuid/subgid, this is mandatory
podman system migrate

Missing or too-small /etc/subuid entries are the single most common rootless failure, and the errors are unhelpful. Check that first, every time.

Rootless networking has changed in both ecosystems, in different directions, and “rootless containers use slirp4netns” is now wrong twice over. Podman made pasta the default rootless network tool in 5.0, and removed slirp4netns entirely in 6.0. Docker went elsewhere: gvisor-tap-vsock became the default rootless network driver in v29.5.0, and slirp4netns is no longer shipped in Docker’s packaging at all.

Binding a port below 1024 rootless needs one of two things, neither of them a container setting: setcap cap_net_bind_service=ep on the rootlesskit binary, or lowering net.ipv4.ip_unprivileged_port_start. Publishing 8080 and putting a reverse proxy in front is usually the better answer — reverse proxy covers it.

What still does not work rootless, from Podman’s own list: no resource limits at all on cgroup v1 systems; no source-IP preservation through the default port forwarder; container-to-container connections needing explicit configuration under pasta; images not easily shared between users; no NFS or parallel-filesystem home directories, and no home mounted noexec or nodev; only the overlay and VFS storage drivers; reduced checkpoint/restore; and creating device nodes failing even in a privileged container.

The cgroup delegation dance is a case where the documentation is behind reality. runc’s docs still tell you to add Delegate=cpu cpuset io memory pids under /etc/systemd/system/user@.service.d/ so rootless containers can set limits. On current systemd the cpu controller is delegated to user@.service by default — kind’s rootless documentation puts that at systemd 252, and every current distribution is well past it, with Ubuntu 26.04 shipping 259. I could not find the upstream systemd release note that made the change, so treat the version number as reported rather than verified; cpuset may still need the drop-in.

# What your user session actually has delegated
cat /sys/fs/cgroup/user.slice/user-$(id -u).slice/user@$(id -u).service/cgroup.controllers

# If cpu is missing, this is the drop-in
# /etc/systemd/system/user@.service.d/delegate.conf
# [Service]
# Delegate=cpu cpuset io memory pids

# What your user session actually has delegated
cat /sys/fs/cgroup/user.slice/user-$(id -u).slice/user@$(id -u).service/cgroup.controllers

# If cpu is missing, this is the drop-in
# /etc/systemd/system/user@.service.d/delegate.conf
# [Service]
# Delegate=cpu cpuset io memory pids

A worked diagnosis

A service that had run in production for eighteen months started failing after its host was rebuilt on a newer OS with a newer Docker. The image was byte-identical — same digest, pinned in the deployment. The application logged one line and exited:

fatal: failed to initialise async I/O backend: Operation not permitted

fatal: failed to initialise async I/O backend: Operation not permitted

“Operation not permitted” with an unchanged image and a changed host reads like a permissions problem, and the first instinct was to compare file ownership in the volume. That was the wrong half of the chain, and the hinge said so immediately:

$ docker inspect -f '{{.State.Pid}} {{.State.ExitCode}}' svc
0 1

$ docker inspect -f '{{.State.Pid}} {{.State.ExitCode}}' svc
0 1

The PID was 0 only because the container had already exited — but it had exited with the application’s own status code 1, not a signal, which means the program ran. Stages 1 to 9 all completed. The image, the layers, the overlay mount, the bundle, the namespaces, the cgroup and the execve were all fine, and the file ownership question was irrelevant. Something inside the sandbox refused a syscall to a program that was otherwise running normally.

Inside the sandbox there are only three candidates: capabilities, seccomp, and an LSM profile. They can be separated in three runs:

docker run --rm --cap-add=ALL svc:pinned                        # still fails
docker run --rm --security-opt apparmor=unconfined svc:pinned   # still fails
docker run --rm --security-opt seccomp=unconfined svc:pinned    # works

docker run --rm --cap-add=ALL svc:pinned                        # still fails
docker run --rm --security-opt apparmor=unconfined svc:pinned   # still fails
docker run --rm --security-opt seccomp=unconfined svc:pinned    # works

Seccomp, then — and specifically the default profile, since nobody had written a custom one. The application’s async I/O backend was io_uring, and Docker added io_uring_setup, io_uring_enter and io_uring_register to the default blocklist in 25.0.0, over container-escape concerns. The old host ran Docker 24. Nothing about the image had changed; the sandbox around it had.

The fix was a minimal custom profile derived from the default with those three syscalls permitted, applied to that one service, with a note recording why. seccomp=unconfined was the diagnostic, not the remedy — it disables the profile entirely, which is a considerably larger change than the one that was needed.

Three things generalise from this. The hinge did its work in one command by proving the failure was inside the sandbox and not in the image or the storage. Bisecting a sandbox is three runs, not a research project, because there are only three mechanisms. And an identical image is not an identical environment: the container is the image plus the host’s runtime policy, and upgrading a host changes the second half silently. Docker 29’s blocking of AF_ALG and socketcall(2) — which can break 32-bit programs — is the same class of change waiting to happen again.

Where things go wrong, by stage

Stage numbers below are the nine stages from the box at the top of this page, which are also the section headings.

What to take from this

Ask for the PID first. It splits nine stages into two halves and tells you which half to stop thinking about. Almost every wasted hour with containers is spent debugging the wrong half.

The mechanisms are independent, and so are their failures. Namespaces hide, cgroups limit, capabilities and seccomp restrict, and none of them knows the others exist. A container with perfect isolation and no limits will take down your host. A container with strict limits and --privileged is a root shell.

The image is only half the environment. The other half is the host’s runtime policy — seccomp profile, default capabilities, cgroup version, storage driver — and it changes underneath a pinned digest every time you upgrade. That is not a flaw in the model; it is what “the container is a process on your host” means.

And the thing worth remembering when something makes no sense at all: there is no container. There is a process. Everything you can do to a container, you can read off that process with ls -l /proc/PID/ns/, cat /proc/PID/cgroup and grep Cap /proc/PID/status, and everything a runtime did to create it, you can do by hand.

Related reading

• What ships in your container image — this page read backwards: what the build put in an image, who can move its name, and when its digest stops resolving.
• Building a container image — the page before both of those: what the builder does with your Dockerfile, and why a green build log says nothing about the image.
• Containers explained — what they are for, and when not to use one
• Docker basics — the commands, in order, with a working example
• Containerising a service — adopting a published image for an existing service, UID first, with a cutover you can undo
• The life of a packet — what happens after a packet leaves the network namespace
• Permissions and privilege, properly — capabilities, user namespaces and why the docker group is root
• The life of a write — what happens below the overlay mount
• From power-on to a login prompt — pivot_root, initramfs and the other root switch
• The shell, in depth — why exec-form CMD is not a style preference
• Signals — the PID 1 rule, and why docker stop waits
• Docker Compose — now at v5, and no longer building images itself
• Container host distributions — where composefs and image-based hosts are going