Almost everything written about Linux permissions stops at chmod 644. That is one layer of six, and it is the only one that has not changed in thirty years. The other five are where your actual problems live: the file you own and cannot read, the capability that survives a sudo and the one that does not, the SELinux denial that produces exactly the same EACCES as a wrong mode bit, and the container whose files belong to user 100000 on the host and to root inside.
This page follows a single question — may this process touch this file? — all the way down through every gate the kernel puts in its path, in the order the kernel checks them. Each layer gets the command that inspects it. At the end there is a worked diagnosis of a failure that looks like a permissions problem, is a permissions problem, and is not on the layer anyone looks at first.
If you have not read chmod, chown and umask yet, start there — it covers the notation this page assumes. This is the level below it.
The order, once, so the rest of the page makes sense. A single open() passes through these in sequence, and every one of them must agree:
- Credentials — which UID, GID and supplementary groups the kernel has attached to this thread.
- Path traversal — execute permission on every single directory component, checked one at a time.
- Discretionary access control — the nine mode bits, or the ACL if the file has one.
- Capability override — if DAC said no, does this thread hold
CAP_DAC_OVERRIDEorCAP_DAC_READ_SEARCH? - The security module — SELinux or AppArmor gets a veto on everything above.
- Namespace translation — which user namespace those IDs were relative to, and what they map to here.
Layers three and five both return EACCES. That single fact is responsible for more wasted afternoons than any other thing in this article.
1. Who the kernel thinks you are
A process does not have a user ID. It has several, and they can differ from each other at the same instant. id shows you the friendly version; /proc shows you the truth.
grep -E '^(Uid|Gid|Groups|CapEff):' /proc/self/status
Uid: 1000 1000 1000 1000
Gid: 1000 1000 1000 1000
Groups: 4 24 27 1000
CapEff: 0000000000000000The four numbers on the Uid line are real, effective, saved and filesystem UID. The effective UID is the one permission checks use. The real UID is who you actually are, and is what a process uses to decide whether you may signal it. The saved UID is a parking space: a program that started privileged and dropped down keeps its old identity there so it can pick it back up. The filesystem UID is a historical oddity that NFS needed; on a modern system it tracks the effective UID and you can ignore it.
The Groups line matters more than people expect. Supplementary groups are resolved once, at login, and copied into the process. Adding yourself to docker and then wondering why docker ps still fails is not a bug — your shell is still carrying the group list it was handed when it started. Nothing short of a new login session (or newgrp, or exec su - $USER) will refresh it. id reads the group database and will happily tell you that you are in the group while the running process is not.
To see the difference for yourself, compare what the database says with what the process holds:
# what the group database says about you
id -nG
# what this actual shell is carrying
tr ' ' '\n' < /proc/self/status | grep -A1 Groups2. Every directory on the way there
Before the kernel looks at your file it has to find it, and finding it means walking the path one component at a time. Each directory in that walk needs the execute bit for you — on a directory, x does not mean “run” but “pass through”. A directory you can pass through but not read is perfectly legal and quite useful: you can open /srv/data/secret/config.yml if you know the name, but ls /srv/data/secret is denied.
On a directory, the three bits mean something else entirely. This is the single most common misreading in Linux permissions.
r— you may list the names in it. Nothing else. Names only, not what they point at.x— you may use it: traverse into it, andstat()things inside it by name.w— you may create and delete entries in it. Deleting a file is a write to its directory, not to the file. This is why you can delete a read-only file you do not own, and why you cannot delete a writable file in a directory you cannot write.
r without x is the useless combination: you can see the names and do nothing with any of them, so even tab-completion misleads you. x without r is the useful one.
When a permission error makes no sense, check the whole path rather than the file. namei does it in one shot:
namei -l /srv/app/data/config.yml
f: /srv/app/data/config.yml
drwxr-xr-x root root /
drwxr-xr-x root root srv
drwxr-x--- root appgrp app
drwx------ deploy deploy data
-rw-r--r-- deploy deploy config.ymlThe file is world-readable. It is also unreachable by anyone except deploy, because data is 0700. Reading the file’s own mode would have told you nothing.
3. The nine bits, and the rule nobody is told
Now the file itself. The kernel picks exactly one of the three permission triads and applies it. It does not combine them, and it does not fall through:
- If your effective UID matches the file’s owner — use the owner bits and stop.
- Otherwise, if the file has an ACL, consult it (section 5).
- Otherwise, if the file’s group is in your group list — use the group bits and stop.
- Otherwise, use the other bits.
“And stop” is the part that surprises people. A file with mode 0477 owned by you is a file you cannot write and everyone else can. You match on rule one, the owner triad says read-only, and the generous group and other bits are never even looked at. It is not a special case or a bug — it is the algorithm working as designed, and it is occasionally deliberate: 0077 on your own file is a way of saying “not by accident, not without a chmod first”.
The other thing worth internalising: permission is checked at open(), once. A process that has a file descriptor open for writing keeps writing after you chmod 000 the file, after you chown it away, and after you delete it. Revoking access to something already open requires killing the process or removing the storage; there is no revoke() on Linux. This is why df and du disagree after you delete a big log file that rsyslog still has open, and it is why changing a secret’s permissions does nothing to a service that read it at startup.
4. The three bits above the nine
Above the nine sit three more, shown by stat -c '%a %A %n' as a leading digit. They do very different things depending on what they are attached to.
| Bit | On an executable | On a directory |
|---|---|---|
| setuid (4000) | Runs with the file owner’s effective UID, not yours. | Nothing on Linux. Ignored. |
| setgid (2000) | Runs with the file’s group as effective GID. | New files inherit the directory’s group instead of yours. New subdirectories inherit the setgid bit too. |
| sticky (1000) | Nothing on modern Linux. | Only the file’s owner (or the directory’s owner, or root) may delete an entry, regardless of write permission. |
Setgid on a directory is the correct answer to almost every “shared folder” problem. Combined with a default ACL (next section) it is the complete answer. Sticky is why /tmp is mode 1777 — everyone can create files there, nobody can delete anyone else’s.
Setuid on an executable is the oldest privilege-escalation mechanism on the system and, thirty years on, still the leakiest: the program inherits your environment, your open file descriptors, your resource limits and your current directory, and it has to defend against all of them. Auditing what is setuid on a machine takes one command, and the list should be short and boring:
find / -xdev \( -perm -4000 -o -perm -2000 \) -type f -printf '%M %u %g %p\n' 2>/dev/nullAnything on that list which is not sudo, su, passwd, mount/umount, pkexec or one of the chsh/chfn/newgrp family deserves an explanation. The list is shorter than it used to be, and section 7 explains what happened to the rest of it.
5. ACLs, and the mask that quietly eats them
Nine bits express exactly one user and one group. The moment you need “the web server may read this, the backup user may read this, the deploy user may write it, and nobody else exists”, you are out of room. POSIX ACLs are the extension, they have been standard on ext4, XFS and Btrfs for years, and on ext4 they are enabled by default when the kernel is built with ACL support — which every distribution kernel is. You do not need to remount anything.
# grant, then inspect
setfacl -m u:www-data:r /srv/app/config.yml
setfacl -m u:deploy:rw /srv/app/config.yml
getfacl /srv/app/config.yml
# file: srv/app/config.yml
# owner: root
# group: root
user::rw-
user:www-data:r--
user:deploy:rw-
group::r--
mask::rw-
other::---An ls -l on that file now shows a trailing + on the mode string, and shows rw-rw---- — which is a lie, or at least a projection. The middle triad is no longer the group’s permissions. When an ACL contains a mask entry, the group bits in ls -l are the mask, not group::. This is the source of nearly every ACL surprise.
The mask is a ceiling. It caps every named user entry, every named group entry and the owning group — everything except user:: and other::. If an entry grants more than the mask allows, getfacl annotates it, and the annotation is the thing to look for:
chmod g-w /srv/app/config.yml
getfacl /srv/app/config.yml
user::rw-
user:www-data:r--
user:deploy:rw- #effective:r--
group::r--
mask::r--
other::---Nothing about the deploy entry changed. A routine chmod g-w — run by a deployment script, a config-management tool, or a person tidying up — lowered the mask, and deploy silently lost write access it still appears to have. That #effective: comment is the only warning you get. When an ACL “stops working” and the entry is visibly still there, this is what happened, essentially every time.
The genuinely powerful part is default ACLs on directories, which are not access rules at all — they are a template stamped onto everything created inside. Combined with setgid, they solve the shared-directory problem permanently, including for files created by processes with an unhelpful umask:
chgrp appgrp /srv/shared
chmod 2770 /srv/shared
setfacl -d -m g:appgrp:rwx /srv/shared
setfacl -d -m u:backup:rx /srv/shared
# and to fix what is already in there
setfacl -R -m g:appgrp:rwX -m u:backup:rX /srv/sharedNote the capital X in the recursive form. It means “execute, but only on directories and on files that already have an execute bit somewhere” — the difference between fixing a tree and making every text file in it executable.
ACLs live in extended attributes, in the system. namespace: system.posix_acl_access and system.posix_acl_default. Two consequences follow. First, they are dropped by any copy that is not told to carry them — rsync without -A, tar without --acls, and cp unless you pass -a or -p. Note which cp options those are: --preserve=xattr is not one that works, because in coreutils an ACL travels with the mode rather than with the extended attributes, so the flag whose name says “xattr” is the wrong one to reach for. A restored backup that “lost its permissions” usually lost its ACLs. Second, a filesystem that does not support extended attributes — most FAT and exFAT volumes, many network mounts — cannot hold them at all, which is why permissions evaporate on a USB stick.
6. Root, taken apart
Everything so far can be overridden by one thing, and that thing is not “being root”. UID 0 is a convenience; what actually bypasses the checks is a set of capabilities, and the kernel has split the powers of root into 41 of them, numbered 0 to 40. The highest one defined is CAP_CHECKPOINT_RESTORE (Linux 5.9); CAP_BPF and CAP_PERFMON arrived just before it in 5.8, carved out of the enormous CAP_SYS_ADMIN. Your kernel will tell you its own number:
cat /proc/sys/kernel/cap_last_cap
40
capsh --print | head -3
capsh --decode=0000003fffffffffThe four that matter for this page:
| Capability | What it lets you ignore |
|---|---|
CAP_DAC_OVERRIDE | All read, write and directory-search checks. (Executing a file still needs at least one x bit to exist somewhere.) |
CAP_DAC_READ_SEARCH | Read and directory-search checks only — the read-only half of the above. |
CAP_FOWNER | The requirement to be the owner in order to chmod, chown, set ACLs or set the sticky-bit exemption. |
CAP_SYS_ADMIN | An enormous grab-bag — mounting, namespaces, quotas, dozens of syscalls. Granting it is granting root. |
A thread carries five capability sets, and the distinction between them explains a great many otherwise baffling behaviours:
- Permitted — the ceiling. What this thread may put into its effective set.
- Effective — what is being checked right now. Well-written privileged programs keep this empty and raise a single capability for the two lines that need it.
- Inheritable — survives
execve(), but only for capabilities the new program’s file also marks inheritable. On its own it does almost nothing, which catches everyone out. - Bounding — a hard ceiling on everything above, for this thread and all its descendants. It can only ever be reduced. This is what
systemd‘sCapabilityBoundingSet=sets, and it is irreversible for the lifetime of the process tree. - Ambient (Linux 4.3 and later) — the one that does what people always assumed inheritable did: capabilities that genuinely survive
execve()into an unprivileged binary with no file capabilities at all. This is howsystemd‘sAmbientCapabilities=gives a serviceCAP_NET_BIND_SERVICEwithout it ever being root.
Files carry capabilities too, in the security.capability extended attribute — a permitted set, an inheritable set, and a single effective bit which says “raise everything the new process is permitted, immediately”. This is the modern replacement for setuid, and you can see it in action:
# what capabilities does a binary carry?
getcap /usr/bin/newuidmap /usr/bin/ping 2>/dev/null
# find every one on the system
getcap -r / 2>/dev/null
# what does a running process actually hold?
grep Cap /proc/$(pgrep -n nginx)/statusTwo things are worth knowing about that hex. CapPrm of all zeroes with a non-zero CapBnd means the process is genuinely unprivileged, whatever its UID. And a full CapEff on something that has no business having one is the fastest way to spot a container running with --privileged when nobody remembers adding it.
7. How privilege is actually acquired now
The setuid list on a modern distribution is shorter than it was five years ago, and the replacements are worth knowing because they change what you look at when something breaks.
ping is the clearest example, and it is now on its third mechanism. It was setuid-root, because raw sockets need CAP_NET_RAW. Then it became a plain binary with cap_net_raw+ep in its file capabilities — less dangerous, since it grants one power rather than all of them. Now it is neither: Fedora since version 31, and Debian since 13, ship ping with no elevated privilege whatsoever, using an ICMP datagram socket that ordinary users may open. The gate is a sysctl:
sysctl net.ipv4.ping_group_range
net.ipv4.ping_group_range = 0 2147483647If that range is 1 0 — the restrictive kernel default — no group qualifies and unprivileged ping fails with Operation not permitted on a binary that has nothing wrong with it. Debian sets the permissive range from a package called linux-sysctl-defaults, and its own release notes warn that some upgrade paths do not pull it in. If ping broke after an upgrade and getcap shows nothing, this sysctl is the answer, not chmod u+s.
The same shift shows up in the container tooling. newuidmap and newgidmap — the helpers rootless Podman and Docker rely on — are still required, but on Debian and Arch they are no longer setuid-root. They ship with cap_setuid+ep and cap_setgid+ep, which is precisely the power they need and nothing else. getcap finds them; find -perm -4000 does not. If you audit setuid binaries and never look at file capabilities, you are auditing half the system. They are also easy to lose without noticing, because the flags are asymmetric: tar --acls --xattrs writes security.capability into the archive and then declines to restore it by design, and the option that overrides that belongs on the extract — --xattrs-include='*'. A binary that comes back from a backup without its capability fails at some later restart, with a permission error on a file whose mode bits are visibly correct. Restoring a Linux Server measures it.
There is also a structural alternative to sudo now. run0, added in systemd 256, is a symlink to systemd-run and works nothing like sudo internally: it is not setuid at all. It asks PID 1 to fork a fresh, isolated service with the privileges you asked for, authenticates you through polkit rather than through a program running in your own process with your own environment, and allocates a separate pseudo-terminal so the privileged process is not sharing a tty with your shell. That last detail closes a whole class of terminal-injection attacks. Separately — and it is a different axis entirely — Ubuntu switched its default sudo implementation to the Rust sudo-rs in 25.10. That is a memory-safety change; sudo-rs is still setuid.
For services, the practical version of all this is that you should almost never be dropping privileges yourself. The unit file does it, declaratively, before your code starts — hardening a public-facing service covers the directives in full, but the shape is User=, CapabilityBoundingSet=, AmbientCapabilities= and NoNewPrivileges=yes. That last one sets a flag the kernel will not let the process clear: no execve() from here on can ever gain privilege, whatever setuid bits or file capabilities it encounters. It is one line and it removes an entire attack class.
8. The security module, which can only say no
Everything to this point is discretionary: the owner of a file decides who may use it. Layer five is mandatory: a policy shipped with the system decides, and the file’s owner cannot overrule it. The Linux Security Module framework is the hook, and SELinux or AppArmor is what is plugged into it.
The critical property is the direction. An LSM can only deny. It never grants access that DAC refused. A file must pass the mode bits and the ACL and the policy. If any one says no, you get EACCES — and this is why “the permissions are obviously correct and it still will not open the file” is such a common and such a frustrating state.
What you are running depends on your distribution, and one of these changed recently enough to catch out anyone working from memory:
| Distribution | Default |
|---|---|
| Fedora, RHEL and rebuilds | SELinux, enforcing, targeted policy |
| Ubuntu | AppArmor, with enforcing profiles |
| Debian | AppArmor, installed and enabled by default since Debian 10 |
| openSUSE Leap 16 / SLE 16 | SELinux, enforcing — AppArmor was dropped entirely in this release |
| Arch | None active. AppArmor is built into the stock kernel but inert until you enable it on the kernel command line. |
Modules stack, but only partly. Capabilities are always present. Beyond that the kernel permits any number of “minor” modules — Landlock, Yama, LoadPin, SafeSetID, lockdown, IPE, BPF — alongside at most one major module. SELinux, AppArmor, Smack and TOMOYO remain mutually exclusive; you cannot run SELinux and AppArmor together. What is actually loaded is not a guess:
cat /sys/kernel/security/lsm
capability,landlock,lockdown,yama,apparmor,ipe,bpfDiagnosing a denial is the same idea on both, with entirely different commands. On SELinux, every object has a context and the audit log records the exact refusal:
# the context, which ls -l does not show you
ls -Z /srv/www/index.html
ps -eZ | grep nginx
# what was denied, in English, with a suggested fix
ausearch -m AVC -ts recent | audit2why
# the usual cause: a file in the wrong place with the wrong label
semanage fcontext -a -t httpd_sys_content_t "/srv/www(/.*)?"
restorecon -Rv /srv/wwwOn AppArmor there are no labels on files — the policy is written in terms of paths, per program:
aa-status
journalctl -k | grep -i 'apparmor="DENIED"'
# put one profile into complain mode to see what it would have blocked
aa-complain /etc/apparmor.d/usr.sbin.nginxA note on the standard advice to “just disable SELinux”. It is the wrong instinct, and increasingly an expensive one: on a RHEL-family system a large amount of the security posture you are paying for lives in that policy, and the tooling to fix a label is two commands. Putting it in permissive mode temporarily to confirm a diagnosis is reasonable. Leaving it there is not, and audit2why will usually have told you the answer before you get as far as reaching for the switch.
One newer module deserves a mention because it inverts the model. Landlock (Linux 5.13, and considerably more capable by 7.0) lets an unprivileged process voluntarily restrict itself and its children — no root, no policy file, no administrator involvement. It stacks alongside whichever major module you are running. It is how modern sandboxing is being built, and it is the answer to “I want this build script to be unable to touch anything outside this directory” that does not involve a container.
9. User namespaces, where the numbers stop being absolute
The last layer changes the meaning of every layer above it. A user namespace is a mapping: inside it, UID 0 is a real, fully-capable root — but only over resources that namespace owns. Outside, that same process is UID 100000 with no capabilities at all.
The mapping is per-process and readable:
cat /proc/self/uid_map
0 0 4294967295
# inside a rootless container, the same file reads something like
# 0 100000 1
# 1 165536 65536Three columns: the first ID inside, the first ID outside, and how many. The ranges a user is permitted to claim come from /etc/subuid and /etc/subgid, one line per user, in the form name:start:count. The shadow tooling hands out 65536 IDs to each new user by default, starting at 100000.
grep "^$USER:" /etc/subuid /etc/subgid
/etc/subuid:kevin:100000:65536
/etc/subgid:kevin:100000:65536This is where the single most common container permissions problem comes from. A file written as root inside a rootless container appears on the host owned by UID 100000. A file you own on the host, UID 1000, bind-mounted in, appears inside the container as nobody — because 1000 is not in the container’s map at all. Nothing is broken; you are looking at the same inode through two different translations.
The old fix was to recursively chown the volume, which is slow, destructive and wrong if anything else uses the directory. The modern fix is an idmapped mount — a kernel feature from Linux 5.12, exposed through mount_setattr() — which applies the translation at the mount instead of on disk. Podman exposes it directly:
podman run --rm -it \
--mount type=bind,source=/srv/data,destination=/data,idmap \
alpine ls -ln /dataBecause a user namespace hands out real capabilities, it is also an attack surface — many kernel privilege-escalation bugs of the last decade were reachable only through one. Ubuntu responded with an AppArmor-based restriction: unprivileged processes may still create a user namespace, but may not use capabilities inside it unless a profile permits it. It was introduced opt-in in 23.10 and has been on by default since 24.04 LTS.
sysctl kernel.apparmor_restrict_unprivileged_userns
kernel.apparmor_restrict_unprivileged_userns = 1If a sandboxed application, an AppImage or a build tool fails on Ubuntu with a permission error deep inside unshare() or clone(), this is very often why — and it will not reproduce on Debian, which has not adopted the same restriction. Two caveats worth carrying: installing LXD turns the feature off entirely at runtime, and bypasses have been published, so treat it as a useful speed bump rather than a boundary.
10. A worked diagnosis
Here is the shape of a real one. A service that has run for months stops being able to write its uploads directory after a routine reboot. Nothing was deployed. The error is Permission denied.
The instinct is to look at the directory. It looks fine — drwxrws---+ 3 root appgrp, setgid set, an ACL present. Everyone stops here, concludes the permissions are correct, and starts suspecting the application. Go down the layers instead, in order.
Layer one, credentials. What is the service actually running as, and what does it hold?
systemctl show app.service -p User -p MainPID
id -nG appuser
grep -E 'Uid|Groups|CapEff' /proc/$(systemctl show -p MainPID --value app.service)/statusRunning as appuser, and appuser is in appgrp. The process’s own Groups line confirms it — so this is not the stale-group-membership trap. Move down.
Layer two, the path. namei -l /srv/app/uploads walks clean: every component is traversable by appgrp. Move down.
Layer three, DAC and the ACL. The + means ls -l has been showing us the mask, not the group. So look properly:
getfacl /srv/app/uploads
user::rwx
group::rwx
group:appgrp:rwx #effective:r-x
mask::r-x
other::---
default:group:appgrp:rwxThere it is. The appgrp entry still says rwx and is worth r-x, because the mask is r-x. Something lowered the mask. That something is almost always a chmod on the group triad — and the culprit here is a config-management run that enforces mode: '0750' on that directory. It has been doing so for months; the directory only carried the mask into effect after the reboot recreated it from the default ACL on the parent.
The fix is one command, and it is not chmod:
setfacl -m m::rwx /srv/app/uploadsThe durable fix is to stop the config-management tool asserting a mode on a directory whose access control lives in an ACL. The two mechanisms are fighting, and the one that runs last wins.
Notice what the method bought you. Had the ACL been clean, the next step would have been layer five — ausearch -m AVC -ts recent or journalctl -k | grep DENIED — and had that been clean, layer six, checking whether the path is inside a mount namespace the service cannot see. Each layer has one command, they are cheap, and going through them in order takes about ninety seconds. Guessing takes an afternoon.
Before you type chmod 777. It is the universal solvent, it works about half the time, and both outcomes are bad. If it fixes the problem you have made a directory world-writable and learned nothing. If it does not fix the problem — which happens constantly — you have proved the failure was never on layer three, and you now have a world-writable directory and the original bug.
The five-second version of the right move: run namei -l on the full path, and getfacl on the last component that looks suspicious. Between them they resolve layers two and three, which is where most real failures are. If both are clean, it is an LSM or a namespace, and chmod was never going to help.
11. Symptom, cause, command
| What you see | Layer and likely cause | What to run |
|---|---|---|
| Added to a group, still denied | 1 — the shell has the old group list | id -nG vs /proc/self/status; log out and back in |
File is 644 and readable by nobody | 2 — a parent directory blocks traversal | namei -l /full/path |
| You own the file and cannot write it | 3 — owner triad matched and denied; group bits never consulted | stat -c '%A %U %G' file |
| Cannot delete a file you can write | 2 — deleting needs w on the directory, or sticky bit applies | ls -ld $(dirname file) |
| ACL entry present but not working | 3 — the mask was lowered by a chmod | getfacl file, look for #effective: |
| New files in a shared dir get the wrong group | 3 — no setgid bit on the directory | chmod g+s dir |
| New files in a shared dir get the wrong mode | 3 — no default ACL; the creator’s umask decides | setfacl -d -m g:grp:rwx dir |
| Permissions lost after a restore | 3 — ACLs are xattrs and were not copied | rsync -A, tar --acls, cp -a |
Works as root, fails under sudo -u | 1 or 4 — different supplementary groups, or a capability not inherited | sudo -u user id -nG; getpcaps |
| Service cannot bind port 80 as non-root | 4 — no CAP_NET_BIND_SERVICE | AmbientCapabilities=CAP_NET_BIND_SERVICE in the unit |
| Binary works from a shell, fails from a unit | 4 — CapabilityBoundingSet= or NoNewPrivileges= stripped it | systemctl show unit -p CapabilityBoundingSet |
ping fails as a normal user | 4 — ping_group_range is restrictive; no caps on the binary | sysctl net.ipv4.ping_group_range; getcap $(which ping) |
| Setuid audit looks clean but privilege is escalating | 4 — file capabilities, which find -perm -4000 misses | getcap -r / 2>/dev/null |
Modes are correct and it still returns EACCES | 5 — SELinux or AppArmor denial | ausearch -m AVC -ts recent; journalctl -k | grep DENIED |
Works in /var/www, fails in /srv/www | 5 — SELinux file label follows the path | ls -Z; semanage fcontext then restorecon -Rv |
| Container files owned by 100000 on the host | 6 — rootless UID mapping, working correctly | cat /proc/self/uid_map; grep $USER /etc/subuid |
Bind-mounted files show as nobody in the container | 6 — host UID is outside the container’s map | bind-mount with ,idmap |
| Sandbox or AppImage fails only on Ubuntu | 6 — restricted unprivileged user namespaces | sysctl kernel.apparmor_restrict_unprivileged_userns |
| Path exists for you, not for the service | 6 — mount namespace; ProtectSystem= or PrivateTmp= | systemctl show unit -p ProtectSystem -p PrivateTmp |
| Rootless podman fails to start at all | 6 — no subuid range allocated for the user | grep $USER /etc/subuid; usermod --add-subuids |
12. The shape of the whole thing
Six layers, and the useful summary is what each one is for.
Mode bits and ACLs are discretionary — the owner decides, and the owner can be wrong. They are the layer you edit. Capabilities exist to make “needs privilege” mean something narrower than “needs to be root”, and the direction of travel across every distribution is to replace setuid binaries with them, or with a sysctl, or with a service that PID 1 forks on your behalf. The security module is mandatory, cannot grant anything, and exists precisely because the owner of a file can be wrong. User namespaces make the identities themselves relative, which is what allows an unprivileged user to run something that believes it is root without that belief being dangerous.
Nearly every hard permissions problem is a layer confusion: a fix applied at layer three for a failure occurring at layer five, or a container mapping mistaken for corruption. The diagnostic discipline is unglamorous and it works — go down the list, one command per layer, and let the layer that actually refuses tell you what it wants.
Next, if this was useful: what actually happens when you run a program covers the execve() half of the capability rules from the other side, and the life of a packet does the same thing this page does, for the network stack.