Your application calls write() on a socket. Some milliseconds later, bytes arrive at another machine. In between, the kernel makes about a dozen decisions, any one of which can be the reason your connection is slow, silently dropped, or arriving from an address you did not expect.
Most networking advice treats that middle as a single opaque step. This walks through it properly, in order, with the command that inspects each stage — because once you can name the layer a problem lives in, diagnosis stops being guesswork.
If you have not read Networking explained, start there. This assumes addresses, routes and ports are already familiar and goes underneath them.
1. The socket, and what it actually is
A socket is a file descriptor, in the same numbering as an open file, because the kernel deliberately made networking look like file I/O. Behind the descriptor is a kernel structure holding the connection’s state, and — the part that matters for performance — two buffers: a send buffer and a receive buffer.
write() does not send anything. It copies your bytes into the send buffer and returns. If the buffer is full, a blocking socket waits and a non-blocking one returns EAGAIN — which is the whole basis of asynchronous network programming. The actual transmission happens later, on the kernel’s schedule, according to TCP’s rules about what the far end has acknowledged and how much it has said it can accept.
This is why a successful write() tells you nothing about delivery. The bytes are in a queue on your own machine.
# what a process has open, sockets included
sudo lsof -p 1234 -a -i
# every socket, with the queues that matter
ss -tan
# Recv-Q Send-Q Local Address:Port Peer Address:Port
# per-socket memory and timers
ss -tim state establishedThe two queue columns in ss are a diagnostic most people never use. A persistently large Send-Q means the far end is not acknowledging fast enough — a slow network or a slow receiver. A persistently large Recv-Q means data has arrived and your own application is not reading it. That single distinction separates “the network is slow” from “my application is blocked”, and it takes one command.
On a listening socket the columns mean something different again: Recv-Q is the number of connections waiting to be accepted and Send-Q is the backlog limit. A Recv-Q pinned at the backlog value on a listener means connections are being dropped before your application ever sees them.
One limit on everything in this stage, and it is a large one: none of it holds for Unix sockets. A local service reached through /run/something.sock rather than through a port is a different address family, and three of the statements above are false for it. Its queue columns are fabricated zeros, with exit status 0, wherever the unix_diag module is unavailable — which includes most containers. Its flow control is the sender’s SO_SNDBUF, so tuning the receiver’s buffer does nothing at all. And a full listen backlog blocks a connection rather than refusing it, so a stuck accept loop presents as clients hanging rather than clients being turned away. The Life of a Unix Socket follows that family through the same kind of spine, and it is worth reading before you apply anything on this page to a socket in the filesystem.
2. Choosing a route, and therefore a source address
Before anything can be sent, the kernel must decide which interface to send it from and which next hop to send it to. It consults the routing table, most-specific prefix first.
The command worth knowing is not ip route — which shows you the table and leaves you to simulate the lookup in your head — but ip route get, which performs the actual lookup and tells you the answer:
ip route get 93.184.216.34
# 93.184.216.34 via 10.0.0.1 dev eth0 src 10.0.0.42 uid 1000
ip route get 10.0.0.99 # a local network destination
ip route get 93.184.216.34 from 192.168.1.5 # simulate a different sourceThree things come out of that one line: the next hop (via), the interface (dev), and the source address (src). The third is the one that surprises people. Unless the application explicitly bound to an address, the kernel picks the source address as a consequence of the route — so on a machine with several addresses, the address your traffic appears to come from is decided by the routing table, not by anything in your application’s configuration.
That is why a server with a second interface sometimes has its connections rejected by a firewall elsewhere that only knows about the first address, and why ip route get is the first command to run when that happens.
There is also more than one routing table. The kernel consults a policy database first, which decides which table to use — and rules there can select a table based on the source address, a firewall mark, or the interface:
ip rule show # the policy database
ip route show table main
ip route show table all | head -30If ip route shows a route that plainly should work and traffic still goes somewhere else, look at ip rule. Something — often a VPN client, sometimes a container runtime — has inserted a rule that sends your traffic to a different table entirely.
3. netfilter: the five places the kernel lets you interfere
netfilter is a set of hook points in the network stack where code can inspect, alter or drop a packet. Everything you know as a firewall — iptables, nftables, ufw, firewalld, Docker’s rules — is a way of putting rules at those hooks.
There are five, and knowing which one a packet passes through explains a great deal:
| Hook | When | Sees |
|---|---|---|
PREROUTING | Immediately on arrival, before the routing decision | Everything inbound, including traffic that will be forwarded |
INPUT | After routing decided the packet is for this machine | Traffic destined for local sockets |
FORWARD | After routing decided it is for somewhere else | Routed traffic — containers, VPNs, gateways |
OUTPUT | Locally generated traffic, after the route lookup | What your own applications send |
POSTROUTING | Last stop before the interface | Everything outbound, local or forwarded |
The single most useful consequence: a packet being forwarded never passes through INPUT. It goes PREROUTING → FORWARD → POSTROUTING. So a rule in the INPUT chain has no effect whatsoever on traffic to a container or through a VPN, which is why so many carefully written firewall rules do nothing at all.
Source NAT — rewriting the source address so replies come back to you — happens in POSTROUTING, after the routing decision, which is why a masquerade rule cannot influence which interface was chosen. Destination NAT happens in PREROUTING, before routing, which is exactly why it can: rewriting the destination early means the routing decision uses the new address.
sudo nft list ruleset # the modern view of everything
sudo iptables-save # the legacy view, still what many tools write
sudo iptables -L -n -v --line-numbers # with packet counters, which is the pointRead the counters. A rule you believe is matching, with a packet count of zero, is not matching — and that is a faster answer than any amount of reasoning about the rule’s text.
4. Connection tracking, and the table that fills up
A firewall rule saying “allow replies to connections we started” requires the kernel to remember which connections were started. That memory is conntrack, and it is a fixed-size hash table of connection state.
Conntrack is what makes ESTABLISHED,RELATED rules possible, and it is what makes NAT possible at all — to translate a reply back, the kernel must remember what it translated on the way out.
When the conntrack table fills, the kernel drops packets — and the only evidence is one line in the kernel log. The message is nf_conntrack: table full, dropping packet. What you experience is a machine that works fine for most traffic and refuses a fraction of new connections at random, with nothing wrong in the application logs, nothing wrong with the routing, and a firewall whose rules are all correct. It is one of the most confusing failures on this page precisely because every layer you would normally check is healthy. Busy proxies, NAT gateways and container hosts are the usual victims, and a machine handling a flood of short-lived connections can exhaust the table long before it runs out of CPU or memory.
# how full is it, right now
cat /proc/sys/net/netfilter/nf_conntrack_count
cat /proc/sys/net/netfilter/nf_conntrack_max
# what is in it
sudo conntrack -L | head
sudo conntrack -S # per-CPU stats, including insert_failed and drop
# the evidence, after the fact
sudo dmesg | grep -i conntrackTwo things worth internalising. First, UDP gets conntrack entries too, despite having no connections — the kernel invents a notion of a flow and times it out, and a busy DNS resolver can fill the table with them. Second, entries for closed TCP connections linger in TIME_WAIT for a while by design, so the count is always higher than the number of connections you think you have.
Raising nf_conntrack_max is the usual fix and costs memory. The better fix, where you can apply it, is to stop tracking traffic that does not need it — a NOTRACK rule for high-volume flows you are not filtering statefully.
5. Finding the neighbour
The packet now has a source address, a destination address and a decided next hop. To put it on an Ethernet segment it needs one more thing: a destination MAC address.
And here is the detail that reframes how people think about routing: the destination MAC is the next hop’s, not the destination’s. When you send to a server on the other side of the world, the Ethernet frame leaving your machine is addressed to your router, sitting a few metres away. Each hop rewrites the MAC addresses and leaves the IP addresses alone. Layer 2 is local; layer 3 is end to end.
Finding that MAC is ARP on IPv4 and Neighbour Discovery on IPv6, and both populate the same table:
ip neigh show
# 10.0.0.1 dev eth0 lladdr 00:1a:2b:3c:4d:5e REACHABLE
# 10.0.0.77 dev eth0 FAILED
ip neigh flush all # force re-resolution
ip -s neigh show # with statisticsThe states are worth reading. REACHABLE is confirmed recently. STALE is known but unconfirmed, and is normal. FAILED means resolution was attempted and nothing answered — which on a local network means the address does not exist, is powered off, or is separated from you by something that is not forwarding ARP.
A FAILED entry for your own gateway is a complete explanation for “the network is down” on a machine whose configuration is perfect.
6. The queue, and why tcpdump lies to you
The frame is ready. It does not go straight to the wire; it goes into a queueing discipline — a qdisc — attached to the interface, which decides the order packets leave and whether any are dropped when the queue is full.
tc qdisc show dev eth0
tc -s qdisc show dev eth0 # with drops and backlog
# per-interface error and drop counters
ip -s link show dev eth0Non-zero dropped or overlimits in the qdisc statistics means the machine is producing traffic faster than the interface can send it. Non-zero errors on the link statistics is a different and more physical problem — cabling, duplex mismatch, a failing NIC.
Then the packet is handed to the driver, which puts a descriptor in a ring buffer, and the card transmits. Except that increasingly, it does not transmit what you think.
tcpdump shows you packets that never existed on the wire. Modern network cards do segmentation themselves: the kernel hands down one 64 KB buffer and the card chops it into MTU-sized frames — TCP segmentation offload. On the way in, the reverse happens: the card reassembles many small frames into one large buffer before the kernel sees it, which is generic receive offload. tcpdump taps the stack above both, so it faithfully reports a 64 KB “packet” that was actually forty-odd frames. This matters when you are debugging MTU problems, counting packets, or comparing a local capture against one taken on a switch, because the two will not agree and neither is wrong. Turn the offloads off with ethtool -K while capturing if the packet boundaries are what you are investigating — and turn them back on afterwards, because they are worth a great deal of CPU.
ethtool -k eth0 | grep -E 'segmentation|offload'
sudo ethtool -K eth0 tso off gso off gro off # while debugging only
sudo ethtool -K eth0 tso on gso on gro on # put it back7. MTU, and the failure that looks like nothing
Every link has a maximum frame size. If a packet is too large for the next link and is marked “do not fragment” — which almost all TCP traffic is — the router is supposed to drop it and send back an ICMP message saying so, and the sender then uses smaller packets. That mechanism is path MTU discovery.
It breaks when something between you and the destination blocks ICMP, which a great many badly-configured firewalls do on the theory that ICMP is dangerous. The result is a PMTU black hole, and its signature is unmistakable once you have seen it: small requests work perfectly, the connection establishes fine, and then anything large hangs forever. SSH connects and then freezes when you run a command with long output. A web page returns headers and never finishes.
# find the real path MTU: increase until it fails
ping -M do -s 1472 -c 2 example.com # 1472 + 28 = 1500
ping -M do -s 1420 -c 2 example.com
ip link show eth0 | grep mtu
ip route get 93.184.216.34 # a cached PMTU appears hereThis is common on tunnels, because encapsulation adds overhead to every packet — a WireGuard or VPN interface has a smaller MTU than the physical one underneath it, and a tunnel configured without accounting for that works for pings and fails for real traffic.
8. The return journey
Receiving is not simply the same steps backwards, and the difference explains a common performance mystery.
A frame arrives; the card writes it into a receive ring buffer and raises an interrupt. Under load, taking an interrupt per packet would be ruinous, so the kernel switches to NAPI: it disables interrupts for that queue and polls instead, draining many packets per poll. That polling happens in a software interrupt context — softirq.
Which is why, on a machine under heavy network load, top shows a large si figure and you cannot find a process responsible for it. That time is not attributable to any process, because it is the kernel processing packets on behalf of everyone.
top # look at the si column in the CPU line
cat /proc/softirqs | head -5 # NET_RX and NET_TX, per CPU
cat /proc/net/softnet_stat # column 2 non-zero = dropped for backlog
ethtool -S eth0 | grep -i drop # what the card itself dropped
ethtool -g eth0 # ring buffer sizes, current and maximumIf softirq load is concentrated on one CPU while the others idle, the card is delivering everything to a single queue. Receive-side scaling spreads it across queues and CPUs; if the hardware does not support it, receive packet steering does the same thing in software.
Then the packet goes up through PREROUTING, the routing decision that determines it is for this machine, INPUT, and finally into the receive buffer of the matching socket — where it waits for your application to call read(). If it never does, that is the growing Recv-Q from section one, and the circle closes.
9. Where containers change everything
A container has its own network namespace: its own interfaces, routing table, neighbour table and netfilter rules. It is a complete, separate copy of everything above.
The connection to the outside is a veth pair — two virtual interfaces joined like a pipe, one end inside the namespace and one end on the host, usually attached to a bridge. A packet leaving a container therefore traverses the stack described above twice: once inside the namespace, and again on the host after emerging from the veth.
That doubling is the source of most container networking confusion. It means:
- There are two routing tables in play, and
ip routeon the host tells you nothing about the container’s. - Container traffic is forwarded on the host, so it passes FORWARD and never INPUT — the point from section three, now with consequences.
- It is NAT’d on the way out, consuming conntrack entries on the host, which is why container hosts are the classic conntrack exhaustion victims.
- A published port is a DNAT rule in PREROUTING, inserted by the container runtime directly into netfilter — which is precisely why it bypasses ufw and firewalld, as the firewall guide warns. Those tools manage their own chains; Docker writes its own.
To debug inside a container’s namespace, enter it rather than guessing from outside:
ip netns list # namespaces created the manual way
sudo nsenter -t $(docker inspect -f '{{.State.Pid}}' mycontainer) -n ip addr
sudo nsenter -t <pid> -n ip route
sudo nsenter -t <pid> -n ss -tulpn
ip -d link show type veth # the host ends, and what they are paired with
bridge link show # what is attached to which bridgensenter -n runs a command in another process’s network namespace, using the host’s binaries — so you get ss and ip inside a container image that contains neither. It is the single most useful container networking command and it is not a Docker command at all.
A worked diagnosis
The symptom: an application in a container can reach some external services and not others. It is intermittent. Restarting it helps for a while.
Working down the layers rather than guessing:
# 1. Is it name resolution or connectivity? Test with an address, not a name.
sudo nsenter -t <pid> -n getent hosts api.example.com
sudo nsenter -t <pid> -n ping -c1 93.184.216.34
# 2. Does the container have a route, and which source address will it use?
sudo nsenter -t <pid> -n ip route get 93.184.216.34
# 3. Is the host willing to forward it, and are the counters moving?
sysctl net.ipv4.ip_forward
sudo iptables -L FORWARD -n -v
# 4. Is the return path being tracked?
cat /proc/sys/net/netfilter/nf_conntrack_count
cat /proc/sys/net/netfilter/nf_conntrack_max
sudo conntrack -S | grep -E 'insert_failed|drop'
# 5. Is it a size problem rather than a reachability problem?
sudo nsenter -t <pid> -n ping -M do -s 1400 -c2 93.184.216.34In this case step 4 answers it: the count is at the maximum, insert_failed is climbing, and dmesg has the table-full line. “Some services work and some do not, intermittently” is exactly what a full conntrack table produces — new flows fail while established ones continue, and a restart helps because the old entries eventually time out.
Notice that the application logs, the DNS configuration and the firewall rules were all correct throughout, and no amount of examining them would have found it. That is the argument for knowing the layers.
Symptom, and the layer it lives in
| Symptom | Layer | Command |
|---|---|---|
| Recv-Q growing on an established socket | Your application is not reading | ss -tan |
| Send-Q growing | The far end or the path, not you | ss -tan |
| Recv-Q pinned on a listening socket | Accept backlog full; connections dropped | ss -tanl |
| Traffic leaves from the wrong address | Route selection chose the source | ip route get |
| Route looks right, traffic goes elsewhere | Policy routing rule | ip rule show |
| Firewall rule has no effect on container traffic | It is forwarded, not INPUT | iptables -L FORWARD -n -v |
| Random new connections fail, existing ones fine | conntrack table full | conntrack -S, dmesg |
| Whole local network unreachable | Neighbour resolution failed | ip neigh show |
| Small requests fine, large ones hang | PMTU black hole | ping -M do -s |
| Capture shows huge packets that cannot exist | Offloads | ethtool -k |
High si, no process responsible | softirq packet processing | /proc/softirqs |
| Drops with CPU to spare | Single receive queue, or ring buffer too small | ethtool -S, -g |
| Interface errors climbing | Physical — cable, duplex, NIC | ip -s link |
| Published container port ignores the firewall | Runtime DNAT in PREROUTING | iptables -t nat -L -n |
The shape to remember
Outbound: socket buffer → route lookup (which sets the source address) → OUTPUT → POSTROUTING and any NAT → neighbour resolution → qdisc → driver and offloads → wire.
Inbound: wire → ring buffer → NAPI poll in softirq → PREROUTING and any NAT → route lookup → INPUT or FORWARD → socket receive buffer → your application’s read().
Nearly every network problem you will meet is one of those steps failing, and nearly every one has a command that shows you its state directly. The skill is not memorising the commands — it is having a list of layers to walk down, so that “the network is broken” becomes a question with a next step rather than a shrug.
Related reading
- Networking explained — addresses, routes and ports, if this went too fast
- How a name is actually resolved — the step before any of this begins
- Containers explained — namespaces, from the other direction
- ip — the command that inspects most of the layers above
- Linux firewalls — netfilter from the practical end
- lsof and fuser — finding which process owns a socket