Linux Networking from First Principles

“The network is broken” is one of the least useful sentences in system administration, because six different things can produce it and they need six different fixes. Getting from that sentence to an actual cause takes about a minute if you know what the layers are and which order to test them in.

This is the conceptual half. Networking basics covers the commands; this page covers what they are telling you.

Nearly every network problem is one of those four, and the symptom column tells you which. That table alone does most of the work.

An IP address identifies an interface on a network. It always comes with a prefix — the /24 in 192.168.1.10/24 — and the prefix is what splits the address into “which network” and “which machine on it”.

The number is how many bits belong to the network. /24 means the first three of the four numbers identify the network, so 192.168.1.x is the network and x is the machine.

Why this matters practically: two machines can talk directly only if they are on the same network by this calculation. If they are not, every packet goes to the gateway instead — and a mismatched prefix is a classic cause of “I can reach some machines on this LAN and not others”.

Certain ranges are reserved for private use and never routed on the internet:

A 169.254.x.x address is worth recognising, but read it carefully. It is assigned by a client that asked for an address, got no answer, and fell back to link-local: dhcpcd does this, and so do Windows and macOS. systemd-networkd and NetworkManager, which configure most Linux servers and desktops, do not: when DHCP fails they leave the interface UP with no IPv4 address, and the failure is silent in ip -br addr. Either way the problem is DHCP or the physical link, not anything further up. The Life of a Network Interface has the measurements.

# What am I?
ip addr show
ip -brief addr        # much easier to read

# What am I?
ip addr show
ip -brief addr        # much easier to read

The routing table answers one question for every outgoing packet: which interface, and to whom next?

ip route

# default via 192.168.1.1 dev eth0
# 192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.10

ip route

# default via 192.168.1.1 dev eth0
# 192.168.1.0/24 dev eth0 proto kernel scope link src 192.168.1.10

Read that as two rules. The second says “anything in 192.168.1.0/24 is on my own network, send it straight out”. The first — the default route — says “everything else, hand to 192.168.1.1 and let it deal with it”.

Most specific match wins, and no default route means no internet while the local network keeps working perfectly. That combination is diagnostic on its own.

# Which route would this packet take?
ip route get 8.8.8.8

# Where does it break down?
traceroute 8.8.8.8
mtr 8.8.8.8          # traceroute and ping combined, live

# Which route would this packet take?
ip route get 8.8.8.8

# Where does it break down?
traceroute 8.8.8.8
mtr 8.8.8.8          # traceroute and ping combined, live

ip route get is underused and answers a real question: on a machine with several interfaces — a VPN, Docker bridges, two NICs — it tells you which one traffic to a given destination will actually use.

Private addresses cannot be routed on the internet, so your router rewrites them. Outgoing packets get the router’s public address; replies get translated back. That is NAT, and it is why ip addr shows 192.168.1.10 while a website sees something entirely different.

# What the internet sees
curl -s https://api.ipify.org; echo

# What the internet sees
curl -s https://api.ipify.org; echo

The consequence that shapes a lot of infrastructure: NAT is one-way by default. You can reach out; nothing can reach in, because the router has no idea which internal machine an unsolicited packet was meant for. Port forwarding exists to tell it, and Tailscale exists largely to avoid needing to.

Docker does the same thing on a smaller scale — containers get private addresses on a bridge network and are NATed out through the host, which is exactly why publishing a port is a separate deliberate act, and why those published ports bypass ufw. Linux firewalls covers that trap.

An address gets you to a machine. A port gets you to a program on it. A service listens on a port; a client connects from a random high-numbered one.

The detail that matters most is which address a service binds to:

sudo ss -tulpn

Binding to localhost is a stronger control than a firewall rule, because it cannot be undone by a misconfigured rule elsewhere. Where a service supports it, bind it there and let the firewall be the backup.

One refinement to that, because there is a stronger control still. A service bound to 127.0.0.1 is still reachable by every user and every process on the machine — localhost is a network, and a port has no owner and no permissions. A Unix socket has no address at all: it lives in the filesystem with an owner, a mode and a directory you control, and the kernel will tell the server which process is on the other end. That is what your database, your container runtime, systemd and your logging daemon are all using locally. The Life of a Unix Socket follows one through all seven stages — starting with the fact that only the write bit gates a connection, so the mode that looks safest is usually the one that locks everybody out.

Four outcomes worth distinguishing, because they point at different layers:

• Connection refused — a RST came back at once. You reached a machine, and either nothing is listening on that port or a firewall answered for it with --reject-with tcp-reset. The network is fine.
• Connection timed out — nothing answered at all, for about two minutes. A firewall dropped it, the host is dead, or a route leads nowhere.
• No route to host — an ICMP error, and on a host you can ping it is almost always a firewall’s REJECT rule, not your routing table.
• Success, in steps of whole seconds — no error at all. The server’s accept queue is full, its kernel dropped your SYN, and yours retried. The Life of a TCP Connection has all five outcomes and the rule of reading the errno and the elapsed time together.

DNS turns names into addresses, and it is responsible for a wildly disproportionate share of problems that feel like something else — because when it is slow, everything is slow, and when it is wrong, everything is wrong in confusing ways.

Resolution order on a Linux machine:

1. /etc/hosts — checked first, overrides everything. This is what makes the pre-DNS testing trick in migrating a server work.
2. The configured resolver, from /etc/resolv.conf or systemd-resolved.
3. The wider DNS hierarchy, if the resolver has no cached answer.

# Ask your configured resolver
dig example.com
dig +short example.com

# Bypass it and ask a public one - this is the key test
dig +short example.com @1.1.1.1

# Which resolver am I even using?
resolvectl status
cat /etc/resolv.conf

# Mail, name servers, everything
dig example.com MX
dig example.com NS
dig +trace example.com

# Ask your configured resolver
dig example.com
dig +short example.com

# Bypass it and ask a public one - this is the key test
dig +short example.com @1.1.1.1

# Which resolver am I even using?
resolvectl status
cat /etc/resolv.conf

# Mail, name servers, everything
dig example.com MX
dig example.com NS
dig +trace example.com

The comparison is the diagnosis. If dig @1.1.1.1 returns the right answer and your normal resolver does not, the problem is your resolver or a stale cache — not the domain, and not the far end.

Use dig rather than ping to test DNS. ping mixes name resolution and reachability into a single result, so a failure tells you nothing about which one broke.

Work outwards. Each step assumes the previous one passed, so the first failure names the layer.

# 1. Do I have an address? (169.254.x.x means DHCP failed)
ip -brief addr

# 2. Is the interface actually up?
ip link

# 3. Can I reach my own gateway?
ip route
ping -c3 $(ip route | awk '/default/ {print $3}')

# 4. Can I reach the internet by address, ignoring DNS?
ping -c3 1.1.1.1

# 5. Does DNS work?
dig +short example.com
dig +short example.com @1.1.1.1

# 6. Is the service listening, and on what?
sudo ss -tulpn | grep :443

# 7. Can I reach that port from outside?
nc -zv example.com 443
curl -I https://example.com

# 1. Do I have an address? (169.254.x.x means DHCP failed)
ip -brief addr

# 2. Is the interface actually up?
ip link

# 3. Can I reach my own gateway?
ip route
ping -c3 $(ip route | awk '/default/ {print $3}')

# 4. Can I reach the internet by address, ignoring DNS?
ping -c3 1.1.1.1

# 5. Does DNS work?
dig +short example.com
dig +short example.com @1.1.1.1

# 6. Is the service listening, and on what?
sudo ss -tulpn | grep :443

# 7. Can I reach that port from outside?
nc -zv example.com 443
curl -I https://example.com

Step 4 versus step 5 is the split worth internalising: if ping 1.1.1.1 works and dig example.com does not, your network is fine and DNS is broken. That single distinction resolves a large fraction of “the internet is down” reports.

The TTL one catches people during migrations especially. A DNS record you changed is still cached by resolvers everywhere for as long as its TTL said, and nothing you do locally speeds that up — which is why lowering the TTL days in advance is the first step of any cutover.

• Networking basics — the commands in practice
• The Life of a Network Interface — the long one on how an interface gets its name, its owner and its address, and why a failed DHCP leaves no address at all on most machines
• The Life of a TCP Connection — the long one on what “refused” and “timed out” actually prove, the three outcomes between them, and the connection both ends call established that neither can use
• Linux firewalls — what is dropping your packets
• Migrating a server — DNS TTLs and the hosts-file trick
• curl and wget — testing a service once you can reach it
• Tailscale — sidestepping NAT entirely
• Reading Linux logs — what the service says about the connection