With one server, journalctl is the whole answer. With three, you are guessing which machine to look at, and a request that touched two of them cannot be followed at all. Centralising logs fixes that, and the usual failure is picking something built for a hundred machines when you have four.
This builds a small central log store using Loki, with Grafana Alloy shipping logs to it from each machine, and Grafana to search them.
What this does not do. It is not a backup of your logs — the retention you set is the retention you get, and the log server needs backing up like anything else. It is not an audit trail, and the gap there is wider than it sounds: it is not only that somebody with root on a machine can stop the agent or edit what it sends. Anything that can reach the ingest endpoint — including anything holding the shared credential your shippers use — can write a record carrying whatever _HOSTNAME, _UID, _PID, _COMM and _SYSTEMD_UNIT it likes, and the store will render it exactly like a witnessed one. The underscore-prefixed fields are guaranteed on the machine that produced them and nowhere else: The Life of a Log Line. It is not monitoring — shipping logs does not tell you when something breaks, and you still need checks that alert you. And it is one more service to run: if the log server is down, you are back to logging in to each machine, so do not put anything on it that the machines need in order to work.
Promtail is gone — read this before following an old tutorial
Almost every Loki guide written before 2025 tells you to install Promtail. Do not. Grafana put Promtail into a long-term-support phase in February 2025, declared it end of life on 2 March 2026, and removed it from Loki entirely as of version 3.7.3. It is not merely discouraged; recent Loki releases do not ship it.
The replacement is Grafana Alloy, which absorbed Promtail’s functionality. If you already have a Promtail configuration, there is a converter:
alloy convert --source-format=promtail \
--output=/etc/alloy/config.alloy /etc/promtail/config.yml
# or run the old config directly while you migrate
alloy run --config.format=promtail /etc/promtail/config.ymlThe conversion is not perfect — Promtail’s command-line flags do not transfer, the positions file moves, and the agent’s own metric names change, so any dashboards or alerts about the shipper itself need adjusting.
Step 1: pick where the logs will live
One machine, separate from the ones you are collecting from. A small VPS is fine. Give it disk rather than CPU — logs are write-heavy and mostly never read.
Loki’s monolithic mode — the single binary — is documented as suitable up to roughly 20 GB of logs per day, which is far more than a handful of servers will produce. Ignore every article about microservices deployment; it is not for you.
Verify: estimate your volume before building anything. On each machine, journalctl --disk-usage and the size of /var/log over a week gives you the number. If the total is under a gigabyte a day, everything below is comfortably oversized.
Step 2: run Loki and Grafana on the log server
# compose.yaml on the log server
services:
loki:
image: grafana/loki:3.7.6
restart: unless-stopped
command: -config.file=/etc/loki/loki.yaml
ports:
- "127.0.0.1:3100:3100"
volumes:
- ./loki.yaml:/etc/loki/loki.yaml:ro
- lokidata:/loki
grafana:
image: grafana/grafana:latest
restart: unless-stopped
ports:
- "127.0.0.1:3000:3000"
environment:
GF_SECURITY_ADMIN_PASSWORD: ${GRAFANA_PASSWORD}
volumes:
- grafanadata:/var/lib/grafana
volumes:
lokidata:
grafanadata:Both bound to localhost, with a reverse proxy in front providing TLS. Loki has no authentication of its own in this mode — it assumes something in front of it is handling that. Publishing port 3100 to the internet means anyone can write logs into your store, or read them.
A minimal loki.yaml with retention:
auth_enabled: false
server:
http_listen_port: 3100
common:
path_prefix: /loki
storage:
filesystem:
chunks_directory: /loki/chunks
rules_directory: /loki/rules
replication_factor: 1
ring:
kvstore:
store: inmemory
schema_config:
configs:
- from: 2024-04-01
store: tsdb
object_store: filesystem
schema: v13
index:
prefix: index_
period: 24h
limits_config:
retention_period: 720h # 30 days
compactor:
working_directory: /loki/compactor
retention_enabled: true
delete_request_store: filesystemUse schema v13. It is what enables structured metadata, which matters in step 4.
Verify: curl -s localhost:3100/ready returns ready, and curl -s localhost:3100/metrics | head returns Prometheus metrics. If /ready says it is still starting, wait — it takes a few seconds on first run.
Step 3: ship logs from each machine
Install Alloy on every machine whose logs you want, including the log server itself.
# /etc/alloy/config.alloy
loki.write "central" {
endpoint {
url = "https://logs.example.com/loki/api/v1/push"
basic_auth {
username = "shipper"
password_file = "/etc/alloy/password"
}
}
}
loki.relabel "journal" {
forward_to = []
rule {
source_labels = ["__journal__systemd_unit"]
target_label = "unit"
}
}
loki.source.journal "read" {
forward_to = [loki.write.central.receiver]
relabel_rules = loki.relabel.journal.rules
labels = {
host = constants.hostname,
job = "systemd-journal",
}
}
// plus any plain files not in the journal
local.file_match "applogs" {
path_targets = [{__path__ = "/var/log/myapp/*.log"}]
}
loki.source.file "applogs" {
targets = local.file_match.applogs.targets
forward_to = [loki.write.central.receiver]
}Reading the journal rather than files is the right default on any systemd machine — it picks up everything, including services that log nothing to disk. What arrives at the far end is not the whole entry, though. Only the journal fields a relabelling rule promotes become labels you can query; the rest of what journalctl shows you locally — PRIORITY, _PID, SYSLOG_IDENTIFIER, and every field your own application sets — is not there to filter on unless you promote it or attach it as structured metadata. Work out which fields you actually filter on locally before you decide which rules to write.
Alloy needs to be able to read the journal, which means adding its user to the systemd-journal group, and the password file should be chmod 600.
Verify: on the log server, query for the new host. In Grafana, add Loki as a data source pointing at http://loki:3100, then run {host="web1"} in Explore. Lines should appear within a few seconds. If nothing arrives, journalctl -u alloy -f on the sending machine says why — usually TLS, authentication, or the journal group.
Step 4: label sparingly — this is the part people get wrong
Loki does not index your log lines. It indexes labels, and groups lines into streams by them. Every distinct combination of label values is a separate stream with its own index entry and its own chunks.
So a label whose value varies a lot — a request ID, a user ID, a client IP, a timestamp — creates a stream per value. The index explodes, the chunks become tiny, and performance collapses. Grafana’s guidance is to stay around ten to fifteen labels at most, all with a small set of possible values.
Good labels: host, job, unit, env, level. That is close to the whole list — with the caveat that each of them needs a rule of its own. The configuration above promotes unit and nothing else, so level will not exist until you add a rule for it; for journal entries the source label to map from is __journal_priority_keyword.
For the high-cardinality things you still want to search on, the answer is structured metadata — attached to log entries without being indexed as labels. That is what schema v13 was for. Query performance for those is not as fast as a label, and that is the correct trade.
Searching the line content itself needs no labels at all, because Loki filters lines after selecting streams:
{job="systemd-journal"} |= "error" # substring
{host="web1", unit="nginx.service"} |= "500" # narrow first, then filter
{job="systemd-journal"} | json | duration > 5s # parse and compare
sum by (host) (rate({job="systemd-journal"} |= "error" [5m]))Verify: in Grafana, check the number of streams you have created — if it is in the thousands for a handful of servers, a label is carrying variable data. Find it before the index gets large.
Step 5: decide what happens when the log server is unreachable
Alloy buffers in memory and retries. That covers a restart or a brief network problem; it does not cover the log server being down for a day. Logs that cannot be delivered are eventually dropped.
This is usually acceptable, on one condition: keep local logging on. Do not turn off journald persistence because logs are being shipped elsewhere. The local journal is what you read when the shipping stops working, and it is the only copy that exists during an incident that takes out the network.
# /etc/systemd/journald.conf
[Journal]
Storage=persistent
SystemMaxUse=2GVerify: stop Loki on the log server for five minutes, then start it again. Logs from the gap should arrive; anything longer will not. Knowing your actual tolerance is better than assuming it.
The alternatives, briefly
| Approach | Suits | Cost |
|---|---|---|
| Loki + Alloy | Small estates; you already use Grafana | Two services to run |
| Vector into Loki or S3 | Routing and transforming on the way | Another config language to learn |
systemd-journal-upload | Journals only, no extra ecosystem | Maintained in systemd; a separate package on most distributions. Its buffering is one saved cursor (--save-state) and it resumes from there — so if the local journal rotated or was vacuumed past that point, the gap is never sent and nothing at either end records that it existed |
| rsyslog to a central host | Existing syslog setups | Text files at the far end; you are grepping again |
| OpenSearch | Full-text search over everything | Its own docs suggest 4 GB of host RAM as a floor |
Two notes worth having. Vector is a genuinely good pipeline tool, MPL-licensed and maintained by Datadog, and it is the right answer if you need to send the same logs to two places or reshape them in flight. And on syslog: rsyslog is still actively released and is still in Ubuntu’s default install, but Debian has not installed it by default since Debian 12 — on a modern Debian machine there is no /var/log/syslog unless you added one.
Before you call it done
- Loki reachable only through a proxy with TLS and authentication — never port 3100 on a public address
- Retention set deliberately, and the log server’s disk usage watched
- Every machine shipping, including the log server itself
- Labels reviewed — nothing variable, nothing above about fifteen
- One real entry queried at the far end and compared against the same entry locally, so you know which fields survived the trip
- Local journald persistence still on, with a size cap
- The log server included in your backups
- A monitoring check that tells you when logs stop arriving — silence looks identical to “nothing went wrong”
- One saved query you would actually run during an incident, written now rather than at 3am
Related reading
- The Life of a Log Line — the long one: what each field in an entry is, who wrote it, what the transport keeps and what it drops, and why the one rule that makes a log trustworthy stops at the edge of the machine
- Reading logs — the single-machine skills this builds on
- journalctl — what you are shipping
- Monitoring without a full stack — the thing that actually alerts you
- A reverse proxy with automatic TLS — what sits in front of Loki
- Docker Compose — the file above, explained