Linux logs reference: journalctl, journald storage, logrotate and centralized logging

On a modern Linux server, logs live in two places: the systemd journal, which collects output from services, the kernel and syslog, and classic text files under /var/log written by applications or by a syslog daemon. You read the journal with journalctl, and you keep text files from filling the disk with logrotate.

journalctl: reading the journal

journalctl -u nginx                      # one service
journalctl -u nginx -f                   # follow new messages, like tail -f
journalctl -u nginx -n 100               # last 100 lines
journalctl -e                            # jump to the end in the pager
journalctl -b                            # since the current boot
journalctl -b -1                         # the previous boot
journalctl --list-boots
journalctl -k                            # kernel messages (dmesg)
journalctl -p err -b                     # priority err and more severe, this boot
journalctl --since "2026-10-15 08:00" --until "2026-10-15 09:00"
journalctl --since yesterday
journalctl --since "-1h" -u sshd
journalctl -g "timeout" -u nginx         # grep the message text
journalctl -u nginx -o json-pretty -n 1  # all fields of one entry

Priorities follow syslog levels: emerg (0), alert (1), crit (2), err (3), warning (4), notice (5), info (6), debug (7). A single level shows that level and everything more important; a range in the form FROM..TO shows just that range. Options combine, so -u, -p and --since together narrow a search fast.

Regular users may see only their own logs; add your admin user to the systemd-journal or adm group (depending on the distribution) or use sudo to read system logs.

journald storage and disk use

The Storage= setting in /etc/systemd/journald.conf decides where the journal is kept: persistent under /var/log/journal, volatile in memory under /run/log/journal (lost at reboot), or auto, which is persistent only if /var/log/journal exists. Defaults differ between distributions, so if journalctl --list-boots shows only the current boot, your journal is probably volatile.

journalctl --disk-usage
sudo journalctl --vacuum-time=2weeks
sudo journalctl --vacuum-size=500M

Vacuuming deletes the oldest archived journal files until the limit is met; the deleted entries cannot be recovered. Check --disk-usage first. For a permanent limit set SystemMaxUse= (for example SystemMaxUse=1G) in journald.conf and restart systemd-journald.

logrotate: rotating text log files

Applications that write their own log files would grow them forever. logrotate, usually run daily by a timer or cron job, renames the current file, optionally compresses old ones and deletes the oldest. Per-application settings go in /etc/logrotate.d/.

# /etc/logrotate.d/myapp
/var/log/myapp/*.log {
    daily
    rotate 14
    missingok
    notifempty
    compress
    delaycompress
    create 0640 appuser adm
    sharedscripts
    postrotate
        systemctl reload myapp.service > /dev/null 2>&1 || true
    endscript
}

• daily / weekly / monthly: how often; maxsize also rotates earlier when the file grows past a size.
• rotate 14: keep 14 old files, delete older ones.
• missingok: no error if the log is missing. notifempty: skip empty files.
• compress gzips old files; delaycompress waits one cycle, for programs that keep writing to the old file briefly.
• create MODE OWNER GROUP: create a fresh empty log after rotating.
• postrotate … endscript: a command to make the program reopen its log file. With sharedscripts it runs once for the whole block, not once per file.
• copytruncate: for programs that cannot reopen their log. It copies and then truncates the file in place; a few lines written in between can be lost.

sudo logrotate -d /etc/logrotate.d/myapp   # debug: show what would happen, change nothing
sudo logrotate -f /etc/logrotate.d/myapp   # force a rotation now

Use myapp.service only as a placeholder: replace the reload command with whatever makes your program reopen its log file, or use copytruncate if it cannot.

Centralized logging

Logging in to each server to read logs stops working beyond a handful of machines, and logs on a server that crashed or was deleted are gone. Centralized logging ships logs from every host to one system where you can search, keep them for a set time and alert on them. The usual pieces are an agent on each host that reads the journal and files, a log store, and a UI.

• Grafana Loki is a log aggregation system inspired by Prometheus. It does not index the full text of logs, only a set of labels per log stream (such as host, job, unit), and stores compressed log chunks, often in object storage. You query it with LogQL in Grafana. An agent such as Grafana Alloy collects logs, attaches labels and pushes them to Loki over HTTP.
• The Elastic stack (Elasticsearch with Kibana, fed by Beats agents or Logstash) indexes the log content itself, which makes full-text search fast at the cost of more storage and resources.
• Classic syslog forwarding (rsyslog or syslog-ng sending to a central log server) is the simplest option and is often what network devices support.

Whatever you choose, keep labels or fields low in variety (host, service, environment), not values like user IDs or request IDs, and set a retention period that matches your needs and any legal requirements.

Official documentation

• journalctl(1) manual page
• logrotate(8) manual page
• Grafana Loki overview