Where are the logs? journalctl, /var/log and container logs

On a modern Linux server logs live in three places: the systemd journal (everything started by systemd, plus the kernel), plain files under /var/log (programs that write their own files, and often a copy of the journal), and container runtimes, which capture whatever a container writes to stdout and stderr. The first question when something misbehaves is which of the three applies.

The journal: journalctl

Any service managed by systemd has its output captured automatically. The most useful filters:

journalctl -u nginx.service -n 100
journalctl -u nginx.service -f
journalctl -u nginx.service --since "1 hour ago"
journalctl -u nginx.service --since "2026-10-14 02:00" --until "2026-10-14 03:00"
journalctl -p err -b
journalctl -k
journalctl _PID=12345
journalctl -u nginx.service -o json-pretty -n 1

• -u selects a unit, -n the number of lines, -f follows like tail -f.
• -p err shows priority error and worse; -b limits to the current boot (-b -1 is the previous boot, useful after a crash).
• -k is the kernel ring buffer, the same thing dmesg shows, with timestamps.
• -o json-pretty reveals every field a message carries; any field such as _PID, _UID or _COMM can be used as a filter.

If journalctl -b -1 says there is no previous boot, the journal is volatile. Make it persistent with sudo mkdir -p /var/log/journal && sudo systemctl restart systemd-journald. Check how much space it uses with journalctl --disk-usage.

Files under /var/log

Programs that predate systemd, or that handle high volumes, write their own files. The names differ by distribution, but these are the usual suspects:

ls -lt /var/log | head
sudo tail -f /var/log/syslog
sudo tail -f /var/log/auth.log
sudo tail -f /var/log/nginx/error.log
sudo less +F /var/log/apache2/error.log
sudo zgrep -i "out of memory" /var/log/syslog*

/var/log/syslog and /var/log/auth.log are the general and the login/sudo logs on Debian and Ubuntu; on RHEL, Fedora and SUSE the same roles are played by /var/log/messages and /var/log/secure. ls -lt sorts by modification time, so the file that changed most recently is at the top, which is often the one you want. less +F follows a file like tail -f but lets you press Ctrl-C to scroll back and F to resume. Rotated files end in .1, .2.gz and so on; zgrep searches the compressed ones without unpacking them. Rotation is controlled by /etc/logrotate.conf and /etc/logrotate.d/.

Not sure which file a program writes to? Ask the process: sudo ls -l /proc/$(pidof nginx | cut -d' ' -f1)/fd | grep log lists the log files it has open.

Docker and Podman

A container’s logs are whatever its main process printed to stdout and stderr. Docker keeps them per container as JSON files under /var/lib/docker/containers/<id>/ and exposes them through one command:

docker ps
docker logs --tail 100 -f web
docker logs --since 15m web
docker logs --timestamps web 2>&1 | grep -i error
docker compose logs -f --tail 50 web db
podman logs --tail 100 -f web

Docker writes stdout and stderr to separate streams, so pipe 2>&1 before grep to see both. If a container logs to a file inside its filesystem instead of stdout, you have to go in: docker exec -it web tail -f /var/log/app.log. Docker’s default json-file driver grows forever unless you set max-size and max-file in /etc/docker/daemon.json; the journald driver sends everything to the system journal instead, where journalctl CONTAINER_NAME=web finds it.

Kubernetes

kubectl get pods -n shop
kubectl logs -n shop deploy/web --tail 100 -f
kubectl logs -n shop web-7c9f6d-xk2lp -c sidecar
kubectl logs -n shop web-7c9f6d-xk2lp --previous
kubectl logs -n shop -l app=web --all-containers --since 10m

--previous is the single most useful flag when a pod is in CrashLoopBackOff: it shows the output of the container that just died. -l selects by label so you see all replicas at once. On the node itself the files are under /var/log/pods/ and /var/log/containers/, which is where Loki’s agent reads them.

When the log is not where it should be

• The service may log to a file you have not found: check the program’s configuration for log_file, error_log, logging or similar keys.
• The service may run under a different name: systemctl list-units --type=service | grep -i name.
• Logging may be disabled or set to a level above the message you expect; turning on debug logging briefly is often the fastest way to see what a program is doing.
• On a host with many services, searching every place by hand does not scale. That is the moment to ship logs to a central store such as Loki, covered in Loki: ship and search logs.

Official documentation

• journalctl manual
• docker container logs
• kubectl logs