Fix: Failed to start service, how to read systemctl status and journalctl

Job for nginx.service failed because the control process exited with error code.
See "systemctl status nginx.service" and "journalctl -xeu nginx.service" for details.

Failed to start myapp.service: Unit myapp.service not found.
myapp.service: Main process exited, code=exited, status=203/EXEC
myapp.service: Failed to determine user credentials: No such process
myapp.service: Start request repeated too quickly.

systemd started the service’s process and the process exited with an error, or systemd could not even launch it. The one-line message is a pointer, not the cause. The cause is always in two places: the unit’s status, which shows the last few log lines and the exit code, and the journal, which has the full output of the program.

Step 1: read the status and the journal

systemctl status myapp.service --no-pager -l
journalctl -u myapp.service -b --no-pager -n 100     # this boot, last 100 lines
journalctl -xeu myapp.service                        # with explanations, jump to end
systemctl cat myapp.service                          # the unit file actually in use

In the status output, look at Active: (failed, activating auto-restart), Process: with status=..., and the log lines at the bottom. The exit code narrows it down fast:

• status=203/EXEC: the file in ExecStart= does not exist, is not executable, or has a wrong interpreter path. Check it with ls -l and run it by hand.
• status=217/USER or Failed to determine user credentials: the User= in the unit does not exist. Create it, or fix the name.
• status=226/NAMESPACE: a sandboxing option such as ProtectSystem=, ReadWritePaths= or a WorkingDirectory= points at a path that does not exist.
• status=1/FAILURE, status=2, or other small numbers: the program itself started and exited with an error. Its own message is in the journal lines just above.
• Unit ... not found: the unit file is not in /etc/systemd/system/ or you did not run daemon-reload after creating it.
• Start request repeated too quickly: it is crashing in a loop; systemctl reset-failed myapp after you fix the cause, then start again.

Cause 1: the program's own configuration is broken

For well-known daemons the quickest check is the built-in config test; it prints the file and line number.

sudo nginx -t
sudo apachectl configtest
sudo sshd -t
sudo postgres -D /var/lib/postgresql/16/main -C data_directory     # or check postgresql.log
sudo haproxy -c -f /etc/haproxy/haproxy.cfg

Cause 2: a port is already in use or an address is missing

The journal shows bind() to 0.0.0.0:80 failed (98: Address already in use) or Cannot assign requested address. Find the holder with sudo ss -ltnp 'sport = :80' and stop it or move one of the two; see the Fix article on EADDRINUSE. Cannot assign requested address means the IP in the config is not configured on any interface.

Cause 3: wrong path, user or permissions in the unit

# Run exactly what systemd runs, as the same user
sudo -u myapp /opt/myapp/bin/server --config /etc/myapp.toml

# Permission denied on a file or socket? Check ownership of what the service touches
ls -l /etc/myapp.toml /var/log/myapp /run/myapp

# Edit the unit safely with an override, then reload
sudo systemctl edit myapp.service
sudo systemctl daemon-reload
sudo systemctl restart myapp.service

Remember that a service has no PATH to speak of and no shell: ExecStart= needs an absolute path, cannot use ~, pipes or && unless you call /bin/sh -c '...', and does not read your .bashrc. Set variables with Environment= or EnvironmentFile=.

Cause 4: a dependency is not ready

An app that needs the network, a database or a mounted disk may start before it is available. After=network-online.target plus Wants=network-online.target waits for the network; After=postgresql.service orders it behind the database; RequiresMountsFor=/data waits for a mount. For flaky dependencies, Restart=on-failure and RestartSec=5 let it retry instead of failing once at boot.

Check that it is fixed

sudo systemctl restart myapp.service
systemctl is-active myapp.service           # active
systemctl status myapp.service               # Active: active (running), no error lines
journalctl -u myapp.service -f               # watch it for a minute
systemctl is-enabled myapp.service           # enabled, so it comes back after a reboot

When it is not what you think

• active (exited) is normal for Type=oneshot units and some init scripts; it is not a failure.
• Warning: The unit file ... changed on disk. Run 'systemctl daemon-reload' means systemd is still using the old version of the unit; your edit has not taken effect yet.
• If the journal is empty for the service, logging may go to a file (/var/log/myapp/) rather than stdout, or journald storage is volatile and the lines are from a previous boot: try journalctl -u myapp -b -1.
• A unit that fails only at boot but starts fine by hand is almost always an ordering or dependency problem (Cause 4), not a configuration error.

Official documentation

• systemctl(1) and journalctl(1)
• systemd.exec: process exit codes (203/EXEC, 217/USER, …)
• systemd.service