systemd Log Management in Practice: From journalctl to Log Troubleshooting

systemd's logging system ——journal, collected and stored the standard output/error output from the kernel, initrd, services, and applications. Mastery journalctl, which means mastering half of Linux troubleshooting.

1. Basic viewing: from full data to pagination

# View all logs (default pagination, less mode)
journalctl

# Real-time scrolling tracking (similar to tail -f)
journalctl -f

# Display in reverse order (newest logs first)
journalctl -r

# Limit display to the most recent 100 lines.
journalctl -n 100

2. Filter by time and startup offset

Time filtering(support -S/--since and -U/--until):

# From the early hours of today until now
journalctl --since="today"

# Past 1 hour
journalctl --since="1 hour ago"

# Precise time range
journalctl --since="2024-03-20 10:00:00" --until="2024-03-20 11:00:00"

Startup offset:

# The log for the current startup
journalctl -b

# Log from the last startup
journalctl -b -1

# List all startup sequence numbers
journalctl --list-boots

3. Filter by unit/process/priority

By service unit:

journalctl -u nginx.service
journalctl -u nginx -u php-fpm    # Multi-unit joint

Real-time monitoring service logs(One of the most practical O&M operations):

# Continuously follow the log output of a specific service (similar to tail -f)
journalctl -u nginx.service -f

# Monitor real-time logs of multiple services simultaneously
journalctl -u nginx -u php-fpm -f

# Real-time viewing with ISO timestamp format
journalctl -u nfs-server -f -o short-iso

Practical Example: When a connection problem occurs with the NFS service, open the real-time monitoring window:

journalctl -u nfs-server -f

Then initiate an NFS mount request in another terminal, and the current window will immediately display detailed logs of the service response, making it easy to quickly identify issues such as permissions, timeouts, or network problems.

By PID/User:

journalctl _PID=1234
journalctl _UID=1000

By priority(similar to syslog level):

# emerg(0) alert(1) crit(2) err(3) warning(4) notice(5) info(6) debug(7)
journalctl -p err               # Error and above only
journalctl -p 3..5              # err to notice

4. Advanced Filtering and Output Control

Kernel and boot messages:

journalctl -k            # Kernel logs only
journalctl --dmesg       # Same as above.
journalctl --grep="OOM"  # Regular expression filtering of message content

Output format:

journalctl -o json-pretty   # JSON format, convenient for script processing
journalctl -o cat           # Message body only, no metadata.
journalctl -o verbose       # Display all fields

5. Log Maintenance and Persistence

The default journal is Volatility of (existence /run/log/journal, lost upon restart). If persistence is required:

sudo mkdir -p /var/log/journal
sudo systemd-tmpfiles --create --prefix /var/log/journal
sudo systemctl restart systemd-journald

Space constraints(Modify /etc/systemd/journald.conf):

SystemMaxUse=2G
MaxFileSec=1month

After applying the configuration:

sudo systemctl restart systemd-journald

6. Typical troubleshooting scenarios

Scenario 1: The service suddenly went down, and there was no time to check the logs.

# View logs from the last boot up to the current time
journalctl -u myapp.service -b -1

Scenario 2: System OOM, but dmesg has already rolled over

journalctl -k --grep="Out of memory"

Scenario 3: Troubleshooting SSH Login Failure

journalctl -u sshd --since="1 hour ago" -p 3..5

Scenario 4: Real-time location service issue (using NFS as an example)

# Start real-time monitoring terminal
journalctl -u nfs-server -f

# Reproduce the issue in another terminal (e.g., mount failure)
mount -t nfs 192.168.1.100:/exports /mnt

# The real-time terminal will immediately output the detailed response of the NFS service:
# - Authentication failed (permission issue)
# - Timeout error (network issue)
# - Export directory does not exist (configuration issue)

Scenario 5: Export logs for offline analysis

journalctl -u mysql --since="yesterday" > mysql_issue.log

7. Core Metadata Fields Quick Reference

field	meaning
_PID	Process ID
_UID	User ID
_COMM	Command name (e.g., sshd)
_SYSTEMD_UNIT	Associated service unit
PRIORITY	0-7 priority
SYSLOG_FACILITY	syslog facility
MESSAGE	Actual log content

Conclusion

journalctl is a Structured log query engine, rather than simple tail /var/log/syslog. Proficient use of the above filter combinations can greatly shorten fault localization time. Finally, remember two life-saving parameters:

journalctl -xe      # Show additional information + jump to end
journalctl --vacuum-size=500M   # Emergency log slimming

📌Prompt: If using a container environment (docker/podman), their logs do not go through journal by default and need to be configured separately.log-driver=journald.