journald / journalctl

Journal, or commonly referenced by the name of its running deamon process journald or by its log reading utility journalctl, is a system for collecting, storing and reading logs. It was introduced with systemd service management system.

The journald daemon usually records the structured and indexed logs under /var/log/journal/ but this might differ according to configuration. You are meant to search and read the logs using journalctl utility tool.

Journal classifies messages by priority and facility.

Priority: how severe is the event
e.g.
0 emerg     System is unusable, should not be used by applications but the kernel
1 alert     Should be corrected immediately such as vital subsystem crash.
2 crit      Critical conditions such as system application crash.
3 err       Non-system-severing error reported by an application.
4 warning   Might lead to error if no action is taken like running out of space soon.
5 notice    Unusual event, but not yet an error or warning.
6 info      Normal operation, no action required.
7 debug     Information useful for developers of the application.

Facility: the type of application that does the logging
e.g.
0  kern
1  user
3  daemon
15 cron

# show the full contents of the journal, starting with the oldest entry collected
journalctl

# the underlying interface is to specify field filters
journalctl "FIELD=VALUE" FIELD=VALUE + ALTERNATIVE=SEARCH
# you find all the common fields in the manual
man systemd.journal-fields

# but it's more common to use shorthands provided by `journalctl` like the `-p` here:
journalctl PRIORITY=0 PRIORITY=1 PRIORITY=2 PRIORITY=3 PRIORITY=4
journalctl -p warning
# both of the above do the same thing
# `err` and above are red, `notice` and above are bold

Here are the more important parameters for journalctl:

-p, --priority=     # filter by priority, if single given, also shows lower numbers
-f, --follow        # print `-n` newest logs and watch for new logs
-n, --lines=10      # show this many last logs on start
-r, --reverse       # reverse output, show newest lines first
-o, --output=short  # how output is shown e.g. `short-full`,`short-iso`,`verbose`,`json`
--output-fields=    # override which fields are included
-q, --quiet         # suppress all non-log line messages e.g. `-- Reboot --`
--list-boots        # show recorded system boots
-b [ID][±offset]    # show messages from a specific boot
-k                  # show kernel messages of the current boot
-u, --unit=UNIT     # show messages of a specific systemd unit e.g. a service
-g, --grep=         # filter messages with `grep`, if the support has been installed
--since=, --until=  # filter by time, usually "1 min ago" or `short-full` output syntax
--system, --user    # switch context, by default searches all logs you can access
-N, --fields        # show all all unique fields used in your logs
-F, --field=        # show all unique values used for this field
--file=GLOB         # change which files to search, useful if you have a lot of logs

Some example commands:

journalctl --field=_EXE | sort   # list all executables that has recorded something
journalctl /opt/Todoist/todoist  # show logs from this executable

journalctl -u NetworkManager.service  # show logs from this systemd unit

journalctl --field=_UID  # show all unique user ids in all your Journal logs

journalctl --disk-usage  # show current disk usage

journalctl --since "20 min ago"  # show logs since 20 minutes ago...

Users are granted access to only their personal journals. Only root and members of systemd-journal, adm, and wheel groups can read all log files.