Finding and analyzing system log files in Linux – RHEL/Rocky/CentOS/Alma

by

In this blog we will learn finding and analyzing system log files in linux operating system specifically to RHEL and its fork such as Rocky , CentOS and Alma Linux

Log files are the most important aspect in a linux troubleshooting for system administrator. They provide enormous of information about a system and are instrumental during troubleshooting and auditing.

Log files contain events and messages generated by the kernel, applications, and users that log into the system. Let us deep dive to what such logs are available for troubleshooting

Rsyslog

Syslog and rsyslog have a long historical presence on linux system and has been used to provide the logging. Systemd became the default service manager with RHEL & its fork, and it introduced its own logging system called systemd-journald. systemd-journald continues to be the logging mechanism on all Rocky 8 and 9 while keeping rsyslog for backward compatibility.

The rsyslog service keeps various log files in the /var/log directory. You can open these files using native commands such as tail, head, more, less, cat, and so forth, depending on what you are looking for.

For example, to display boot and other kernel messages, view /var/log/messages:

[rocky-server]$ cat /var/log/messages

We can also use grep and other filtering tools to gather more specific events from a file. One can also use tail to view files as they are continuously updated:

[rocky-server]$ tail -f /var/log/messages

In the command above, the -f option updates the output when new log file entries are added.

Check the /var/log/secure file to check users & their activities 

[rocky-server]$ tail -f /var/log/secure

systemd-journald

The systemd-journald service does not keep separate files, as rsyslog does. The idea is to avoid checking different files for issues. Systemd-journald saves the events and messages in a binary format that cannot be read with a text editor. You can query the journal with the journalctl command

To check all the events yiu can check the below logs 

[rocky-server]$ journalctl

The above is equivalent to /var/log/messages in the rsyslog service.

To check the last 10 events use the command 

[rocky-server]$ journalctl -n

As per your requirement you can also view the last n entries . For example if you need to see the last 30 events you can use the command 

[rocky-server]$ journalctl -n 30

To view the new entries while being written at the same time. This is very useful when you try to check the logs while you changed some configuration on the system in real time 

[rocky-server]$ journalctl -f

To display the kernel message log from the last boot:

[rocky-server]$ journalctl -k

The journalctl command provides several options that can make querying the journal easier. You can query the log based on applications, time frame, systemd units, priority, and many other options. You can check all such options using the command journalctl -help command to list the available options.

To view event entries based on their critical priority, run:

[rocky-server]$ journalctl -p crit

To view journal entries for today, use:

[rocky-server]$ journalctl --since today

To view entries related to the sshd daemon, run:

[rocky-server]$ journalctl -u sshd

You can change the service name running under the systemd based on the need of troubleshooting.

Let see another examaple where you need to check httpd logs for the past hours , run: 

[rocky-server]$ journalctl -u httpd –since "1 hour ago"

Conclusion

Current Rocky (All RHEL fork) distributions rely on systemd and the related journald logging tool but rsyslog still plays a major role in logging for many administrators. When it comes to 7log forwarding and centralization rsyslog is still preferred by majority of the engineers hence its quite mandatory to know how to use both log mechanisms effectively. These commands will help you learn and use system logging for troubleshooting and audits. Go ahead with your troubleshooting sessions making use of above command/logs.

If you need to get a Linux system installed for your hands on practice follow How to Install Rocky Linux 8.

You may also like to go through Journalctl Cheat Sheet

Leave a comment