Introduction
System administrators working with Ubuntu and Debian often need to consult log files to troubleshoot issues. This is a fundamental step in diagnosing and resolving problems on these operating systems.
Linux-based systems, including Ubuntu and Debian, generate a variety of messages that are recorded in log files. These log messages are crucial for understanding the system’s behavior and identifying issues. Linux relies on a combination of configuration files, directories, programs, commands, and daemons to manage and organize these log messages. Familiarity with the locations of log files and the associated commands can significantly expedite the troubleshooting process.
In this guide, we will explore the key aspects of the Linux logging system in the context of Ubuntu and Debian.
Step 1 – Identifying Default Log File Locations
By default, log files on Ubuntu and Debian are stored in the /var/log directory. You can view the list of log files in this directory using the following command:
$ ls -l /var/logWhen executed on your system, the output will resemble the following:
total 49316
drwxr-xr-x. 2 root root 6 Sep 27 19:17 anaconda
drwx------. 2 root root 99 Jan 3 08:23 audit
-rw-rw----. 1 root utmp 1234560 Jan 3 16:16 btmp
-rw-rw----. 1 root utmp 17305344 Jan 1 00:00 btmp-20230101
drwxr-x---. 2 chrony chrony 6 Aug 10 2021 chrony
-rw-r--r--. 1 root root 130466 Dec 8 22:12 cloud-init.log
-rw-r-----. 1 root adm 10306 Dec 8 22:12 cloud-init-output.log
-rw-------. 1 root root 36979 Jan 3 16:03 cron
-rw-------. 1 root root 27360 Dec 10 23:15 cron-20221211
-rw-------. 1 root root 94140 Dec 17 23:07 cron-20221218
-rw-------. 1 root root 95126 Dec 24 23:14 cron-20221225
-rw-------. 1 root root 95309 Dec 31 23:04 cron-20230101
…Step 2 – Viewing Log Contents
Under /var/log, you’ll encounter various log files with specific purposes. Some common examples include:
wtmputmpdmesgmessagesmaillogormail.logspoolerauth.logorsecure
wtmp and utmp files track user logins and logouts. To see the currently logged-in users on a Linux server, use the who command. On Ubuntu, the output could look like this:
root@ubuntu-22:~# who
root pts/0 2023-01-03 16:23 (198.7.211.4)To retrieve login history, the last command is useful:
root@ubuntu-22:~# last
root pts/0 198.7.211.4 Tue Jan 3 16:23 still logged in
reboot system boot 5.19.0-23-generi Thu Dec 8 21:48 still running
wtmp begins Thu Dec 8 21:48:51 2022For determining the last reboot time, use:
$ last rebootIn Debian, the output might appear as follows:
root@debian-11-trim:~# last reboot
reboot system boot 5.10.0-11-amd64 Thu Dec 8 21:49 still running
wtmp begins Thu Dec 8 21:49:39 2022The lastlog command can reveal the last login times for users:
$ lastlogOutput on a Debian system might resemble:
root@debian-11-trim:~# lastlog
Username Port From Latest
root pts/0 162.243.188.66 Tue Jan 3 16:23:03 +0000 2023
daemon **Never logged in**
bin **Never logged in**
sys **Never logged in**
sync **Never logged in**
games **Never logged in**
man **Never logged in**
lp **Never logged in**
mail **Never logged in**
news **Never logged in**
uucp **Never logged in**
proxy **Never logged in**
www-data **Never logged in**
backup **Never logged in**
list **Never logged in**
irc **Never logged in**
gnats **Never logged in**
nobody **Never logged in**
_apt **Never logged in**
messagebus **Never logged in**
uuidd **Never logged in**
…To access the contents of text-based log files, commands like cat, head, or tail can be employed. For example, to view the last ten lines of the /var/log/messages file on a Debian server:
$ sudo tail /var/log/messagesExpected output:
root@debian-11-trim:~# tail /var/log/messages
Jan 1 00:10:14 debian-11-trim rsyslogd: [origin software="rsyslogd" swVersion="8.2102.0" x-pid="30025" x-info="https://www.rsyslog.com"] rsyslogd was HUPed
Jan 3 16:23:01 debian-11-trim DropletAgent[808]: INFO:2023/01/03 16:23:01 ssh_watcher.go:65: [SSH Watcher] Port knocking detected.
Jan 3 16:23:01 debian-11-trim DropletAgent[808]: INFO:2023/01/03 16:23:01 do_managed_keys_actioner.go:43: [DO-Managed Keys Actioner] Metadata contains 1 ssh keys and 1 dotty keys
Jan 3 16:23:01 debian-11-trim DropletAgent[808]: INFO:2023/01/03 16:23:01 do_managed_keys_actioner.go:49: [DO-Managed Keys Actioner] Attempting to update 1 dotty keys
Jan 3 16:23:01 debian-11-trim DropletAgent[808]: INFO:2023/01/03 16:23:01 do_managed_keys_actioner.go:65: [DO-Managed Keys Actioner] Attempting to create 1 ssh keys
Jan 3 16:23:01 debian-11-trim DropletAgent[808]: INFO:2023/01/03 16:23:01 do_managed_keys_actioner.go:105: [DO-Managed Keys Actioner] Action metadata updated
Jan 3 16:23:02 debian-11-trim kernel: [ 22.685314] Bluetooth: RFCOMM TTY layer initialized
Jan 3 16:23:02 debian-11-trim kernel: [ 22.685320] Bluetooth: RFCOMM socket layer initialized
Jan 3 16:23:02 debian-11-trim kernel: [ 22.685321] Bluetooth: RFCOMM ver 1.11
Jan 3 16:23:03 debian-11-trim login[956]: ROOT LOGIN on '/dev/pts/0'For logs that update in real-time, the tail -f command can be used to “follow” the log and display new entries as they are added:
$ sudo tail -f /var/log/syslogTo view the beginning of a log file, use head:
$ sudo head /var/log/syslogStep 3 – Using Logrotate for Log Management
The logrotate utility automates the management of log files. It can archive old log files, compress logs, and clean up log directories to prevent excessive disk usage.
To configure logrotate, edit the /etc/logrotate.conf file using a text editor (such as nano or vim). The logrotate configuration file is well-commented and provides clear guidelines on how to set up rotation rules for log files.
Here’s an example of what the configuration file might look like:
# see "man logrotate" for details
# rotate log files weekly
weekly
# keep 4 weeks worth of backlogs
rotate 4
# create new (empty) log files after rotating old ones
create
# use date as a suffix of the rotated file
dateext
# uncomment this if you want your log files compressed
# compress
# packages drop log rotation information into this directory
include /etc/logrotate.d
# no packages own wtmp, or btmp -- we'll rotate them here
/var/log/wtmp {
missingok
monthly
create 0664 root utmp
minsize 1M
rotate 1
}
/var/log/btmp {
missingok
monthly
create 0660 root utmp
rotate 1
}
# system-specific logs may be also be configured hereIn this example, log files are rotated weekly, kept for a total of 4 weeks, and new log files are created after rotation. Additionally, log files are compressed after rotation.
To add or edit specific log rotation rules, you can create or edit files in the /etc/logrotate.d directory. For instance, a file named /etc/logrotate.d/nginx would contain rules specifically for rotating the Nginx web server logs.
Step 4 – Using the rsyslog Daemon
The rsyslog daemon is a powerful and flexible system logging service available on Ubuntu and Debian systems. It replaces the older syslog system and provides advanced features for handling log messages. rsyslog allows you to filter, route, and store log messages in various formats and destinations.
Installation and Basic Configuration
- Installation:
Check if rsyslog is already installed on your system by running:
$ sudo dpkg -l | grep rsyslogIf it’s not installed, you can install it using the following command:
$ sudo apt-get update
$ sudo apt-get install rsyslog- Configuration Files:
rsyslog‘s main configuration file is located at /etc/rsyslog.conf. However, it’s recommended to make configuration changes in separate files in the /etc/rsyslog.d/ directory to keep things organized and avoid directly modifying the main file.
Understanding Configuration Lines
The rsyslog.conf file consists of a series of configuration lines, each comprising a two-part instruction: a selector and an action. These parts are separated by white space.
- Selector: The selector part determines the source and importance of the log message. It is divided into two components by a dot (
.):- Facility: Denotes the origin of the message, such as
auth,kern,mail,user, etc. - Priority: Represents the severity level of the message, ranging from
debug(lowest) toemerg(highest).
- Facility: Denotes the origin of the message, such as
- Action: The action part specifies what
rsyslogshould do with log messages that match the provided selector. This can involve writing messages to specific log files, forwarding them to remote servers, and more.
Example Configuration Line
Here’s an example of a configuration line that logs kernel messages (kern) of priority err (error) level:
kern.err /var/log/kernel_errors.logIn this example:
kern.erris the selector, indicating that it’s for kernel messages of error priority./var/log/kernel_errors.logis the action, specifying that matching log messages should be saved to thekernel_errors.logfile.
Viewing Configuration Files
On Ubuntu systems, the default configuration file is often found at /etc/rsyslog.d/50-default.conf. You can use the following command to view its contents:
$ cat /etc/rsyslog.d/50-default.confFacilities and Priorities
Different facilities represent various sources of log messages, such as auth, kern, mail, and more. Each message is assigned a priority indicating its severity level, ranging from debug (lowest) to emerg (highest).
Here are a few facilities and priorities recognized by rsyslog:
Facilities:
authorauthpriv: Authorization and security-related eventsuser: Log messages coming from user programsmail: Log messages generated by the mail subsystemlocal0tolocal7: Reserved for local usekern: Messages originating from the Linux kernel
Priorities:
debug: Debug information from programsinfo: Simple informational messagesnotice: Conditions that may require attentionwarn: Warningserr: Errorscrit: Critical conditionsalert: Conditions that need immediate interventionemerg: Emergency conditions
Understanding these facilities and priorities is essential for configuring rsyslog effectively.
- Basic Logging: By default,
rsysloglogs messages to the/var/log/syslogfile. You can view the contents of this file usingcator other commands as mentioned earlier.
Advanced Configuration and Filtering
- Filtering with Templates:
rsyslog allows you to filter and process log messages based on various attributes like facility, severity, or program name. You can create custom templates to format log messages. For instance, to forward specific logs to a remote server:
- Create a custom configuration file in
/etc/rsyslog.d/, like/etc/rsyslog.d/50-remote.conf. - Add the following lines to forward messages from the
authfacility to a remote server with IP1.2.3.4:
if $programname == 'sshd' and $syslogfacility-text == 'auth' then @@1.2.3.4:514- Creating Custom Log Files:
You can create custom log files for specific applications or services. For example, to create a log file for a service named myapp:
- Create a custom configuration file in
/etc/rsyslog.d/, like/etc/rsyslog.d/60-myapp.conf. - Add the following lines to create a log file at
/var/log/myapp.logfor messages from themyappprogram:
if $programname == 'myapp' then /var/log/myapp.logApplying Changes and Restarting rsyslog
After making changes to rsyslog configuration files, you need to restart the rsyslog service to apply them:
$ sudo service rsyslog restartDebugging and Troubleshooting
If you encounter issues with logging or configuration, you can check the /var/log/syslog file for rsyslog related messages. Additionally, rsyslog itself logs its actions, so you can use the following command to view these logs:
$ sudo journalctl -u rsyslogThis will show you rsyslog‘s logs, which can be helpful for diagnosing any problems related to the logging service itself.
Conclusion
Accessing and configuring system logs is an essential skill for Linux system administrators. Ubuntu and Debian provide a robust set of tools and commands for managing log files, which are crucial for diagnosing issues and monitoring system behavior.
By understanding default log file locations, utilizing log viewing commands like cat, tail, and head, and configuring log rotation with logrotate, administrators can effectively manage and maintain log files on their systems. This ensures that important log data is preserved while preventing the accumulation of excessive disk usage.
Français
English
العربية
Webhi.Technology
@WebHiTechnology