Malware in Linux: Rootkits, concealment and detection

In past articles Malware in Linux: Rootkits, introduction and classification and Malware in Linux: Kernel-mode-rootkits, descriptions were given of the principal strategies used in the creation of rootkits and their types. This article will cover the mechanisms that are used for their concealment as well as the techniques and tools used for their discovery.

Looking back at the most frequent types of rootkits we can find:

• Rootkits in Usermode: Those that are executed in Usermode (ring 3) and that modify library configuration files and system binaries.
• Rootkits in Kernelmode: those which are inserted in the kernel code during execution either directly (route/dev/kmem) or through dynamic modules. They are executed in the highest operating system privilege ring 0 to intercept calls to the system and to modify the behaviour of the kernel itself.

Rootkits in Usermode

Regarding rootkits in Usermode (ring 3), due to privilege limitations, the concealment mechanisms are not as effective as those used in Kernelmode (ring 0). Generally the strategies to remain hidden are based on using symbolic links, changes in the environment variables (LD_PRELOAD) and file alias, as well as the modification of binary, library and configurations (ld.so.preload, ld.so.cache, etc).

Although it is necessary to have a privileged user (root) to make these modifications, these exist in the Usermode and therefore these types of rootkits are relatively simple to detect.

By means of integrity checks (hashes of binary) or using statically compiled tools outside the system it is trying to analyse in order to verify differences with the results obtained in the execution of native binary. To conceal themselves, these rootkits normally manipulate the binaries used to show network connections (netstat, lsof, tcpdump, ip …), files (ls, lsof, stat …) or processes (ps, top ...).

In the following example we can see how to detect the existence of a rootkit:

An alternative method to detect rootkits in Usermode is to check file integrity. A tool such as aide can be used for this operation. This tool initially creates an inventory and obtains a hash for each system file which is then stored in a database. When a test is carried out results are compared with those stored in the database and any difference is reported. It is necessary to be completely certain that the initial system database is completely free from infection. It is advisable to carry out periodic tests, check the changes and update the database.

- Results of an integrity test using aide -

It is necessary to bear in mind that whenever there is suspicion of any type of intrusion, statically compiled tools and binaries external to the system should be used, in order to avoid any interception by dynamic libraries or interference by the rootkit. If these precautions are not taken the results can become manipulated.

Rootkits in kernelmode

In this type of rootkit the approach changes radically since it is the kernel itself which has been compromised, therefore it is not possible to detect in a simple way any anomalous behaviour. As described in the article Malware in Linux: Rootkits in kernelmode, these rootkits are designed to intercept communications between Usermode and Kernelmode through calls to the system or syscalls. A rootkit in kernelmode is able to effectively conceal itself by taking control of the calls to the system, which constitute the link between the kernel and the user.

Bearing in mind that the detection of rootkits in kernelmode will never be 100 % guaranteed, the strategies used will principally be:

• Monitoring of syscalls: observe changes in the address memory of the syscalls table. Compare the original System.map’s table of symbols with the current table. In any case, we have to take into account that this file also could have been modified.
• Monitoring changes in the table of interruptions (IDT): Many rootkits modify the interruption used in syscalls so that it has a modified table instead of the original call table.
• Discovery of hidden modules: compare the list of dynamic modules obtained through lsmod with the files and directories /proc/modules and /sys/module.
• Use tools designed to search for Rootkits. Generally these tools carry out verifications to discover discrepancies between the information obtained through system commands and the information available in /proc (unhide), as well as anomalies in syscalls (samhain, unhide). Searches are also carried out to find files or patterns that have already been discovered and documented on Rootkits (rkhunter, chkrookit)

- Execution flow of a system call -

Logically in a system with a compromised kernel it is difficult to carry out this task as it is necessary to deal with elements that are controlled at the maximum level of privileges. In the case of the detection or the suspicion of the existence of a rootkit in kernelmode it is highly recommended to reinstall the system.

It is possible to conclude that, whether it be a question of a rootkit in usermode or a rootkit in kernelmode we will always be faced with a difficult threat, and, depending on the importance of the compromised system, it requires a total reinstallation of the system to guarantee its complete elimination.

At the same time, when analyzing an intrusion it is possible to obtain information of great value to understand the safety deficiencies that allowed the compromise of the affected platform and to correct deficient or erroneous practices. One always learns more from one mistake than from being successful a hundred times.

- Rootkits in Usermode VS rootkits in Kernelmode-