Investigation of hacked Linux Servers

Here you find some hints and commands to use in case of a compromised server or suspicious processes. They should help to get a clearer picture of the situation and to act in a structured way.

1. Do thing slowly, remain calm

2. Document precisely everything you do

3. If you are in a company it makes sense to inform the management about the situation

4. Decide if you a) do a live investigation of the server or b) immediately switch off the server (directly unplug electric cable, don't shut down) and investigate the hard disk later.
Both has pros and cons. In the latter case you lose the information from the RAM like process table, sockets etc. In the first case you risk that a potential attacker notices you. He might start to erase his tracks immediately, disappear or even delete data. But it is extremely unlikely to directly meet a hacker at the command line. What you find on servers are mostly scripts running. Therefor I personally tend to method a).

Possibilities to save the content of the main memory (RAM):

With /dev/mem and netcat:
Log host:

netcat -v -l -p 9000 | dd of=ram.dd

Compromised host:

dd if=/dev/mem | netcat IP_LOG_HOST 9000

But the whole thing is not so significant as the content of /dev/mem could have been manipulated.

A way to save the RAM of non-virtualised systems is msramdumper. But this has to be prepared in advance on an USB stick. And the system must be bootable via USB.

In case of virtualised systems there is a third possiblity. The RAM can be saved from the physical host.
With XEN it could look like this:

xm save VM1 VM1.chk

a) Live Investigation

Here the goal is to examine the whole system regarding different parameters. These are hard disk space, system uptime, cpu usage, memory consumption, network load, mail traffic, etc.

It is a good idea to make a copy of all output of the following commands.

Overview about the hard disk space.

df -h

What is the condition and configuration of the network interfaces?

ip a

What is the content of the routing table?

route -n

What is mounted?

mount

Which kernel modules have been loaded?

lsmod

Print the running processes, the process table.

ps gauxwww

Sometimes the names of processes are camouflaged with square brackets. You may then overlook them because you think they are kernel processes. Special attention is needed for every unusual process running with the user of the web server. Also you should check if important processes DON'T run, like the system logging daemon.

Find an executable binary which belongs to a process ID (PID).

cat /proc/PID/cmdline

Print the memory map of a process in the extended format.

pmap -x PID

... and in the device format

pmap -d PID

Overview about sockets and ports.

netstat -patune

A typical example here is the port of an IRC client: 6667. It is also a good idea to scan the server from the outside with nmap for example. The reason is to find out which ports are really open.

Examine the system calls of a running process.

strace -p PID

Display open file handles of a process.

lsof -p PID

Find out who is accessing a suspicious file or a partition.

fuser -v FILE
fuser -m MOUNTED_PARTITION

check wether a network interface is in promiscuous mode. This may be used to listen to network traffic.

dmesg

Look in the output for "device entered promiscuous mode".

Monitor the network traffic.

tcpdump -i ethX

Check the system for rootkits or manipulated system files with rkhunter. It is important to first download the most actual data.

rkhunter --update
rkhunter --checkall

If you want to block a certain IP from outside you can use iptables.

iptables -A INPUT -s IPADDRESS -j DROP

b) Later examination of the hard disk

The partitions of the hard disk of the hacked system should only be mounted read-only. Then make a copy and work only with the copy.

mount -o ro [-t ext3] /dev/xyz /mnt/xyz

Show the latest access of a file.

stat FILE

Look for access data or other strings in suspicious binaries.

strings -a BINARY

In generall all available log files must be examined, like logs of the firewall, mod_security, web server, mail server, IDS, etc. You may find there suspicous records or strings. Unfortunately this has to be done up to several months before you noticed the intruder. Because often an attacker waits a long time from the moment he has access to a machine until he really uses it.

Especially the web server log files should be examined very carefully. Strings like "wget", "fetch", "shell", "=http://" indicate an attempt to attack the server.

Also the crash or hanging of applications can be a warning signal which may be visible in the logs.

Is the free disk space according to your expectations?

Scripts of hackers are often stored in /tmp or /var/tmp. In these folders you should look for script code. Sometimes these files contain source code in an unreadable format. you see source code but you can hardly find out what it does because it has been obfuscated. Variables and function names have confusing names or are just numbers. Line breaks and indentions have been removed.

Furthermore all uploaded images and files must be examined if they contain source code. Some hackers upload images but in fact they are script files. Also look at all files which have strange names and/or are hidden.

All files which belong to the content of the web server should be investigated for phishing content. You can do a grep for strings like "paypal", "ebay" etc.

Ideally you had a tool like Aide running before. In this case you can check the system if important system binaries have been manipulated like ps or top.

The files /etc/passwd, /etc/shadow and /etc/group should not contain unusual, new or missing records.

If you have found a malignat file you may want to find out what it does exactly. The last possibility for this is to use nm(Binutils) to read the symbol tables and objdump to disassemble the code. But this is far beyond the scope of everyday life because you need so much know-how and time that it's probably not worth trying.

Apache hardening
IDS/IPS systems with Snort
SELinux Training