Thursday, February 25, 2010

Per Socket Goodput

One of the holy grails of denial of service testing is the measurement of effective throughput of a service when under attack. There is no way to measure this in Linux because all the statistics are per-interface or system wide. What you actually need is to measure the number of bytes read or written by an open socket using a particular protocol (UDP or TCP). The new systemtap utility in Linux can do that, but how do you get the data out and into a MIB so you can read it using standard SNMP methods? There is a nice script supplied with systemtap called tcpipstat.stp, which outputs among other things bytes received and transmitted for each port, but it writes to the console.

I tried to capture the console output and read it into a c-program, which would enable me to port it into my MIB code. But try as I might you can't apparently read a file that is in the process of being written in C. You can in the underlying Linux OS but the bottom line is what the C library supports, and not all OSes allow it I guess.

After a bit of lateral thinking I realised that systemtap allows you to do other kinds of outputting than simple printf statements. In particular it allows system commands, as in C. So I can just call:

command = sprintf("snmpset -c public -v2c 
    localhost DOSTF-MIB::dosTFTCPRecvBytes.0 = %d",SockSendbytes[key])
system( command )

And I get the MIB variables set directly.

Monday, February 22, 2010

Building a debug kernel on Ubuntu

I thought I would write this down before I forget it. It is quite important because without it you can't use systemtap. You can download a debug build of the Ubuntu (Debian) kernel here. Unfortunately, only the latest kernels are published, and these may not be compatible with your software. Also it is big. To build one from scratch takes time - lots of it, so don't make any mistakes! Have a cup of tea while you are waiting, and throw in some biscuits and a chat. Or go for a walk. Reckon on a hour or two, depending on your processor and disk speed.

First you have to download the kernel source tree. Then you configure it, compile it, and finally install it. The instructions are here. I used the 'Old-fashioned Debian Way' because it works. The only caveats I would add are firstly that the existing .config file you copy from /boot/config-`uname -r` already has debugging turned on, even though the kernel you are using does not! Check in the 'make menuconfig' step that the "Compile the kernel with debug info" option under "Kernel hacking"->"Kernel debugging" is turned ON if not already so. Secondly, I found that setting the number of processors as advised in the above link may have speeded it up somewhat, although contention for the hard disk may have slowed it down. I tried both and didn't notice much difference.

The only other step is to install systemtap. I recommend below that you do this from source, until the Ubuntu people rebuild the default package.

Sunday, February 21, 2010

Building Systemtap on Ubuntu

How quickly software rots! Systemtap is a tool for monitoring the kernel in Linux. You write scripts in a C or javascript like language and use a debug version of your kernel, and the scripts get compiled into code and the code monitors the system in the way specified, a bit like a set of debug statements in an IDE.

So I wanted to use a script written by D Wilder at IBM (tcpipstat) for monitoring throughput on a per socket basis. You can't do this without systemtap or without hacking the kernel (harder). Unfortunately the script does not run using the version installed by the latest version of Ubuntu, Karmic Koala (9.10). Apparently missing are some 'tapsets', specifically tcpmib.stp and a few others. So, I thought, I would compile my own version of systemtap. So I downloaded the latest version using git from the repository and tried to compile that. It needed a version of elf-utils-dev, which is not available for Ubuntu. So I had to download the full source code of elfutils. Then I passed into the configure script the option --with-elfutils=/path/to/elfutils. (I still have no idea why they say configure with '--disable-pie' on Ubuntu).

Configure complained that 'missing', which is a script, had a bad format. Turned out to be the carriage returns were DOS not UNIX. So I downloaded flip and fixed that. Now configure worked but make failed with the obscure error 'configure.lineno xxx: bad substitution'. I had no idea and Googling made little difference. I decided I had had enough and downloaded a 'release' of systemtap, version 0.9.5, which I thought was the latest. Same problem. Then I found a post that hinted I should substitute /bin/bash for /bin/sh as the shell for the configure script. I did and it worked. But tcpmib.stp was not installed! It seems that this was introduced in 0.9.9, and I needed the latest code. I downloaded that and rebuilt and hey presto, it worked!

Version 1.1 in fact builds without a glitch. The problem was trying to use a version that was just 11 months old, though you do need elfutils.

Tuesday, February 9, 2010

Limiting CPU on Linux

Cpulimit is a tool that sends SIGSTOP and SIGCONT signals to a process to limit its percentage of CPU usage. That strikes me as being a bit heavy-handed. I was after something I could set in the kernel that would do it more elegantly. The problem is that for DDoS experiments the CPU gets so tied up handling the attack that there's not enough CPU oomph left to tell the monitoring program how it is going. So during the attack we just don't know what is happening.

So I decided to use nice values for the service under attack and a higher nice value for the monitoring program. That way, at least in theory, I should be able to continue monitoring. It doesn't seem possible to limit the service under attack to a fixed percentage of CPU.

Unfortunately, this doesn't work either. Although the query runs outside the service OK, as soon as you try to get a response out of the service under attack, even if you limit it to 5% of total CPU, it won't respond, because it is too busy.