Running an snmpd plugin

To run an snmpd plugin (shared object file) as part of the snmpd service just add the following line to the /etc/snmp/snmpd.conf file:

dlmod dosTFAgentPluginObject /home/desmond/.snmp/dostf-plugin.so

Or, replace the name of my plugin with yours. Then just restart the snmpd service:

sudo service snmpd restart

And Bob is your uncle. Hopefuly this will be a bit more robust than running it as a separate demon.

Addendum: it is more robust, and hasn't fallen over yet.

Extending Per Process Monitoring to Windows

Someone asked me how difficult it would be to get the same information as for Linux on Windows. The answer is: about the same.

Thread-count, cpu and memory usage can be measured via WMI (Microsoft's SNMP) either using the commandline tool WMIC or via C/C++ directly. The amount of required work here is small.

Response time can be measured using exactly the same technique as on Linux. The code just has to be checked so that it works also using the winsock API. Again, trivial.

Goodput is harder, but was also on Linux. You can write a 'shim', that emulates each call to winsock.dll and wsock32.dll. Then you instrument the shim to call snmpset to set the MIB directly for the calls you want to 'instrument'. I think this would take about a month for a moderately good programmer or enthusiastic beginner with a bit of instruction.

There is also the issue of how to integrate it into the SNMP service. Presumably the same techniques work as on Linux: I would be able to create a DLL or separate service that would link with the SNMP service containing the specialised code for monitoring processes.

Crashing of DosTF Demon

Another issue that came up was the failure of the dostf-demon, the program that attaches itself to the main snmp demon to extend its functionality. Under flood attack the communication between the two demons seems to break down and it fails. It says 'broken pipe'. It might actually be more robust if more inconvenient to run the dostf MIB as a DLL. This means that we would have to start and stop the main snmp service to make it work.

Pid of a commandline command

In order to find out anything about a process you need its pid. But how do you find that if all you have, typically, is the command that launched it or the name of the process? The problem is that if I launch a java program using java myprogram then the process listed in top and java -U <user> is "java". And if the system or someone else is running other java processes they will also be called "java". So pidof java or ps -C java -o pid= will retrieve many pids for "java" or only the latest one. It turns out there is a safe way to ascertain the pid of a process that was launched at the commandline. You can use pgrep, such that:

pgrep "java myprogram"

will print out the pid of that instance of java.

Addendum: A bit more complex than pgrep but just as effective is:

ps aux | awk '{if ($11 ~ /java myprogram/) {print $2}}'

Per Socket Goodput

One of the holy grails of denial of service testing is the measurement of effective throughput of a service when under attack. There is no way to measure this in Linux because all the statistics are per-interface or system wide. What you actually need is to measure the number of bytes read or written by an open socket using a particular protocol (UDP or TCP). The new systemtap utility in Linux can do that, but how do you get the data out and into a MIB so you can read it using standard SNMP methods? There is a nice script supplied with systemtap called tcpipstat.stp, which outputs among other things bytes received and transmitted for each port, but it writes to the console.

I tried to capture the console output and read it into a c-program, which would enable me to port it into my MIB code. But try as I might you can't apparently read a file that is in the process of being written in C. You can in the underlying Linux OS but the bottom line is what the C library supports, and not all OSes allow it I guess.

After a bit of lateral thinking I realised that systemtap allows you to do other kinds of outputting than simple printf statements. In particular it allows system commands, as in C. So I can just call:

command = sprintf("snmpset -c public -v2c 
    localhost DOSTF-MIB::dosTFTCPRecvBytes.0 = %d",SockSendbytes[key])
system( command )

And I get the MIB variables set directly.

Building a debug kernel on Ubuntu

I thought I would write this down before I forget it. It is quite important because without it you can't use systemtap. You can download a debug build of the Ubuntu (Debian) kernel here. Unfortunately, only the latest kernels are published, and these may not be compatible with your software. Also it is big. To build one from scratch takes time - lots of it, so don't make any mistakes! Have a cup of tea while you are waiting, and throw in some biscuits and a chat. Or go for a walk. Reckon on a hour or two, depending on your processor and disk speed.

First you have to download the kernel source tree. Then you configure it, compile it, and finally install it. The instructions are here. I used the 'Old-fashioned Debian Way' because it works. The only caveats I would add are firstly that the existing .config file you copy from /boot/config-`uname -r` already has debugging turned on, even though the kernel you are using does not! Check in the 'make menuconfig' step that the "Compile the kernel with debug info" option under "Kernel hacking"->"Kernel debugging" is turned ON if not already so. Secondly, I found that setting the number of processors as advised in the above link may have speeded it up somewhat, although contention for the hard disk may have slowed it down. I tried both and didn't notice much difference.

The only other step is to install systemtap. I recommend below that you do this from source, until the Ubuntu people rebuild the default package.

Building Systemtap on Ubuntu

How quickly software rots! Systemtap is a tool for monitoring the kernel in Linux. You write scripts in a C or javascript like language and use a debug version of your kernel, and the scripts get compiled into code and the code monitors the system in the way specified, a bit like a set of debug statements in an IDE.

So I wanted to use a script written by D Wilder at IBM (tcpipstat) for monitoring throughput on a per socket basis. You can't do this without systemtap or without hacking the kernel (harder). Unfortunately the script does not run using the version installed by the latest version of Ubuntu, Karmic Koala (9.10). Apparently missing are some 'tapsets', specifically tcpmib.stp and a few others. So, I thought, I would compile my own version of systemtap. So I downloaded the latest version using git from the repository and tried to compile that. It needed a version of elf-utils-dev, which is not available for Ubuntu. So I had to download the full source code of elfutils. Then I passed into the configure script the option --with-elfutils=/path/to/elfutils. (I still have no idea why they say configure with '--disable-pie' on Ubuntu).

Configure complained that 'missing', which is a script, had a bad format. Turned out to be the carriage returns were DOS not UNIX. So I downloaded flip and fixed that. Now configure worked but make failed with the obscure error 'configure.lineno xxx: bad substitution'. I had no idea and Googling made little difference. I decided I had had enough and downloaded a 'release' of systemtap, version 0.9.5, which I thought was the latest. Same problem. Then I found a post that hinted I should substitute /bin/bash for /bin/sh as the shell for the configure script. I did and it worked. But tcpmib.stp was not installed! It seems that this was introduced in 0.9.9, and I needed the latest code. I downloaded that and rebuilt and hey presto, it worked!

Version 1.1 in fact builds without a glitch. The problem was trying to use a version that was just 11 months old, though you do need elfutils.

Limiting CPU on Linux

Cpulimit is a tool that sends SIGSTOP and SIGCONT signals to a process to limit its percentage of CPU usage. That strikes me as being a bit heavy-handed. I was after something I could set in the kernel that would do it more elegantly. The problem is that for DDoS experiments the CPU gets so tied up handling the attack that there's not enough CPU oomph left to tell the monitoring program how it is going. So during the attack we just don't know what is happening.

So I decided to use nice values for the service under attack and a higher nice value for the monitoring program. That way, at least in theory, I should be able to continue monitoring. It doesn't seem possible to limit the service under attack to a fixed percentage of CPU.

Unfortunately, this doesn't work either. Although the query runs outside the service OK, as soon as you try to get a response out of the service under attack, even if you limit it to 5% of total CPU, it won't respond, because it is too busy.