Building Systemtap on Ubuntu

How quickly software rots! Systemtap is a tool for monitoring the kernel in Linux. You write scripts in a C or javascript like language and use a debug version of your kernel, and the scripts get compiled into code and the code monitors the system in the way specified, a bit like a set of debug statements in an IDE.

So I wanted to use a script written by D Wilder at IBM (tcpipstat) for monitoring throughput on a per socket basis. You can't do this without systemtap or without hacking the kernel (harder). Unfortunately the script does not run using the version installed by the latest version of Ubuntu, Karmic Koala (9.10). Apparently missing are some 'tapsets', specifically tcpmib.stp and a few others. So, I thought, I would compile my own version of systemtap. So I downloaded the latest version using git from the repository and tried to compile that. It needed a version of elf-utils-dev, which is not available for Ubuntu. So I had to download the full source code of elfutils. Then I passed into the configure script the option --with-elfutils=/path/to/elfutils. (I still have no idea why they say configure with '--disable-pie' on Ubuntu).

Configure complained that 'missing', which is a script, had a bad format. Turned out to be the carriage returns were DOS not UNIX. So I downloaded flip and fixed that. Now configure worked but make failed with the obscure error 'configure.lineno xxx: bad substitution'. I had no idea and Googling made little difference. I decided I had had enough and downloaded a 'release' of systemtap, version 0.9.5, which I thought was the latest. Same problem. Then I found a post that hinted I should substitute /bin/bash for /bin/sh as the shell for the configure script. I did and it worked. But tcpmib.stp was not installed! It seems that this was introduced in 0.9.9, and I needed the latest code. I downloaded that and rebuilt and hey presto, it worked!

Version 1.1 in fact builds without a glitch. The problem was trying to use a version that was just 11 months old, though you do need elfutils.

Limiting CPU on Linux

Cpulimit is a tool that sends SIGSTOP and SIGCONT signals to a process to limit its percentage of CPU usage. That strikes me as being a bit heavy-handed. I was after something I could set in the kernel that would do it more elegantly. The problem is that for DDoS experiments the CPU gets so tied up handling the attack that there's not enough CPU oomph left to tell the monitoring program how it is going. So during the attack we just don't know what is happening.

So I decided to use nice values for the service under attack and a higher nice value for the monitoring program. That way, at least in theory, I should be able to continue monitoring. It doesn't seem possible to limit the service under attack to a fixed percentage of CPU.

Unfortunately, this doesn't work either. Although the query runs outside the service OK, as soon as you try to get a response out of the service under attack, even if you limit it to 5% of total CPU, it won't respond, because it is too busy.

DNS status request

OK, so there is a general query you can make of a DNS service. It's called a status request. Type 0 requests are standard queries, requests to ask what is the ip-address of "www.somecompany.com". Type 1 requests are reverse lookups such as what is the domain-name of "131.181.6.130"? Type 2 requests are "What is your current status?" That's just what we want, because it works with any DNS server, and it doesn't require any actual lookups, which might get delegated to other servers.

A DNS status query is easy to construct, but it has to be sent via UDP. So far all the challenges I have been sending are via TCP. So it wasn't responding, even though the DNS server is supposed to be listening on UDP and TCP.

The challenge turns out to be just:

\25\14\20\0\0\0\0\0\0\0\0\0

The first two bytes are insignificant, just a unique ID generated by the client. The next byte holds the status request command, and everything else has to be 0.

Generating reverse dns queries

An obvious way of getting a response time out of any dns server is to ask it to perform a reverse dns query on itself. The format of a dns query is documented elsewhere, and a reverse dns query can be generated, captured using tcpdump and analysed in wireshark easily enough. The trick is to generate the dns query programmatically since a fixed string won't work equally well for all dns servers. What we don't want to do is ask a server for information it must get from a remote service. This will vary too much, whereas the objective is to measure the responsiveness of the service itself.

It might be possible to make an irregular query, one that doesn't require it to look up anything. That way the string could be fixed. I'll investigate on Wednesday.

telnet and similar services on Ubuntu

I wanted to test a telnet service on Ubuntu. Telnet is an insecure terminal service that has largely been superseded by ssh. However, I wanted to add it to my collection of services I could measure responsiveness on. But, boy was that a lot harder than I thought! Ubuntu already installs a telnet client for you. But in order to get the telnet service running you have to install inetd, the super-service, which then spawns telnet when you ask for it, based on the inetd.conf file in /etc/inetd.conf. In fact the inetd 'superserver' spawns other services too, not just telnet. So it will be needed for lots of other services I want to test. Cool, so I entered a line in inetd.conf thus:

telnet  stream  tcp  nowait  root  /usr/sbin/telnetd  telnetd

The fields are supposed to represent service-name, socket-type, protocol name, wait/nowait, username, serverpath server-args. Now if I run the inetd service I can indeed log into localhost using telnet, but not from another machine on another interface. Furthermore, there is no way to specify that the telnetd use another interface. After trying everything I could think of the solution was quite simple: install xinetd instead of old inetd, and hey presto, it all worked.

Stress testing pf versus iptables

We wanted to know whether pf or iptables was faster at implementing a white or blacklist of ip-addresses. How can you generate packets on one computer pretending to be from a wide range of ip-addresses (i.e. spoofed addresses) and then detect and measure the receipt of them on another computer? A third machine running pf or iptables would need to be in the middle to act as a router/firewall. In fact generating spoofed packets can be achieved with a number of tools. We chose hping3, since it does this but also has a listen function that examines packets with a signature and then prints out the packet contents after this signature. So if we send 10,000 spoofed packets as fast as possible from machine A to machine C via machine B (running pf/iptables) then they should all be dropped if the spoofed address is on the black list. Each such lookup in the black list on machine B will take some CPU time, and the effectiveness of that lookup in pf versus iptables is what we were hoping to measure. The only remaining problem was how to generate a mixture of black/white-listed addresses and others that were definitely not on the list. The lists can be generated by a simple perl script, and by adjusting it we managed to also generate a shell script containing a long series of hping3 commands. We then set up hping3 on machine C and pipe the resulting output to a file. Counting the number of words in the file tells us how many packets made it through the firewall. If the flood of packets overwhelmed the router as we hoped then more will be dropped by the more inefficient firewall.

Computing Number of Threads for a Process

Someone suggested that a good way to measure DDoS might be to monitor the number of threads in a process. Under attack, the process of a service might react by creating more threads to handle the new connections. But, how to do it? Someone suggested that a command like:

ps uH p `ps -C <appname> -o pid=` | wc -l

would do it, where '<appname>' is the name of the process being queried. The bit inside the backticks merely retrieves the pid of the named process. It doesn't have to be the precise name – but it does have to be a unique substring of that process's name. The 'wc -l' bit counts the number of lines in the output of ps, which corresponds to the number of threads. Using the same technique as for the current CPU and memory usage it is easy to retrieve the current number of threads from any process. First set the current app name:

snmpset -v2c -c public localhost DOSTF-MIB::dosTFCurrentAppName.0 = firefox

Then, issue an snmpget on the number of threads for the current application:

snmpget -v2c -c public localhost DOSTF-MIB::dosTFCurrentAppThreadCount.0

And you should get the number of threads printed out:

DOSTF-MIB::dosTFCurrentAppThreadCount.0 = INTEGER: 13