Sunday, November 7, 2010

Getting/Setting the netmask in Linux/BSD

Recently I have been wrestling with a method for generating multiple aliases for one interface, as in the last post. Part of this was finding suitable ip-addresses to act as aliases. My first idea was to choose free ip-addresses on the local subnet, depending on how big it was, then pinging all possible addresses to find out which ones were already taken. This turned out much more complex than I had thought, unreliable and slow.

Then I had the idea of just expanding the interface's netmask up a level, ignoring all the addresses of the existing subnet without testing them, and generating alias ips that couldn't possibly conflict with any already on the network. To do that I needed to get and set the current netmask programmatically. I didn't find much on the setting (you won't find it in Steven's book, for example). So I wrote my own two routines get_netmask and set_netmask and put them into an example program that even works on BSD:

#include <stdio.h>
#include <stdlib.h>
#include <sys/types.h>
#include <sys/socket.h>
#include <sys/ioctl.h>
#include <net/if.h>
#include <arpa/inet.h>
#include <string.h>

/**
 * @param skfd an open socket
 * @param intf the interface name, e.g. eth0
 * @return the netmask as a string
 */
char *get_netmask( int skfd, char *intf )
{
    struct ifreq ifr;
    ifr.ifr_addr.sa_family = AF_INET;
    strncpy( ifr.ifr_name, intf, IFNAMSIZ-1 );
    if ( ioctl(skfd,SIOCGIFNETMASK,&ifr) == -1 )
    {
        printf("could not read interface %s\n",intf);
        exit( 0 );
    }
    /* display result */
    return inet_ntoa(((struct sockaddr_in *)&ifr.ifr_addr)->sin_addr);
}
/**
 * @param skfd an open socket
 * @param intf the interface name, e.g. eth0
 * @param newmask the new mask a a string e.g. "255.255.0.0"
 */
void set_netmask( int skfd, char *intf, char *newmask )
{
    struct ifreq ifr;
    unsigned int dst;
    struct sockaddr_in *sin = (struct sockaddr_in *)&ifr.ifr_addr;
    memset(&ifr, 0, sizeof(ifr));
    sin->sin_family = AF_INET;
    if ( !inet_pton(AF_INET, newmask, &sin->sin_addr) )
    {
        printf("failed to convert netmask\n");
        exit( 0 );
    }
    strncpy( ifr.ifr_name, intf, IFNAMSIZ-1 );
    if ( ioctl(skfd,SIOCSIFNETMASK,&ifr) == -1 )
    {
        printf("could not read interface %s\n",intf);
        exit( 0 );
    }
}
/**
 * main entry routine
 * accepts 2 arguments: the interface name and the new netmask
 * e.g. ./intf eth0 255.255.0.0
 */
int main( int argc, char **argv )
{
    char *str;
    int skfd = socket( AF_INET, SOCK_DGRAM, 0);
    if ( skfd == -1 )
    {
        printf("Couldn't open socket\n");
        exit(0);
    }
    printf("netmask was: %s\n", get_netmask(skfd,argv[1]) );
    set_netmask( skfd, argv[1], argv[2] );
    printf("netmask is now: %s\n", get_netmask(skfd,argv[1]) );
    close( skfd );
    return 0;
}

Friday, October 8, 2010

Programmatic interface alias creation in Linux

In *nix you can create an alias for an interface. So for my ethernet card residing at eth0, whose address is normally 192.168.100.4, I can also create a large number of aliases, for example, 192.168.100.45 for interface eth0:45. I have to give it a unique name not more than 16 bytes long, or it won't be able to distinguish it from my normal interface. Now why would I want to do that? Because, in my case I want to define hundreds or thousands of aliased addresses on my machine, so I can bind on to each of them and send ip-packets from those aliased addresses to a server. That way, I can simulate a DDoS attack of 200 or 5000 machines with just one machine. That's cool. But how do you do it?

You're supposed to use ifconfig, a commandline tool, but since I needed speed I wanted to be able to do it programmatically. How? On BSD you can call ioctl with SIOCAIFADDR to set the ip-address of an interface. But on Linux you have to use SIOCSIFADDR instead. Just give it a unique name and call ioctl with that argument and you're done. Since I didn't see any sample code for this when I Googled, I thought I'd save other people's time by putting this up. It copies the default netmask and broadcast address from the default interface, so you don't have to set these. You'll need to be supervisor to execute it.

#include <stdio.h>
#include <string.h>
#include <sys/socket.h>
#include <sys/ioctl.h>
#include <netinet/in.h>
#include <net/if.h>
#include <arpa/inet.h>
#include <errno.h>
#include <stdlib.h>

int skfd;
int set_ip_using(const char *name, int c, unsigned long ip)
{
    struct ifreq ifr;
    struct sockaddr_in sin;

    strncpy(ifr.ifr_name, name, IFNAMSIZ);
    memset(&sin, 0, sizeof(struct sockaddr));
    sin.sin_family = AF_INET;
    sin.sin_addr.s_addr = ip;
    memcpy(&ifr.ifr_addr, &sin, sizeof(struct sockaddr));
    if (ioctl(skfd, c, &ifr) < 0)
 {
  printf("failed to set ip-address\n");
  exit(0);
    }
 return 0;
}
int main ( int argc, char **argv )
{
 char *ip_address=NULL;
 char *interface;
 if ( argc==3 )
 {
  struct sockaddr_in sin;
  unsigned long ip;
  ip_address=argv[2];
  interface = argv[1];
  skfd = socket( AF_INET, SOCK_DGRAM, 0);
  if ( skfd == -1 )
  {
   printf("Couldn't open socket\n");
   exit(0);
  }
  inet_pton( AF_INET, ip_address, &sin );
  memcpy(&ip, &sin.sin_addr.s_addr, sizeof(unsigned long));  
  set_ip_using( interface,SIOCSIFADDR,ip );
  close( skfd );
 }
 else
  printf("usage: setip <interface-name> <ip-address>\n");
}

Tuesday, September 28, 2010

Two-way dynamic linking in C

Building a dynamic library (or shared object lib) on Linux is easy. All you do is compile your code with the -fPIC or -fpic flags then use the -shared flag for the link-step:

gcc -fPIC foo.c -shared -o foo.so

Then you can load the dynamic library in your main program by using dl_open and dl_sym to get any global symbol from the library.

But what if you also want your shared object library to link with symbols in the main program? This might be necessary, for example, if the dynamic module wanted to obtain a key from the main program. It would have to call a routine in main just as main is calling routines in the library. In this case you must instruct the main program also to export its symbols during linking:

gcc main.c -ldl -rdynamic -o main

Then, main, when it loads the shared object lib foo, will enable both foo to call routines in main, and main to call routines in foo.

Here's main.c. You can use the last command above to compile it

#include <dlfcn.h>
#include <stdio.h>
int bar()
{
 printf("dynamic lib called bar\n");
 return 0;
}
int main( int argc, char **argv )
{
 void *handle;
 handle = dlopen( "./foo.so", RTLD_GLOBAL|RTLD_LAZY );
 if ( handle != NULL )
 {
  int (*r)(void);
  int res;
  *(void **)(&r) = dlsym(handle, "foo");
  res = (*r)();
  printf("foo returned %d\n",res );
 }
 else
  printf("failed to open foo.so\n");
}

And foo.c. This compiles with the first command above:

#include <stdio.h>
extern int bar();
int foo()
{
 printf("called foo inside lib\n");
 bar();
 return 0;
}

Then run it by calling main:

./main
called foo inside lib
dynamic lib called bar
foo returned 0

Wednesday, May 19, 2010

Capturing packets from an Interface in a MIB

I had to use pcap to capture packets arriving at a particular interface. For that the snmp daemon needs to run as root. You can configure the options for starting up the snmpd program in /etc/default/snmpd. What's not cool, though, is trying to start it up and shut it down. It seems that nanosleep on Linux doesn't work, or rather it only works with a resolution of 1 second. No matter how many nanoseconds you specify nanosleep doesn't sleep until you fill in the tv_sec field. This is, however, more annoying than a show-stopper. I can now start packet capture via pcap_loop and kill it with pcap_breakloop. Stopping takes a few seconds but mostly works. So now we have a MIB that actually detects DDoS attacks!!!

Thursday, May 13, 2010

Average CPU usage for a terminating process

Someone asked me if we can easily compute the average CPU usage of a particular program over its lifetime when the program is scheduled to terminate at some point in the future. At that time it should print out the result. Here's what I came up with:

#!/bin/bash
# get average CPU for a process
total=0
times=0
pid=`pgrep "$1"`
while [ -n "$pid" ]; do
temp=`ps -eo pid,pcpu | grep $pid | awk '{print $2}'`
total=`echo "scale=2;$total+$temp" | bc`
times=$[$times+1]
sleep 1
pid=`pgrep "$1"`
done
average=`echo "scale=2;$total/$times" | bc`
echo "Average cpu usage for $1 is: $average"

To run it give it an argument, which should be the name of the process or part or all of the command that launched it, and background the process, e.g.:

./perprocesscpu.sh "java myprogram" &

So, when "java myprogram" terminates, this script tells me its average CPU usage.

Tuesday, April 27, 2010

Reading from the commandline in C

What's the best way to read the output from a commandline command in C? If you use the int system(char*) function you receive back a return code, not the output. Not much use. To receive the output you could call freopen on STDOUT, then read from the file you redirected it to. But when you close the file (which you must), STDOUT is left in a broken state. There has to be a better way, and there is. You use popen, instead of system. Then, the output of the command is supplied via the FILE handle returned by popen. Here's an example that finds out the memory usage of the firefox process under Linux:


#include <stdio.h>
#include <string.h>
static char sysCmd[256];
/*
 * Issue a system command to get memory usage
 * @return the percent value times 100
 */
int get_memory()
{
    int res = 0;
    float percent;
    sysCmd[0] = 0;
    strcat( sysCmd, "ps aux | awk '{if ($11 ~ /" );
    strcat( sysCmd, "firefox" );
    strcat( sysCmd, "/&& !/awk/) {print $4}}'" );
    FILE *CONSOLE = popen( sysCmd, "r" );
    if ( CONSOLE != NULL )
    {
        res = fscanf( CONSOLE, "%f", &percent );
        if ( res <=0 )
            syslog(LOG_ALERT,"Failed to parse result of %s\n",sysCmd);
        pclose( CONSOLE );
        res = (int)(percent * 100.0f);
    }
    else
        syslog(LOG_ALERT,"Failed to open pipe for %s\n",sysCmd);
    return res;
}

Monday, April 12, 2010

Adding or removing portlets from a page programmatically in Pluto

What I wanted to do in the dosTF portal was to add or remove portlets, which in my case were monitoring applets, in response to the contents of the loaded scenario. If an experiment called for monitoring of a particular SNMP parameter, I wanted to be able to automatically add the monitoring portlet and configure it to monitor the correct parameter on the correct target. But it seems that Pluto is designed rather simply to prevent that. You are supposed to add portlets in the GUI, which kind of defeats the whole purpose of saving the experiment in an XML file. I needed to restore the state of the experiment from the XML and to do that I needed to add or remove portlets from the page programmatically.

Pluto has an undocumented way to do this. There is the PageConfig class used by the Pluto Page Admin portlet. It gets the correct instance of this class from the portlet context:

PortletContext pc = getPortletContext();
DriverConfiguration driverConfig = 
    (DriverConfiguration) pc.getAttribute("driverConfig");
PageConfig config = driverConfig.getPageConfig( "About Apache Pluto" );

The problem with this code is that only the Pluto application has access to the relevant portlet context. You can't get access to it from another application containing another portlet. So, to get around this, I simply copied my ScenarioEditor portlet into the pluto application and hey presto, it works! Removing a portlet is a bit harder. You have to get the portlet ids on the page from the PageConfig and then note the differences before and after. Then record the portlet id of the newly added portlet. It would make more sense if addPortlet just returned the portlet id.

However, configuring the portlet before display seems to be hard with this method. Maybe there is some way but PageConfig doesn't seem to have any knowledge of the 'portlet' itself, just its portlet id. But I'll leave that problem for the next post.

Sunday, March 7, 2010

Running an snmpd plugin

To run an snmpd plugin (shared object file) as part of the snmpd service just add the following line to the /etc/snmp/snmpd.conf file:

dlmod dosTFAgentPluginObject /home/desmond/.snmp/dostf-plugin.so

Or, replace the name of my plugin with yours. Then just restart the snmpd service:

sudo service snmpd restart

And Bob is your uncle. Hopefuly this will be a bit more robust than running it as a separate demon.

Addendum: it is more robust, and hasn't fallen over yet.

Thursday, March 4, 2010

Extending Per Process Monitoring to Windows

Someone asked me how difficult it would be to get the same information as for Linux on Windows. The answer is: about the same.

Thread-count, cpu and memory usage can be measured via WMI (Microsoft's SNMP) either using the commandline tool WMIC or via C/C++ directly. The amount of required work here is small.

Response time can be measured using exactly the same technique as on Linux. The code just has to be checked so that it works also using the winsock API. Again, trivial.

Goodput is harder, but was also on Linux. You can write a 'shim', that emulates each call to winsock.dll and wsock32.dll. Then you instrument the shim to call snmpset to set the MIB directly for the calls you want to 'instrument'. I think this would take about a month for a moderately good programmer or enthusiastic beginner with a bit of instruction.

There is also the issue of how to integrate it into the SNMP service. Presumably the same techniques work as on Linux: I would be able to create a DLL or separate service that would link with the SNMP service containing the specialised code for monitoring processes.

Crashing of DosTF Demon

Another issue that came up was the failure of the dostf-demon, the program that attaches itself to the main snmp demon to extend its functionality. Under flood attack the communication between the two demons seems to break down and it fails. It says 'broken pipe'. It might actually be more robust if more inconvenient to run the dostf MIB as a DLL. This means that we would have to start and stop the main snmp service to make it work.

Wednesday, March 3, 2010

Pid of a commandline command

In order to find out anything about a process you need its pid. But how do you find that if all you have, typically, is the command that launched it or the name of the process? The problem is that if I launch a java program using java myprogram then the process listed in top and java -U <user> is "java". And if the system or someone else is running other java processes they will also be called "java". So pidof java or ps -C java -o pid= will retrieve many pids for "java" or only the latest one. It turns out there is a safe way to ascertain the pid of a process that was launched at the commandline. You can use pgrep, such that:

pgrep "java myprogram"
will print out the pid of that instance of java.

Addendum: A bit more complex than pgrep but just as effective is:

ps aux | awk '{if ($11 ~ /java myprogram/) {print $2}}'

Thursday, February 25, 2010

Per Socket Goodput

One of the holy grails of denial of service testing is the measurement of effective throughput of a service when under attack. There is no way to measure this in Linux because all the statistics are per-interface or system wide. What you actually need is to measure the number of bytes read or written by an open socket using a particular protocol (UDP or TCP). The new systemtap utility in Linux can do that, but how do you get the data out and into a MIB so you can read it using standard SNMP methods? There is a nice script supplied with systemtap called tcpipstat.stp, which outputs among other things bytes received and transmitted for each port, but it writes to the console.

I tried to capture the console output and read it into a c-program, which would enable me to port it into my MIB code. But try as I might you can't apparently read a file that is in the process of being written in C. You can in the underlying Linux OS but the bottom line is what the C library supports, and not all OSes allow it I guess.

After a bit of lateral thinking I realised that systemtap allows you to do other kinds of outputting than simple printf statements. In particular it allows system commands, as in C. So I can just call:

command = sprintf("snmpset -c public -v2c 
    localhost DOSTF-MIB::dosTFTCPRecvBytes.0 = %d",SockSendbytes[key])
system( command )

And I get the MIB variables set directly.

Monday, February 22, 2010

Building a debug kernel on Ubuntu

I thought I would write this down before I forget it. It is quite important because without it you can't use systemtap. You can download a debug build of the Ubuntu (Debian) kernel here. Unfortunately, only the latest kernels are published, and these may not be compatible with your software. Also it is big. To build one from scratch takes time - lots of it, so don't make any mistakes! Have a cup of tea while you are waiting, and throw in some biscuits and a chat. Or go for a walk. Reckon on a hour or two, depending on your processor and disk speed.

First you have to download the kernel source tree. Then you configure it, compile it, and finally install it. The instructions are here. I used the 'Old-fashioned Debian Way' because it works. The only caveats I would add are firstly that the existing .config file you copy from /boot/config-`uname -r` already has debugging turned on, even though the kernel you are using does not! Check in the 'make menuconfig' step that the "Compile the kernel with debug info" option under "Kernel hacking"->"Kernel debugging" is turned ON if not already so. Secondly, I found that setting the number of processors as advised in the above link may have speeded it up somewhat, although contention for the hard disk may have slowed it down. I tried both and didn't notice much difference.

The only other step is to install systemtap. I recommend below that you do this from source, until the Ubuntu people rebuild the default package.

Sunday, February 21, 2010

Building Systemtap on Ubuntu

How quickly software rots! Systemtap is a tool for monitoring the kernel in Linux. You write scripts in a C or javascript like language and use a debug version of your kernel, and the scripts get compiled into code and the code monitors the system in the way specified, a bit like a set of debug statements in an IDE.

So I wanted to use a script written by D Wilder at IBM (tcpipstat) for monitoring throughput on a per socket basis. You can't do this without systemtap or without hacking the kernel (harder). Unfortunately the script does not run using the version installed by the latest version of Ubuntu, Karmic Koala (9.10). Apparently missing are some 'tapsets', specifically tcpmib.stp and a few others. So, I thought, I would compile my own version of systemtap. So I downloaded the latest version using git from the repository and tried to compile that. It needed a version of elf-utils-dev, which is not available for Ubuntu. So I had to download the full source code of elfutils. Then I passed into the configure script the option --with-elfutils=/path/to/elfutils. (I still have no idea why they say configure with '--disable-pie' on Ubuntu).

Configure complained that 'missing', which is a script, had a bad format. Turned out to be the carriage returns were DOS not UNIX. So I downloaded flip and fixed that. Now configure worked but make failed with the obscure error 'configure.lineno xxx: bad substitution'. I had no idea and Googling made little difference. I decided I had had enough and downloaded a 'release' of systemtap, version 0.9.5, which I thought was the latest. Same problem. Then I found a post that hinted I should substitute /bin/bash for /bin/sh as the shell for the configure script. I did and it worked. But tcpmib.stp was not installed! It seems that this was introduced in 0.9.9, and I needed the latest code. I downloaded that and rebuilt and hey presto, it worked!

Version 1.1 in fact builds without a glitch. The problem was trying to use a version that was just 11 months old, though you do need elfutils.

Tuesday, February 9, 2010

Limiting CPU on Linux

Cpulimit is a tool that sends SIGSTOP and SIGCONT signals to a process to limit its percentage of CPU usage. That strikes me as being a bit heavy-handed. I was after something I could set in the kernel that would do it more elegantly. The problem is that for DDoS experiments the CPU gets so tied up handling the attack that there's not enough CPU oomph left to tell the monitoring program how it is going. So during the attack we just don't know what is happening.

So I decided to use nice values for the service under attack and a higher nice value for the monitoring program. That way, at least in theory, I should be able to continue monitoring. It doesn't seem possible to limit the service under attack to a fixed percentage of CPU.

Unfortunately, this doesn't work either. Although the query runs outside the service OK, as soon as you try to get a response out of the service under attack, even if you limit it to 5% of total CPU, it won't respond, because it is too busy.

Tuesday, January 26, 2010

DNS status request

OK, so there is a general query you can make of a DNS service. It's called a status request. Type 0 requests are standard queries, requests to ask what is the ip-address of "www.somecompany.com". Type 1 requests are reverse lookups such as what is the domain-name of "131.181.6.130"? Type 2 requests are "What is your current status?" That's just what we want, because it works with any DNS server, and it doesn't require any actual lookups, which might get delegated to other servers.

A DNS status query is easy to construct, but it has to be sent via UDP. So far all the challenges I have been sending are via TCP. So it wasn't responding, even though the DNS server is supposed to be listening on UDP and TCP.

The challenge turns out to be just:

\25\14\20\0\0\0\0\0\0\0\0\0

The first two bytes are insignificant, just a unique ID generated by the client. The next byte holds the status request command, and everything else has to be 0.

Sunday, January 24, 2010

Generating reverse dns queries

An obvious way of getting a response time out of any dns server is to ask it to perform a reverse dns query on itself. The format of a dns query is documented elsewhere, and a reverse dns query can be generated, captured using tcpdump and analysed in wireshark easily enough. The trick is to generate the dns query programmatically since a fixed string won't work equally well for all dns servers. What we don't want to do is ask a server for information it must get from a remote service. This will vary too much, whereas the objective is to measure the responsiveness of the service itself.

It might be possible to make an irregular query, one that doesn't require it to look up anything. That way the string could be fixed. I'll investigate on Wednesday.

telnet and similar services on Ubuntu

I wanted to test a telnet service on Ubuntu. Telnet is an insecure terminal service that has largely been superseded by ssh. However, I wanted to add it to my collection of services I could measure responsiveness on. But, boy was that a lot harder than I thought! Ubuntu already installs a telnet client for you. But in order to get the telnet service running you have to install inetd, the super-service, which then spawns telnet when you ask for it, based on the inetd.conf file in /etc/inetd.conf. In fact the inetd 'superserver' spawns other services too, not just telnet. So it will be needed for lots of other services I want to test. Cool, so I entered a line in inetd.conf thus:

telnet  stream  tcp  nowait  root  /usr/sbin/telnetd  telnetd

The fields are supposed to represent service-name, socket-type, protocol name, wait/nowait, username, serverpath server-args. Now if I run the inetd service I can indeed log into localhost using telnet, but not from another machine on another interface. Furthermore, there is no way to specify that the telnetd use another interface. After trying everything I could think of the solution was quite simple: install xinetd instead of old inetd, and hey presto, it all worked.

Monday, January 11, 2010

Stress testing pf versus iptables

We wanted to know whether pf or iptables was faster at implementing a white or blacklist of ip-addresses. How can you generate packets on one computer pretending to be from a wide range of ip-addresses (i.e. spoofed addresses) and then detect and measure the receipt of them on another computer? A third machine running pf or iptables would need to be in the middle to act as a router/firewall. In fact generating spoofed packets can be achieved with a number of tools. We chose hping3, since it does this but also has a listen function that examines packets with a signature and then prints out the packet contents after this signature. So if we send 10,000 spoofed packets as fast as possible from machine A to machine C via machine B (running pf/iptables) then they should all be dropped if the spoofed address is on the black list. Each such lookup in the black list on machine B will take some CPU time, and the effectiveness of that lookup in pf versus iptables is what we were hoping to measure. The only remaining problem was how to generate a mixture of black/white-listed addresses and others that were definitely not on the list. The lists can be generated by a simple perl script, and by adjusting it we managed to also generate a shell script containing a long series of hping3 commands. We then set up hping3 on machine C and pipe the resulting output to a file. Counting the number of words in the file tells us how many packets made it through the firewall. If the flood of packets overwhelmed the router as we hoped then more will be dropped by the more inefficient firewall.

Thursday, January 7, 2010

Computing Number of Threads for a Process

Someone suggested that a good way to measure DDoS might be to monitor the number of threads in a process. Under attack, the process of a service might react by creating more threads to handle the new connections. But, how to do it? Someone suggested that a command like:

ps uH p `ps -C <appname> -o pid=` | wc -l

would do it, where '<appname>' is the name of the process being queried. The bit inside the backticks merely retrieves the pid of the named process. It doesn't have to be the precise name – but it does have to be a unique substring of that process's name. The 'wc -l' bit counts the number of lines in the output of ps, which corresponds to the number of threads. Using the same technique as for the current CPU and memory usage it is easy to retrieve the current number of threads from any process. First set the current app name:

snmpset -v2c -c public localhost DOSTF-MIB::dosTFCurrentAppName.0 = firefox

Then, issue an snmpget on the number of threads for the current application:

snmpget -v2c -c public localhost DOSTF-MIB::dosTFCurrentAppThreadCount.0

And you should get the number of threads printed out:

DOSTF-MIB::dosTFCurrentAppThreadCount.0 = INTEGER: 13