Capturing packets from an Interface in a MIB

I had to use pcap to capture packets arriving at a particular interface. For that the snmp daemon needs to run as root. You can configure the options for starting up the snmpd program in /etc/default/snmpd. What's not cool, though, is trying to start it up and shut it down. It seems that nanosleep on Linux doesn't work, or rather it only works with a resolution of 1 second. No matter how many nanoseconds you specify nanosleep doesn't sleep until you fill in the tv_sec field. This is, however, more annoying than a show-stopper. I can now start packet capture via pcap_loop and kill it with pcap_breakloop. Stopping takes a few seconds but mostly works. So now we have a MIB that actually detects DDoS attacks!!!

Average CPU usage for a terminating process

Someone asked me if we can easily compute the average CPU usage of a particular program over its lifetime when the program is scheduled to terminate at some point in the future. At that time it should print out the result. Here's what I came up with:

#!/bin/bash
# get average CPU for a process
total=0
times=0
pid=`pgrep "$1"`
while [ -n "$pid" ]; do
temp=`ps -eo pid,pcpu | grep $pid | awk '{print $2}'`
total=`echo "scale=2;$total+$temp" | bc`
times=$[$times+1]
sleep 1
pid=`pgrep "$1"`
done
average=`echo "scale=2;$total/$times" | bc`
echo "Average cpu usage for $1 is: $average"

To run it give it an argument, which should be the name of the process or part or all of the command that launched it, and background the process, e.g.:

./perprocesscpu.sh "java myprogram" &

So, when "java myprogram" terminates, this script tells me its average CPU usage.

Reading from the commandline in C

What's the best way to read the output from a commandline command in C? If you use the int system(char*) function you receive back a return code, not the output. Not much use. To receive the output you could call freopen on STDOUT, then read from the file you redirected it to. But when you close the file (which you must), STDOUT is left in a broken state. There has to be a better way, and there is. You use popen, instead of system. Then, the output of the command is supplied via the FILE handle returned by popen. Here's an example that finds out the memory usage of the firefox process under Linux:

#include <stdio.h>
#include <string.h>
static char sysCmd[256];
/*
 * Issue a system command to get memory usage
 * @return the percent value times 100
 */
int get_memory()
{
    int res = 0;
    float percent;
    sysCmd[0] = 0;
    strcat( sysCmd, "ps aux | awk '{if ($11 ~ /" );
    strcat( sysCmd, "firefox" );
    strcat( sysCmd, "/&& !/awk/) {print $4}}'" );
    FILE *CONSOLE = popen( sysCmd, "r" );
    if ( CONSOLE != NULL )
    {
        res = fscanf( CONSOLE, "%f", &percent );
        if ( res <=0 )
            syslog(LOG_ALERT,"Failed to parse result of %s\n",sysCmd);
        pclose( CONSOLE );
        res = (int)(percent * 100.0f);
    }
    else
        syslog(LOG_ALERT,"Failed to open pipe for %s\n",sysCmd);
    return res;
}

Adding or removing portlets from a page programmatically in Pluto

What I wanted to do in the dosTF portal was to add or remove portlets, which in my case were monitoring applets, in response to the contents of the loaded scenario. If an experiment called for monitoring of a particular SNMP parameter, I wanted to be able to automatically add the monitoring portlet and configure it to monitor the correct parameter on the correct target. But it seems that Pluto is designed rather simply to prevent that. You are supposed to add portlets in the GUI, which kind of defeats the whole purpose of saving the experiment in an XML file. I needed to restore the state of the experiment from the XML and to do that I needed to add or remove portlets from the page programmatically.

Pluto has an undocumented way to do this. There is the PageConfig class used by the Pluto Page Admin portlet. It gets the correct instance of this class from the portlet context:

PortletContext pc = getPortletContext();
DriverConfiguration driverConfig = 
    (DriverConfiguration) pc.getAttribute("driverConfig");
PageConfig config = driverConfig.getPageConfig( "About Apache Pluto" );

The problem with this code is that only the Pluto application has access to the relevant portlet context. You can't get access to it from another application containing another portlet. So, to get around this, I simply copied my ScenarioEditor portlet into the pluto application and hey presto, it works! Removing a portlet is a bit harder. You have to get the portlet ids on the page from the PageConfig and then note the differences before and after. Then record the portlet id of the newly added portlet. It would make more sense if addPortlet just returned the portlet id.

However, configuring the portlet before display seems to be hard with this method. Maybe there is some way but PageConfig doesn't seem to have any knowledge of the 'portlet' itself, just its portlet id. But I'll leave that problem for the next post.

Running an snmpd plugin

To run an snmpd plugin (shared object file) as part of the snmpd service just add the following line to the /etc/snmp/snmpd.conf file:

dlmod dosTFAgentPluginObject /home/desmond/.snmp/dostf-plugin.so

Or, replace the name of my plugin with yours. Then just restart the snmpd service:

sudo service snmpd restart

And Bob is your uncle. Hopefuly this will be a bit more robust than running it as a separate demon.

Addendum: it is more robust, and hasn't fallen over yet.

Extending Per Process Monitoring to Windows

Someone asked me how difficult it would be to get the same information as for Linux on Windows. The answer is: about the same.

Thread-count, cpu and memory usage can be measured via WMI (Microsoft's SNMP) either using the commandline tool WMIC or via C/C++ directly. The amount of required work here is small.

Response time can be measured using exactly the same technique as on Linux. The code just has to be checked so that it works also using the winsock API. Again, trivial.

Goodput is harder, but was also on Linux. You can write a 'shim', that emulates each call to winsock.dll and wsock32.dll. Then you instrument the shim to call snmpset to set the MIB directly for the calls you want to 'instrument'. I think this would take about a month for a moderately good programmer or enthusiastic beginner with a bit of instruction.

There is also the issue of how to integrate it into the SNMP service. Presumably the same techniques work as on Linux: I would be able to create a DLL or separate service that would link with the SNMP service containing the specialised code for monitoring processes.

Crashing of DosTF Demon

Another issue that came up was the failure of the dostf-demon, the program that attaches itself to the main snmp demon to extend its functionality. Under flood attack the communication between the two demons seems to break down and it fails. It says 'broken pipe'. It might actually be more robust if more inconvenient to run the dostf MIB as a DLL. This means that we would have to start and stop the main snmp service to make it work.

Pid of a commandline command

In order to find out anything about a process you need its pid. But how do you find that if all you have, typically, is the command that launched it or the name of the process? The problem is that if I launch a java program using java myprogram then the process listed in top and java -U <user> is "java". And if the system or someone else is running other java processes they will also be called "java". So pidof java or ps -C java -o pid= will retrieve many pids for "java" or only the latest one. It turns out there is a safe way to ascertain the pid of a process that was launched at the commandline. You can use pgrep, such that:

pgrep "java myprogram"

will print out the pid of that instance of java.

Addendum: A bit more complex than pgrep but just as effective is:

ps aux | awk '{if ($11 ~ /java myprogram/) {print $2}}'