Tuesday, December 22, 2009

Reading the output of system commands in C

The system command in the standard C library lets you issue a commandline command and receive a return code. But that's not usually what you want. Let's say I wanted to issue a ps command to tell me which processes I was running, then how would I know the result? Actually there is a way, because the output of the command goes to stdout, and it is possible to redirect the output to a real file. For example:

FILE *console = freopen( "console", "w+", stdout );
redirects all standard output to a file called "console". After issuing the command:
ps aux | awk '/firefox/ && !/awk/ {print $4}'
console will contain the percentage of memory used by the 'firefox' process, if it was initiated by me. Then to read the result all you need to do is say:
rewind( console );
res = fscanf( console, "%f", &percent );
fclose( console );
and the float 'percent' will contain the percentage of the memory being used by Firefox. Not very efficient perhaps, but timing it reveals on my system that it only takes 10 milliseconds to execute, which is quick enough. And the information is precious - I can't find it out any other way (that's relatively platform-independent), and it tells me how much memory is being consumed by that particular process whose PID I don't even know.

So what's next?

So now I can get via SNMP the percent CPU usage and percent memory usage by a particular process every second. Next has to be the throughput or goodput of a particular port on a particular interface. This will be much harder and may be impossible.

Implementing the SET operation in a MIB

I wanted to set the name of the current application being monitored and then use snmpget on values that depend on that setting. I realise this is not quite the snmp way but it's just what I need to monitor a particular application's current state. That way I can retrieve for example the current application's memory and CPU usage as a percentage value. The snmp tutorial isn't very forthcoming about the details of how to do this. In particular it doesn't tell you where the commandline oid and value are located. All you get are four parameters:

  1. netsnmp_mib_handler *handler
  2. netsnmp_handler_registration *reginfo
  3. netsnmp_agent_request_info *reqinfo
  4. netsnmp_request_info *requests
Try searching the net-snmp site for information about these structures. I couldn't find anything. After some experimentation I managed to work out that the value of an snmpset command is stored in requests->requestvb->val.string and its length in requests->requestvb->val_len. The OID is stored in requests->requestvb->name (an array of ints) and its length in requests->requestvb->name_length (consistent huh?). I ignored most of the SET states (MODE_SET_RESERVE1, MODE_SET_RESERVE2, MODE_SET_FREE, MODE_SET_ACTION and MODE_SET_UNDO), and just provided code for MODE_SET_COMMIT. (The other states are for allocating and deallocating temporary memory). The storage for the string is just a global variable in the module. I just declared it as
static char currentAppName[APP_NAME_LEN];
where APP_NAME_LEN is 128. On receipt of a MODE_SET_COMMIT command I just copy the value at requests->requestvb->val.string into currentAppName. Of course I put a check in to stop more than 128 bytes being copied. I'm not that dumb.

What's next?

The next task is to compute currentAppCPUUsage and currentAppMemoryUsage based on the currentAppName value.

Sunday, December 20, 2009

Compiling your own SNMP subagent

I had some trouble remembering how to do this so I thought I would write down how to decipher the net-snmp tutorial instructions on how to build a subagent for your custom MIB.

Now why would you want a subagent? A subagent is just a demon that runs on its own and connects to the snmpd demon, so adding its functionality to the whole. The advantage of this approach is that you don't have to recompile the main agent or (almost as bad) compile it into a 'dynamically loadable object' which boils down to much the same thing as compiling it into the code. With a subagent you can bring your custom agent and its custom MIB up or down without disturbing the main agent.

Before you start you must have installed the libsnmp-dev package, which contains all the development headers. And also all the snmp and snmpd stuff as described in my earlier blog entry. But you knew that, right?

So, OK, how do you do it?

  1. First write a MIB. I used the NET-SNMP-TUTORIAL-MIB as a basis for my own. After examining it carefully for a few minutes changing it to do what I wanted was not hard. You have to change the names of the parts of the MIB tree to suit your variables.
  2. Next, turn the MIB into C. I used mib2c on the commandline:
    mib2c dosTFMIBObjects
    You are supposed to modify this but for a demo you can just leave it as is. You might want to modify the default values in the .c file though. 'dosTFMIBObjects' is the name of the main node of my custom MIB subtree. For this to work you must already have put your MIB file into .snmp/mibs.
  3. Modify the example-demon.c file (I renamed it dostf-demon.c) by changing the name of the init function to match your own.
  4. Now execute the manual compile steps, using the generated C file:
     gcc -I. `net-snmp-config --cflags` -c -o dostf-demon.o dostf-demon.c
     gcc -I. `net-snmp-config --cflags` -c -o dosTFMIB.o dosTFMIB.c
     gcc -o dostf-demon dostf-demon.o dosTFMIB.o `net-snmp-config --agent-libs`
  5. Finally, run your subagent:
    sudo ./dostf-demon
    And type in the password or it will fail to run correctly. You may also need to enable agentx. If so, add the line:
    master agentx
    to your /etc/snmp/snmpd.conf file and restart the snmpd service.

To access the values just use snmpget:

snmpget -v2c -c public localhost DOSTF-MIB::dosTFSubagentObject.0
This is the leaf-node of the MIB tree you created in your MIB. Now all you have to do is to modify the .c file you generated with mib2c so that it does something useful, i.e. actually computes the values you are retrieving.

Tuesday, December 8, 2009

Defining your own MIBs

Net-snmp has a flaw, at least on Linux. If you ask for anything from the IF-MIB at the rate of once a second, it chews up around 35% of the CPU processing power. This is not a good idea if what we want for the DDoS testbed is a discreet monitoring of each host. The cause is, according to this guy:

netsnmp_arch_interface_container_load() function from "if-mib/data_access/interface_linux.c" has a file pointer to "proc/net/dev" from it tries to get updates of each interface in a while loop, which takes too much of cpu time when we have many VLAN interfaces defined in our server this loads CPU 100%

So the way around it seems to be to write your own MIB. But how easy is that, exactly?

MIB files are defined using the arcane SMI syntax. But they can also be converted into C code, and then compiled. But you still need the textual MIB files to translate the even more arcance OIDs or dot-notation, into more meaningful textual identifiers. For example, in the Net-SNMP tutorial NET-SNMP-TUTORIAL-MIB::nstAgentModuleObject.0 represents the OID . Another advantage of using the text-form of an OID is that snmpset then works with a simple "=" sign. Otherwise you have to tell snmpset the data type of the thing you are setting.

So far I have managed to compile the example SNMP mib C file at http://www.net-snmp.org/tutorial/tutorial-5/toolkit/mib_module/nstAgentModuleObject.c. What they suggest is to include this as part of your snmpd demon, so when you launch it, the demon will automatically recognise the numeric OID. If you want it also to recognise the text-form you have to add the text MIB to .snmp/mibs. Also I had to de-install net-snmp before I could get it to work, and also launch it with supervisor privileges. Simply putting the MIB files into .snmp/mibs is sufficient - the snmpd agent does not have to be restarted.

The next step is to rewrite the sample MIB to do something useful for the testbed. Also, to allow it to be added dynamically instead of requiring recompilation of the agent.

Tuesday, December 1, 2009

File Open and Save Buttons in HTML

HTML is designed for the big bad Internet, but if you use it as a format for interaction over a local network, as in our case, the security restrictions get rather annoying. The problem with the multipart form approach described below is that you don't have the path to the original file. I wanted to load a specific file, edit the contents and save it back – all in a HTML form. The loading worked – sort of – but the saving was impossible.

The Answer

What I did to get around the security restrictions was to define two applets, one an OPEN button, and the other a SAVE button. Each button creates a JFileChooser with appropriate settings. The code for this is trivial and is just a few lines of toy Java-SE. (I leave this as an exercise for the reader!) I pass in two parameters to the OPEN button applet: the name of the Javascript function I want to call to set a text field on the HTML page, and the id of that field. For the SAVE button I only pass in the name of the Javascript function to retrieve the file path from that field. This is called via LiveConnect, which is contained in the Netscape plugin.jar library that is included in every Java installation. Basically this technology allows you to call any Javascript function in the HTML page that the applet is embedded in. Using these simple tools I fashioned a pretty good lookalike <input type="file"/> field that actually works. One condition for this to work is that the two applets have to be in a signed jar along with the plugin.jar contents. Otherwise security prevents it from accessing the local file system. Just try to do this in standard HTML, after clicking on the 'SAVE' button:

Wednesday, November 18, 2009

File Upload in Portlets

If you are writing a web application rather than a website you might want to upload files to the server. An example is where the user might have a set of files in his/her filespace that he/she wishes to load and run on the server somehow. But how to do it?

First Shot

Since HTML has a nice <input type="file"/> element, why not use that? If you try it you'll find that when the user posts the form containing the file input field, that only the file name, not its path gets passed to the server. So you can't open the file and do anything with it unless you have predestined knowledge about where it is. This is a security measure so that the servlet designer doesn't get privileged information about the user's hard disk structure and contents. Security 1 Webapp Designers 0.

Second Attempt

I read somewhere that if you use a POST form with an enctype="multipart/form-data" attribute that the form will post the full path name. Cool! I tried that but in my portlet when I examine the parameters passed in the processAction method I get zero parameters, even if there are other parameters besides the file input field. Great! Back to square one.

Third Go

Then I read that if you process the ActionRequest using the Apache fileupload facility you can recover the input file param and the other params. This complicates things but seems to be the way Microsoft get around the restriction in their web Outlook client. So I tried it too. Only problem is that you don't even then get the full path of the original file but the path to the uploaded file. But you can still read it, which is all you need I guess. Since my files are all small XML files this doesn't matter.

Sunday, September 20, 2009

Copy image of applet to clipboard

Apparently this can only be done if you have a signed applet, because otherwise the Security Manager blocks the call to Clipboard.setContents. There are several steps to getting this to work:

First we add a PopupMenu to the applet's init() method:

popup = new PopupMenu();
popup.add("Copy to clipboard");
popup.addActionListener( new PopupActionListener(this) );
addMouseListener( new RightMouseListener(this) );
add( popup );

Here we create a popup menu with a single entry that says "Copy to Clipboard". There's also a listener for the right mouse button, which for historical reasons is button number 3:

public class RightMouseListener extends MouseAdapter
 Monitor applet;
 public RightMouseListener( Monitor applet )
  this.applet = applet;
 public void mouseClicked( MouseEvent e )
  int button = e.getButton();
  if ( button==MouseEvent.BUTTON3 )
   applet.popup.show( this.applet, e.getX(), e.getY() );

Pretty simple. This just shows the popup menu whenever the user clicks the right mouse button. Here's the PopupActionListener class, which fires when any item is selected:

public class PopupActionListener implements ActionListener
 Monitor applet;
 PopupActionListener( Monitor applet )
  this.applet = applet;
 public void actionPerformed( ActionEvent e )
  BufferedImage image = new BufferedImage( applet.getWidth(), 
   applet.getHeight(), BufferedImage.TYPE_INT_RGB );
  Graphics g = image.getGraphics();
  applet.paint( g );
  Clipboard clipboard =
  ImageSelection imageSelection = new ImageSelection( image );
                clipboard.setContents( imageSelection, null );
 class ImageSelection extends TransferHandler implements Transferable 
  private static final long serialVersionUID = 1L;
  Image image;
  ImageSelection( Image image )
   this.image = image;
  public Object getTransferData( DataFlavor flavor ) 
   throws UnsupportedFlavorException, IOException
   return image;
  public DataFlavor[] getTransferDataFlavors() 
   DataFlavor[] flavors = new DataFlavor[1];
   flavors[0] = DataFlavor.imageFlavor;
   return flavors;
  public boolean isDataFlavorSupported( DataFlavor flavor ) 
   return flavor == DataFlavor.imageFlavor;

Here's where it gets slightly interesting. We create a buffered image the same size as the Applet. Then we get the image's Graphics object, and use it to call the Applet's paint method. Then we just call setContents on the system clipboard. The ImageSelection class is required boilerplate. We just subclass TransferHandler.

Now paste the image into a graphics program like GIMP and you have your image. Et voilĂ !

Monday, September 14, 2009

Roundtripping Form Parameters

So you have a form, and each of the fields has a name and a value. In HTML when you press a submit button or call Javascript to submit the form all the names and values get POSTed or added to the URL as GET parameters. In a JSP things are more complex. You are supposed to define a `bean' that gets attached to the JSP file so you can access the parameters that way:

<jsp:useBean id="user" class="au.edu.qut.ddos.monitor.AppletData" scope="session"/>

This creates the bean "user" and associates it with the session. When the form is reloaded after a submit, the parameters are available and can be used to set the form elements. To copy them into the bean you can say:

<jsp:setProperty name="user" property="*"/>

This copies all the passed-in parameters into the bean "user", provided the bean has correctly named get and set methods for them. Now, to set the actual elements of the form we use the JSP EL or "Expression Language":

<mytag:namedlist name="intf" value="${user.intfList}" selected="${user.intf}"/>

I should explain that the "namedlist" tag creates a <select> element and populates it from the comma-separated list supplied as the "value". The EL expression "${user.intf}" sets the selected property of the tag to the value of the intf instance variable in the AppletInfo class. Each such property needs a get and a set method for this to work: in this case the getIntf and setIntf methods.

Meanwhile, in the GenericPortlet class we define a method processAction:

    public void processAction(ActionRequest request, ActionResponse response) 
        throws PortletException, java.io.IOException
        Enumeration names = request.getParameterNames();
        while ( names.hasMoreElements() )
            String param = names.nextElement();
            String value = request.getParameter(param);
            response.setRenderParameter( param, value );

All this does is read the passed-in parameter names and their values and copy them to the response object. We can of course change these before doing so. They then appear in the JSP and get copied into the bean thanks to the <jsp:setProperty id="user" property="*"/> statement.

Note that the portlet code knows nothing of beans. It just sees parameters.

Monday, September 7, 2009

Portlet Edit View with a single dropdown menu

Getting edit view (when you select "Edit" in the portlet title bar) to display dynamically calculated information can be done via a taglib. I'm going to create a tag called snmphosts using a manual method. There is probably a way to define this in Netbeans using a standard template.

First, create a taglib.tld (it's got to be called this I think).

<?xml version="1.0" encoding="ISO-8859-1"?>

<taglib xmlns="http://java.sun.com/xml/ns/j2ee"
    A tag to display a set of snmp-reachable hosts as a dropdown menu.
  <!-- display a set of reachable hosts as a HTML select element -->

What this specifies is the name of the tag, "snmphosts" and associates it with a Java class in the package examples.tablibs. I also declared an attribute, which is an argument to the tag that is required whenever we use it in HTML: the name of the interface to probe. In simple cases though, you may not need to define an attribute. Here is the SNMPHostList.java file that does the work:

package examples.taglibs;
import javax.servlet.jsp.tagext.*;
import javax.servlet.jsp.*;
public class SNMPHostList extends TagSupport
 private static final long serialVersionUID = 1L;
 String intf;
 public int doStartTag() throws JspException
  JspWriter out = pageContext.getOut();
   out.println( "<select>" );
   String[] hosts = inetaddr.getSubnetHosts( intf );
   for ( int i=0;i<hosts.length;i++ )
    out.println( hosts[i] );
  catch ( Exception ioe )
   throw new JspException( ioe );
  return SKIP_BODY;
 public int doEndTag() throws JspException
  return EVAL_PAGE;
  * Get the interface name
  * @return intf the name of the interface
 public String getIntf()
  return intf;
  * Set the interface name
  * @param intf the name of the interface in the scenario, 
  * e.g. "eth1"
 public void setIntf( String intf )
  this.intf = intf;

This is mostly boilerplate stuff from the TagSupport class (check the Java API documentation). By providing the setIntf and getIntf methods the JSP engine will automatically set the intf instance var for you when it is provided via an attribute. You could equally call them setBanana and getBanana if you had a instance var called "banana". For empty tags as in this case you just do the start tag and "skip" the body. That's what the return values are for. doEndTag hence does nothing. In doStartTag all we do is write out a HTML <select> element with embedded <option>s. We get the options from the inetaddr class which computes a list of available snmp (or pingable) hosts on the network.

To build the library all you do is compile the SNMPHostList.java file using appropriate libraries on the classpath, which you can get from Pluto: I used servlet-api.jar, jsp-api.jar and portlet-api_2.0_spec-1.0.jar. Then place the .class file in a folder called anything, e.g. "hello". Also in that folder create a META-INF directory and copy the taglib.tld file into it. Then cd into hello and make a jar out of it:

jar cf ../hello.jar .

Just drop the taglib into the lib folder inside a WEB-INF folder inside a web application (like a portlet) and the tag should now be available.

Using the tag

You should already have an edit.jsp file inside your portlet. Change it so that it calls the tag:

<%@ taglib uri="/WEB-INF/lib/hello.jar" prefix="mytag" %>
 <jsp:attribute name="intf">eth1</jsp:attribute>

"mytag" is just a namespece abbreviation and can be anything. The key is that we define mytag to BE hello.jar so when we invoke it using <mytag:snmphosts> the java class file gets called to produce its output. The attribute is passed in using the ugly XSLT-type syntax because it is more flexible, but you can also use a direct attribute in simple cases.

When you select Edit mode now in the portlet you should get a dropdown list of computed hosts.

Sunday, September 6, 2009

Deploying Portlets (to Pluto) with Netbeans

Continuing on with the topic of my previous post, I have set up Netbeans to deploy the portlet's WAR file directly to the Pluto portal implementation without having to copy a file to the 'webapps' folder manually.

You'll need the bundled version of Pluto, or at least have Pluto installed on top of Apache Tomcat, and the plugin for Netbeans that supports Tomcat as an application server. I won't go through the process of setting up Tomcat as a server in netbeans, suffice to say it's very straight forward once you have installed the plugin (Tools -> Plugins -> Tomcat, install). Configure you're previously created portlet project to use the Tomcat server as a deployment point and you're done.

A few side points:
The default Context Path (url) is '/Hello'. If you want something more meaningful, just go to the properties for your project, and in the 'Run' section, change the context path to something better, eg NetworkStatusPortlet

You'll still need to configure the layout and page for your portlet in Pluto. So go into Pluto admin, and find you're recently deployed portlet and add it to an existing page or a new one.

Thursday, September 3, 2009

Creating custom tags in EDIT view on Pluto

Imagine you have a portlet, and you want to have an EDIT view that displays a form, and on the form you want some precalculated content, such as a dropdown list of available hosts. But how would you do it? You could compute a <select> element and fill it with available hosts by using some Java, but that would clutter up the JSP file. But with a custom tag library you could just do something like:

<mytag:hostlist subnet=""/>

And that would produce a list of hosts that respond to PING or HTTP or SNMP on the specified subnet. Now how do you do that?

Define a taglib

As in the last post, these are manual instructions. A taglib is just a jar containing at least one .tld file in META-INF. You need a tld file to link the classes in the taglib to their names. Here's an example:

<?xml version="1.0" encoding="ISO-8859-1"?>
<taglib xmlns="htp://java.sun.com/xml/ns/j2ee"
    An example Hello World tag.

This can go into a directory META-INF inside a directory tld:

  • tld
    • META-INF
      • hello.tld

Now it's time to build the tag class:

package examples.taglibs;
import javax.servlet.jsp.tagext.*;
import javax.servlet.jsp.*;
import java.io.*;
public class HelloWorldTag extends TagSupport
 private static final long serialVersionUID = 1L;
 public int doStartTag() throws JspException
  JspWriter out = pageContext.getOut();
   out.println( "<h1>Hello World!</h1>");
  catch ( IOException ioe )
   throw new JspException( ioe );
  return SKIP_BODY;
 public int doEndTag() throws JspException
  return EVAL_PAGE;

This doesn't yet generate a <select> element, but you get the idea how to do it. Compile the java thus:

javac lib/jsp-api.jar tld/src/examples/taglibs/*.java -d tld

which puts HelloWorldTag.class into tld/examples/taglibs, surprisingly. Then make a jar of the taglib, by cd-ing into the tld directory:

jar cf ../hello.jar META-INF examples

Put your hello.jar taglib into the WEB-INF/lib directory of your web application and you can then call it up from edit.jsp:

<%@ taglib uri="/WEB-INF/lib/hello.jar" prefix="mytag" %>

That gives you <h1>Hello world!</h1> in EDIT mode. To get a dropdown list just modify the java code to generate a list and enclose it in <select> tags.

Wednesday, September 2, 2009

Creating Portlets (for Pluto) using Netbeans

Upon my inclusion in this project, I decided to use Netbeans for all my development. I'd had more experience using it than Eclipse, and it seemed to have the right level of simplicity and functionality for what I needed. However, it had no built in project templates for creating portlets. So following the instructions Desmond put on this blog for creating the hello world portlet, I went about using Netbeans to create it.

Firstly create a new project using the Java Web -> Web Application template. Give it a a name in the next screen, keep clicking next leaving most of the settings default. You can delete or modify the created index.jsp under the Web Pages section of the project. Also in this section, in the WEB-INF folder, open web.xml and copy in the information from the tutorial below. Create a new xml file in WEB-INF and call it portlet.xml and copy in the code from below for that file. Right click on the Web Pages section and add a new folder calling it META-INF. Create a new xml file underneath that calling it context.xml or hello.xml and copy in the related code from below as well.

To make this project compile we need into include the portlet-api jar file specified below. You can do this by right clicking on the project, selecting properties and going to the libraries section, and adding the portlet-api jar file to the list of included libraries. However this actually packages the library with the portlet in the WAR file, which we don't want because it will conflict with the already inlcuded library in whatever Portal system your using. So once you've selected the jar file to include, you need to uncheck 'Package' for that libary.

Once that's done you can build the project (don't try and deploy), this will produce a war file in the 'dist' directory of the project, which you can then copy into the webapps folder of your portal driver (Pluto in my case).

Monday, August 31, 2009

Serving up applets on Pluto

The problem is, if I write a simple html page with a simple applet I can display it in apache2 no problem:

import java.awt.Graphics;
public class HelloWorld extends java.applet.Applet {
    public void init() {
    public void paint(Graphics g) {
 g.drawString("Hello world!", 50, 25);

And the view.html is:

<applet code="HelloWorld" width="450" height="250">

Now that works, but if you put this into Pluto and serve the page up, with the HelloWorld.class in the same folder as the view.html file you get this error:

java.lang.ClassFormatError: Incompatible magic value 168430090 in class file HelloWorld
 at java.lang.ClassLoader.defineClass1(Native Method)
 at java.lang.ClassLoader.defineClass(ClassLoader.java:621)
 at java.security.SecureClassLoader.defineClass(SecureClassLoader.java:124)
 at sun.plugin2.applet.Applet2ClassLoader.findClass(Applet2ClassLoader.java:140)

which tells me that the class loader tried to load HelloWorld.class and it received a magic hex number of 0A0A0A0A, not CAFEBABE which begins every class file in Java. And the reason is that it received a 404 not found message from the tomcat server, and presumably the 404 HTTP header begins with 4 new line characters.

So there are two possible reasons:

  1. The HelloWorld.class file should be somewhere else
  2. The HelloWorld file needs to be served manually by Tomcat, in the same way that image files are served. But this should happen by default, so I'm going to try option 1 first.

It turns out that the hello world example (once I removed the ".class" from the name) could be served if I specified the full web url: /monitor/jsp as the codebase of the applet. Doh! Another Homer Simpson moment. Time for a slice of delicious Floor-Pie.

OK, I applied the above strategy to the monitor applet and it worked. Only I had to also specify the archive attribute as applet.jar. Now to add an edit page to the portlet so I can change the parameters of the applet.

Setting up Maven2 behind a password-protected NTLM proxy

The nice Maven people suggest that you might be able to get this configuration working, but don't say how. The Jetty people, however, suggest using CNTLM. I tried installing CNTLM (using synaptic package manager) and managed to configure the conf file in /etc/cntlm.conf which now reads in pertinent part:

Username jbloggs
Domain myorg
Password secret

OK, these aren't real values, but you get the idea. The proxy pointed to here is the REAL proxy. The "domain" is the local domain, not the long full domain-name, but maybe that also works. CNTLM is a proxy to the proxy and runs on localhost. Then I started the cntlm service thus:

sudo service cntlm start

I also had to tinker with my maven setings in ~/.m2/settings.xml:


Note that I point maven to the cntlm proxy, which contacts the NTLM proxy via the cntlm.conf file. One nice consequence of this is that if I now change my general web proxy also to the localhost CNTLM one then ALL my requests get relayed through cntlm, so I don't have to keep telling the NTLM proxy who I am every five minutes. Whew!

And, hey presto, maven works.

One proviso: On the cntlm website it suggests testing cntlm on the commandline, and if you do that it responds with something to the effect that there was no user specified. What this means is that when running on the commandline cntlm does NOT look in /etc/cntlm.conf. So you have to supply that information as options. That threw me for a while until I just tested the service and was surprised to find it was running beautifully.

Saturday, August 29, 2009

Hello World on Pluto

I'm going to use the manual method of constructing a portlet because people don't like Maven and they may not be able to get it to work behind an NTLM proxy (but I managed that in the end).

First write yourself a basic HelloWorldPortlet. It doesn't do much, which is good. Because you want to understand it:

package examples.portlet;
import java.io.IOException;
import java.io.PrintWriter;
import javax.portlet.GenericPortlet;
import javax.portlet.PortletException;
import javax.portlet.RenderRequest;
import javax.portlet.RenderResponse;

public class HelloWorldPortlet extends GenericPortlet 
 protected void doEdit(RenderRequest request, RenderResponse response) 
          throws PortletException, IOException 
         PrintWriter writer = new PrintWriter(response.getWriter());
         writer.println("You're now in Edit mode.");  
 protected void doHelp(RenderRequest request, RenderResponse response) 
          throws PortletException, IOException 
         PrintWriter writer = new PrintWriter(response.getWriter());
         writer.println("You're now in Help mode."); 
 protected void doView(RenderRequest request, RenderResponse response) 
          throws PortletException, IOException 
         PrintWriter writer = new PrintWriter(response.getWriter());
         writer.println("Hello world! You're in View mode.");

Save that in a file HelloWorldPortlet.java. We'll assemble it later.

Now create a web application description file. Call it web.xml. Here's the one I used:

<?xml version="1.0" encoding="UTF-8"?>>

<!DOCTYPE web-app PUBLIC "-//Sun Microsystems, Inc.//DTD Web Application 2.3//EN" "http://java.sun.com/dtd/web-app_2_3.dtd">

The org.apache.pluto.container.driver.PortletServlet is really important. This is the class inside Pluto that registers your portlet. Without that it won't know it's there.

Now create a portlet app definition file, call it portlet.xml:

<?xml version="1.0" encoding="UTF-8"?>
    version="2.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"
    <description>Hello, world! portlet</description>
        <title>Hello, Pluto at last!</title>

There's only one portlet inside it, but you can have several. In fact you can define all your portlets inside Pluto but that means hacking Pluto, which I don't recommend. It's much better to provide a portlet app with separate portlets, like the testsuite.

Finally you need to create a context-descriptor. It's very simple. Call it hello.xml:

<Context path="/hello" docBase="hello.war" crossContext="true"> </Context>

Now compile the HelloWorldPortlet. You'll need to add the portlet-api_2.0_spec-1.0.jar which is in the lib folder of your Pluto (aka Tomcat 6) installation to the classpath:
javac -cp ~/pluto/lib/portlet-api_2.0_spec-1.0.jar HelloWorldPortlet.java
(assuming that your Pluto installation is in your home directory)

Now create a directory called hello, and populate it as follows:

    • hello.xml
    • portlet.xml
    • web.xml
    • classes
      • examples
        • portlet
          • HelloWorldPortlet.class

Finally, cd into the hello directory and type:
java cvf hello.war .

Now copy the hello.war file into the webapps directory inside your Pluto installation. Restart Pluto and if you go to Pluto Admin then the hello world portlet will be available for adding to pages. It even works.

Tuesday, August 25, 2009

First Steps on Pluto

Wow, this is heady stuff. I have worked out roughly how Pluto works. It's not too complicated. The basic application is in pluto-portal-2.0.0.war. All this does is run the portal, the pluto page administration and an about box portlet. If you need more functionality you add another .war file, like the testsuite. In order for Pluto to run the other war file, however, you need to:

  1. copy five jars into the shared lib folder of apache-tomcat-6.0.20: pluto-container-api-2.0.0.jar, pluto-container-driver-api-2.0.0.jar, pluto-taglib-2.0.0.jar, portlet-api_2.0_spec-1.0.jar and ccpp-1.0.jar.
  2. copy tomcat-users.xml from the pluto installation into tomcat's conf directory. Or define a role "pluto" and a user "pluto".
  3. Rename pluto-portal-2.0.0.war to pluto.war and pluto-testsuite-2.0.0.war to testsuite.war and copy them from pluto-2.0.0/PlutoDomain to apache-tomcat-6.0.20/webapps.

And that appears to be it. The only drawback is, defining add-on portlets this way means that you have to modify the default tomcat installation. This makes it difficult to use a commercial tomcat server. Hmmm. But otherwise Pluto can't launch the portlets in the other war file.

And I still haven't worked out how to add portlets to the testsuite, or rather how to replace them with my own. But compared to Jetspeed it's as clean as a whistle. It's so nice that the portlets you add are all in a separate webapp.

Tomorrow I'm going to try to replace the testsuite with my own webapp of portlets defined in Eclipse. No fr***in Maven!!!

Monday, August 24, 2009

Including Javascript in XSL

No one tells you how to do this. If you add Javascript to HTML you end up with some left and right angle brackets and ampersands in your javascript, which you can't have in XML, before it is transformed into HTML. The thing about Xalan is that it first interprets all entity references to &amp; and &lt; before it transforms it and then puts them back again afterwards. You can disable the afterwards transformation by enclosing the script in a <xsl:text disable-output-escaping="yes"> ... </xsl:text> pair of tags. But you have to encode your angle brackets and ampersands as entities too, otherwise you'll get a syntax error. So &amp; gets transformed into & and &lt; into < during the input phase, then you get a literal angle bracket etc in your transformed Javascript. Cool.

That took me an hour and a half to work out again (I forgot how to do it) so I thought I would write it down this time.

Monday, August 17, 2009

Testbed Status portlet

I thought a good way to start off my portal would be to write a simple HTML type of report using the DOS test suite I developed. This tests the network looking for machines exposed on the monitoring AND attack networks, and also detects routers and switches, firewalls etc. A HTML readout of the testbed setup would be a good place to start building and would be easy enough to get me started on porting all my code to the portal.

In fact, why not turn it into a fully-fledged topology view? Rather than just list computers and routers using text, why not draw them using icons? Then use colours to indicate their status. We could write out the network status in XML then view it using an interactive applet.

Sunday, August 16, 2009

Possible Extensions to dosTF

There are a few possible directions to continue development of the testing framework. I would like to start using the GUI for real now, not the various components. So I see the tasks ahead as:

  1. Add logging to applet
  2. Enhance scenario-editing GUI
  3. Add a repertoire of real DOS attacks, compiled for each machine architecture, and ready to deploy.

Sunday, August 2, 2009

Enabling snmp on Ubuntu

I thought I'd write down the recipe for enabling snmp on Ubuntu. Much the same instructions also work for RedHat.

First ensure that the packages snmp and snmpd are both installed. Login as root via ssh to the machine you need to configure, and then:

apt-get install snmp and
apt-get install snmpd

Follow the usual prompts to get it installed then

  1. Type snmpconf
  2. From the text-menus choose to load "none" of the existing snmpd.conf files.
  3. Choose option 1: snmpd.conf as the file to create
  4. Select the option "Access Control Setup" (the number varies)
  5. Choose option 3: "a SNMPv1/SNMPv2c read-only access community name"
  6. In answer to the first prompt type "public".
  7. In response to the request to specify a host address to accept this community name just hit return
  8. In response to the OID request just hit return.
  9. Then type "finished", "finished" and "quit"
  10. Copy the resulting snmpd.conf in the current directory to /etc/snmp/, overwriting what is there.
  11. Edit the file /etc/default/snmpd. At the end of the line saying SNMPDOPTS remove the "". This step is not required on RedHat.
  12. Save the file and restart the snmp daemon: service snmpd restart. If you haven't got the service command install the sysvconfig package or just type: /etc/init.d/snmpd restart (same thing)
  13. Test the snmp service: snmpwalk -c public -v2c localhost. You should get a sizeable chunk of responses, several pages worth. If only one page then your snmpd.conf file is not installed or mucked up or overridden by something else.
  14. Also test from another machine that can access the snmp server you just set up. snmpwalk -c public -v2c <snmp-host-ip-address>. You should get the same output. If it hangs, then you failed to remove the "" from the snmpd file above.

snmpd glitch

Oh the joys of overcoming poor software design! You'd have thought that an application like snmpd, which allows you to configure who can access the SNMP information on a server and from which ip-addresses, could be configured as such. Not a bit. That'd be too easy. The default Ubuntu installation screws it up very nicely by launching the snmp daemon with an argument '' i.e. localhost, which secretly overrides whatever you specify in the snmpd.conf file. You can only access from localhost, even if you used snmpconf to specify access for all. To fix the problem you have to edit the /etc/default/snmpd file and delete the '' from the end of the line that starts with 'SNMPDOPTS'. Then it works.

Sunday, July 12, 2009

Shrinking combo boxes

It seemed a simple enough requirement: just shrink the fonts of my three JComboBoxes so they would fit onto my teeny weeny applet:

But, to do that you need to create your own list cell renderer, which returns a JLabel whose font is set to 9.0 points. Also, you have to change the font of the JComboBox itself, or IT will be 12pt. The result is readable and keeps the component small.

Wednesday, July 8, 2009

SNMP Discovery

SNMP is great until you realise that the MIBs installed on a particular machine may not all be the same. What is needed is an automated discovery technique so that the user can choose, from a palette of available options, which SNMP value should be monitored by the graph. In fact, we also need to greatly reduce the number of available values, since most of them are strings or static numbers such as the number of CPUs or ethernet interface cards. What the user wants to see reflected in the graph are numbers that change, and there aren't so many of those for any given machine. Also he/she wants to monitor specific applications currently running on the machines, and for that we also need dynamic discovery.

So, my idea was to provide a default MIB on the monitoring machine. This would be loaded by the applet locally (from the same directory) and then each value searched for on the currently selected host. If found, it will be entered in the menus. And this is the problem. To avoid multiple hierarchies in the menus I opted for a simple design with two "combo boxes". The left hand combo box contains the siblings of the parent of the currently selected value. And the right hand combo shows the siblings of the current selection, which is highlighted, say, by italics or bold.

  1. To go down the hierarchy the user chooses a value on the right. If that has children, then the right menu moves to the left and the selected element's children takes its place on the right. The new selected element is the first child. If we are already at a leaf node then all we can do is choose another leaf. Otherwise nothing happens.
  2. To go up the hierarchy the user chooses a value on the left hand combo. The contents of the left combo moves to the right. The previous parent becomes the selected item, and its parents get displayed on the left, if they can. Otherwise nothing happens.

Simple. The user only sees part of the available hierarchy at any one time. Like those menus you get on hp printers. Except no special buttons to remember for up and down.

Thursday, July 2, 2009

Stopping and Starting

Stopping and then restarting wasn't working as expected. It defaulted back to PING whereas it ought to have saved the previous method and retried that. So it needed a new parameter to the creation function of Graph.

PING continuing after switch to HTTP

A nasty problem arose yesterday, only just solved. When the user switched from PING to HTTP the graph drew at the bottom, not at the top as it was supposed. The reason was that residual PING results were making their way back into the graph, keeping the maximum artificially high. The reason was that the graph sub-panel, which controlled the generation of probe threads that did the actual PINGs or HTTPs queries, was getting duplicated and replaced but not destroyed every time the graph got resized. This was caused by separating off the initialise code for Graph into a separate routine and rerunning it whenever a resize occurred. The side effect was duplication of threads that couldn't be killed. Doh!

Wednesday, July 1, 2009

Slow DNS lookup

I traced the slowness in the Monitor applet down to the DNS lookup performed by currAddr.getHostName(), where currAddr is an InetAddress object. This looks up the domain name if the address was not created with it already, in which case it just remembers it. So if I cycle through the ip-addresses on a C-class subnet and I find one that is reachable after 5 milliseconds, I need to do a reverse DNS lookup to get the domain name. But if it doesn't have one it takes 15 seconds before it times out. Strangely, it then changes the ip-address to instead of the / it started with. (Don't know what the initial "/" is for). What I want to do is specify a timeout of say 10 milliseconds for lookup to succeed within. After that I am happy to go with the numerical address. But you can't do it, apparently.

SNMP fails because it needs SNMP library

Although I provided the SNMP library 1.4.2 for compilation it isn't included in the applet, so it falls over. Doh! OK, I fixed that by including those class files in the jar for the applet, BUT when I run a simple SNMP query (free RAM) it fails on the Windows XP client. Why?

Tuesday, June 30, 2009

Applets failing to load

Now the Monitor applet is failing to load in the browser. There's no error message in the Tools->Error Console and the Java console doesn't load, presumably because no applet is loaded. At this stage I have no idea what the problem is. It works in Applet Viewer, so maybe the signing is faulty. Running any applet in the current browser gives the message that it is simply ignoring applets.

Apparently the reason was that I had openjdk AND sun6 jdk installed. So I had two incompatible versions of java. So I guess the plugin called the wrong java and failed to load the applet.

Although it took ages to load on the testbed (probably because of all the machines it had to detect) it eventually came up and monitored the network.

Running Java Console in Firefox 3.x

There used to be a plugin for the java console. Now you just enable it by running ControlPanel, which is a program in $JRE_HOME/bin. Set "Show Console" to true and you'll get it whenever an applet is loaded in the browser.

Monday, June 29, 2009

Smartflow Demo

The only Smartbits application that came with the Spirent SmartBits 600 was a demo copy of Smartflow. Indeed, if this worked, it would be an excellent way to measure frame loss and throughput of our Vyatta router. However, I can only get it to report 20% packet loss on the first iteration and 100% loss for all subsequent iterations whether you step the load down or up and regardless of the initial loading. That doesn't make sense, unless there is a limitation on the demo copy. So I have given up for now. This program is not marketed any more. It is part of a new SmartBits module which we don't have. And the test methods in SmartWindow don't work either. Too bad.

Sunday, June 28, 2009

Connecting together two Spirent Smartbits 600s

In the manual it says to turn on slaves before primary controllers, but that doesn't apply to two controllers, a primary and a secondary. When you connect via the RJ45 expansion ports (OUT to IN) you have to power up the primary first, wait for its status light to come on and then power up the secondary. Then the link light comes on but no trace of the connection in the SmartWindow application. Hmmm.

The Debug Console, which is supposed to show it just comes up blank, with a blank dropdown menu. My guess is that it doesn't work correctly in Windows XP.

Maybe it doesn't matter. If one port of the Spirent 600 can spew out 1GB of data flat out then that should be enough to saturate a 1GB port on any router.

Thursday, June 25, 2009

Spirent Smartbits

We have obtained on loan two old Spirent SmartBits traffic generators. They are configured with two 1GB ports on the front, presumably for sending and receiving data. We assigned them IP-numbers successfully but I can't get them to work in a master-slave relationship with the supplied ethernet cable. Maybe that is not he right cable. I will try a straight through cable (don't know about the one I am using but it's not cross-over). There is only one application that comes with it to do anything. It is rather out of date, but it is also not made any more by Spirent. It is a 30 day trial license, which should be enough. SmartFlow sends traffic through the Device Under Test or DUT and measure the latency and frame loss. The flows are variable to some extent but the way it works is that it packages up the packets with sequence numbers, time sent etc, so it can see (in the UDP case at least) how many packets got dropped, and in all cases how long they took to come back. I don't know how to do it yet but it would be a good way to test the maximum throughput of the testbed. We need to write down the results because it should become part of a paper. The only problem may be that the modern equipment we are using won't be able to be outpaced by the SmartBits generators. If so, then we can't get it to drop packets, and won't be able to measure maximum throughput.

The manual keeps warning you not to connect cables without Ferrite Blocks. I bought one for $3.00 from Dick Smiths at Buranda, but it seems we need perhaps two more to actually do the tests. (One is needed for the monitoring cable). Otherwise there is a risk of blowing out the ports. Hmmm.

Resizing Applets to Fit a Portlet

If the user resizes a window, or if he/she has a big enough screen, then the graph applet that does most of the monitoring won't look very good. To make it resizeable automatically requires some Javascript:

function resizeApplets()
    var i = 0;
    var applets = document.getElementsByTagName("applet");
    for ( ;i!=applets.length;i++ )
        var width = applets[i].parentNode.offsetWidth;
        var height = applets[i].getHeight();
            applets[i].setSize( width, height );
        applets[i].width = width;
        applets[i].height = height;
In this routine I look for all the applets on the page, and get the height and width of their immediate parents. Then I set the applet width and height to those values. Note also that I have to set the width and height in the HTML independently, so that the browser will know that the Applet is actually of a different size to what it was originally. (An Applet inherits all the public methods of the Java Applet class by the way). The disadvantage of this approach is that the height won't get changed, but perhaps we could feed in a new height from a form in the portlet. Hmm. I'll think about it. What we do with this is call it using the onresize and onload attributes for the body element. Then it gets called at the right times.

Wednesday, June 24, 2009

So it works on 64 bit Linux

So it all works also on 64 bit Linux. Tomcat is still 32bit and it works but I had to recompile the monitor applet for 64bit (not sure if it was really necessary) using the 64 bit version of Eclipse, and also install the java plugin explicitly. Just installing the jdk is not enough using Synaptic. Now for adding SNMP!

Tuesday, June 23, 2009

Differences between AppletViewer and the browser

Applet Viewer and the browser provide different JAVA environments for the applet, and as such they render differently. For example, in AppletViewer I get higher buttons and menus and no contents on the menu, but it works fine in the browser. Sometimes I hate swing.

Handling non-functional protocols

If a particular protocol/host combination is inactive, for example, if you choose HTTP but there is no web-server at that address, then the graph should clear and display an error message on itself. Then when the user chooses a working combination it should restart and adjust the top and bottom limits of the vertical scale, which it is not currently doing. The result is that when you change from PING to HTTP and it works, then the trace is off the bottom of the scale, because HTTP is less responsive in packets per second. I'll fix that after lunch, and try to add SNMP today.

Loading applets into portlets

There used to be an old JavaAppletPortlet in Jetspeed 1 but it fell into disuse and hasn't been updated. After trying to resurrect it for a while I realised that all you needed was an instance of FilePortlet, and then point it to a bit of HTML containing an applet. All a portlet does at the end of the day is generate a bit of HTML. So where do you put the applet to make it see it? I eventually got it to work out of an applets folder that I added to the root directory of the dostf webapp. Then I can refer to it as codebase="http://localhost:8080/dostf/applets/". That wasn't intuitively obvious, but it worked. Now the problem is just that I am defaulting back to PING when HTML probing fails, but silently. In reality not many hosts will support HTTP, and I ought to display nothing or a message saying that the host doesn't support that protocol, not revert to PING and pretend that it is really HTTP. But I'll fix that tomorrow.

Sunday, June 21, 2009

Running dosTF within Jetspeed 2

OK, now it's working fairly well. I have customised the Jetspeed theme, giving it some more readable fonts and different colours. Next step will be to actually wrap my Monitor applet in the Applet portlet. Then the website will need a bit of structure.

Reconfiguring Tomcat for Jetspeed

One modification of Tomcat made by Jetspeed, which perhaps it shouldn't make, is the adding of the following lines to server.xml in the conf directory:

<!-- deployed Apache Portals Jetspeed/APA listener to initialize logging directory system property -->
<Listener className="org.apache.portals.applications.logging.tomcat .LoggingPropertiesServerListener"/>
Without this line the variable org.apache.portals.logdir is unset and as a result it tries to access jetspeed-specific log files located at <blank>/, or /, aka rootdir, which it can't.

There is another modification in conf/Catalina/localhost/dostf.xml (of course the 'dostf' and 'localhost' components may vary), which specifies where the jetspeed database is located. Unfortunately it won't load the database I created after customisation of dostf, nor the working copy of dv's database. These files really shouldn't hold any information about the customised application. Also there are libraries added to the lib directory of tomcat by jetspeed. Presumably because they are needed for all jetspeed applications. Copying this to a commercial webserver would thus be impossible.

At the moment it objects that the 'Table/View PORTLET_APPLICATION doesn't exist'. No idea yet what that means. It seems to find the database itself, but not a specific table within it. However, when I swapped my dosTF database with that provided by another Jetspeed application, it worked.

Wednesday, June 17, 2009

NICs in Optiplex 960s (Small Form Factor)

Since the dos testbed requires both an attack network and a monitoring network, which are wholly separate, we need a second NIC in each of the four Optiplex workstations. The problem is that the PCIe slot is already taken up by the DVI graphics card. Dell recommend that we use a Broadcomm gigabit card in the PCIe slot, and don't say what to do with the remaining PCI one. We tried a low profile DLINK card, of which Gleb found four examples, but the machine won't then boot. A full size Intel card though did work, but then we can't close the case. We're going to try an Intel Pro 1000 GT low profile card, at $40 a go. If one works we can buy another three.

Non-drawing (J)Panels in Applets

I rewrote the Monitor applet, which displays a graph updated every second. This displays the performance of a particular machine in a convenient applet that can be wrapped up as a portlet and rearranged by the user on a webpage. Unfortunately I couldn't get it to draw. It worked, but the paintComponent calls came in with a 'device clip' only 10 pixels high, so preventing most of the graph from drawing. The reason was that I had failed to override the getPreferredSize() method for my subclassed JPanel. I only overrode getSize(), getHeight() and getWidth(). From memory getPreferredSize() is the one Swing uses as the canonical size of the panel. Once I copied the getSize() code into getPreferredSize() it all worked. Doh!

Sunday, June 14, 2009

NTLM Proxy

Since we use an NTLM proxy I can't use Maven, because it can't work through it. Damn!

Applet Portlet

There's already a portlet for Jetspeed that displays an applet. No work to do there, I hope. Just configure it :).

Plan of work

So the idea is to add SNMP querying to the Monitor application so it can dynamically retrieve and display information using SNMP or whatever in the browser. Then, make it into a portlet and display it in Jetspeed. Cool, what?

Transparency of the SNMP interface

We could have the SNMP calls hidden from the user. For example, a request for memory utilisation on a machine could be sent over SNMP and a request for HTTP load would not. Either way the user just perceives it as a request for information about a machine through a graphical Java interface. All that is required is for a machine to have the SNMP daemon or service installed in the testbed, then the information is simply available.

Writing an SNMP Java Client

I was surprised on Friday to discover that NSClient++, which collects private information from Windows PCs for Nagios, is essentially useless. You can already install a free SNMP service that can provide any information you might need to know about a machine. The Nagios author claimed that SNMP was 'anything but simple', but it seems that he just didn't consider the possibility of integrating it into Nagios at an early stage. Now he has support for SNMP traps, whatever they are. But he doesn't need NSClient++ – it's redundant. What a waste of time for me to write my own service to supplant NSClient++ for the dos monitoring! I feel deflated but I need to have something working soon, and the good news is that writing an SNMP client in Java should be easy.

Strange how I was hired to write C code with no mention of Java but I am finding it difficult to justify the writing of any C code :). It seems that all the necessary C parts are already written. If they had hired someone else with just C skills they would be washed up now. Luckily I know quite a lot about Java.

JetSpeed 2

The possibility exists for porting the dosTF application to Jetspeed 2. It would be a neat container for dosTF because it would allow me to create and configure the layout of multiple instances of the one portlet, say the public monitoring portlet, with the graphical interface. The users could even do that. The alternative is to develop a usable interface by response with users. I kinda prefer leaving it to them. The scenario file could also be rewritten as another portlet. Then the whole application could be a collection of custom portlets, which is what I had in mind at the start but didn't have the infrastructure for. I also get login and customisation of the interface appearance for free.