More fun with network graphs

The work that I have done with analyzing a particular kind of network traffic has continued to bear fruit. For reference, please see my earlier posting "Updated graph from silly network attack".

A couple of weeks ago I was dealing with a different ${organization} than I was describing in my previous posts. They have a totally different network setup at this new site, and, I would soon learn, a totally different scale of traffic.

Here is a graph generated from the data that I collected at this site (same protocol as before):
If you compare this graph to the previous ${organization}'s graph, it is easy to see that this new ${organization} has a completely different scale of traffic. In fact, my data-collection scripts collected so much information in 24 hours at this site that it took some of my post-processing scripts over 20 hours to wade through all of the data!

The thing about this graph is that (unfortunately) there is no obvious spike in traffic that we can focus on to find a problem on the network. The graph does show one thing pretty clearly though: there is a LOT of this type of traffic on this network (especially during daytime hours).

The thing is, it is my position that there is TOO MUCH of this type of traffic on this network. So, in an effort to try to justify my position, I decided to look more deeply into the data. Luckily, this was pretty easy to do, since my post-processing tools also generate a sorted list of how much of this traffic each node on the network generates. With this information, I was able to quickly find some nodes on the network that were responsible for a huge amount of this traffic.

In fact, after looking at the data for a while, I learned that the node on the network that generated the most traffic (by far) was transmitting this traffic using the following pattern:

top_of_loop:

   do this 8 times
      send this ${Protocol} ${Verb}
      wait 10ms
   done

   sleep 6 seconds

goto top_of_loop;

The thing is, if you know anything about this ${Protocol}, it is unreasonable for a single node on the network to be transmitting more than one of these ${Verbs} onto the network every, oh.... 10 seconds (and, if everything works out, the node should STOP transmitting this traffic at this point...). It could be argued that 20 seconds might be an even more reasonable value in this situation. At transmission rates higher than this, this node is basically causing a denial-of-service attack, no matter what its intentions are. This is simply an unreasonable amount of traffic for one node to generate.

It turns out that there is a straightforward way to rate-limit all of this traffic on this network. I don't control this ${organization}'s network, but I thought that before I recommended this configuration to this ${organization} that I'd try to model what it would look like if this rate-limit were put into effect.

The thing is, it turns out that we can model this configuration easily enough. All I had to do to model this configuration change was to modify my post-processing scripts to simulate the rate-limit that I was proposing. Then I just took this modified post-processing script and sent all of the original data that I collected at this site through this. This gives us a very good idea of what effect a per-node 10 second rate limit would have on this network's ${Protocol} traffic.

Here is a graph that shows the overall effect of a 10 second rate-limit:

Here is a graph that shows the overall effect of a 20 second rate limit:


Again, putting some sort of rate-limit rule into effect on this network shouldn't be controversial -- really, there is no reason for any node on the network to be hammering the network with this type of traffic. Discussions are ongoing with ${organization} as to the details of this rate-limit rule.

I'm just happy that I could analyze this data, point out some obvious problems, suggest a solution, and provide a way of simulating the results of my suggestion. I am optimistic that my rate-limiting scheme will go a long way towards making this ${organization}'s network a lot more usable and responsive.

Updated graph from silly network attack

As an update to my earlier post regarding silly network attacks, I present an even better graph:

This graph was taken from the same ${organization}'s network.

This graph shows the sum total of a particular type of network traffic. The spike in the graph was caused by a single computer.

Like I said before, I like this type of graph because it lets me quickly focus on and find the offending computer on the network.

The anatomy of a silly network attack

It has been a satisfying week at work. I have recently been working with ${organization} on a weird network problem that they've been having. Actually, "working with ${organization}" isn't really right -- they've been complaining and I've been telling the staff that runs ${organization}'s network that I strongly suspected that their network was suffering from a certain type of problem. But the staff at ${organization} won't follow my simple recommendations...not without any evidence of my theory.

So, I decided to put together some proof for my theory. The network problem that ${organization} is suffering from only happens during odd hours, and it was my observation that this problem involves a particular computer protocol. Specifically, this problem involves a particular verb in this computer protocol. Even more specifically, this problem involves too many of these verbs being transmitted onto the network at once. This is sort-of a denial-of-service attack....

In order to prove my theory as to what was going wrong on ${organization}'s network, I put together a simple deep packet analysis tool. Next I hacked together a postprocessor that went through all of the data that was collected and produced a histogram from this. Here's the final product:


Like I said, the problem occurs at odd hours. I thought I had some pretty compelling evidence of my theory at 10:30pm (their time) on March 11th. But then at 3:30am on March 14th my evidence became overwhelming.

For my analysis, a graph like this is invaluable. With this graph, I was able to zero right in on the problematic traffic on the network.

I'm still working with ${organization} on addressing their network problem. I wish I could say that one evil villain caused this problem, but my evidence shows that multiple end-users were involved with this problem and I have to assume that the problem is more of a mis-configuration problem rather than an actual network attack....

Patch for Unix Network Programming Interface Code

I am a big fan of the book _UNIX Network Programming_ by Rich Stevens. Maybe if you are reading this blog you are too.

Anyways, I have discovered that some of the code in this book is suffering from some bit-rot. Specifically in section 17.6 of the third edition some code is presented that allows the programmer to traverse the list of network interfaces on the local machine. This code uses uses ioctl(...., SIOCGIFCONF, ...) to get the information.

The source of the bit-rot is that Stevens' code made some assumptions about the size of (struct ifreq).....and these assumptions are no longer valid, at least, not for me who is running a relatively modern Linux 2.6 box.

So, I've come up with a patch. I'm pretty pleased with my patch, because

1. the code can be fixed.
2. the code can be simplified.
3. the code can be made to be more portable and to still work even if this union is changed in the future.

I thought that other people would find this to be useful as well, so here it is.

The VLAN Koan

I wrote the following parody/koan one day to help a tech-writer friend of mine understand VLANS. He liked it. I hope you do too.

Goso said: ``When a packet goes out of its egress to the edge of the abyss, its header and its payload all pass through, but why can't the VLAN tag also pass?''

Kevin's comment: If anyone can open one eye at this point and say a word of layer-2 bridging, he or she is qualified to discuss the intricacies of ``one-armed routers'', and, not only that, he or she can save all sentient beings under them from broadcast storms caused by Windoze boxen. But if he or she cannot say such a word of true IEEE 802.1Q VLANs, he or she should turn back to their tag.

If the packet is transmitted, it will fall into the ether;
If it remains in the queue, it will exceed its TTL,
That little VLAN tag
Is a very strange thing.

(apologies to Mumon... (-: )