Mt. Delight is delightful.

It was a nice ride, but no KOM points for me...I'm just glad that I felt strong the whole way.

Bike route 465206 - powered by Bikemap 

Thanks for the great ride T.!

More fun with network graphs

The work that I have done with analyzing a particular kind of network traffic has continued to bear fruit. For reference, please see my earlier posting "Updated graph from silly network attack".

A couple of weeks ago I was dealing with a different ${organization} than I was describing in my previous posts. They have a totally different network setup at this new site, and, I would soon learn, a totally different scale of traffic.

Here is a graph generated from the data that I collected at this site (same protocol as before):
If you compare this graph to the previous ${organization}'s graph, it is easy to see that this new ${organization} has a completely different scale of traffic. In fact, my data-collection scripts collected so much information in 24 hours at this site that it took some of my post-processing scripts over 20 hours to wade through all of the data!

The thing about this graph is that (unfortunately) there is no obvious spike in traffic that we can focus on to find a problem on the network. The graph does show one thing pretty clearly though: there is a LOT of this type of traffic on this network (especially during daytime hours).

The thing is, it is my position that there is TOO MUCH of this type of traffic on this network. So, in an effort to try to justify my position, I decided to look more deeply into the data. Luckily, this was pretty easy to do, since my post-processing tools also generate a sorted list of how much of this traffic each node on the network generates. With this information, I was able to quickly find some nodes on the network that were responsible for a huge amount of this traffic.

In fact, after looking at the data for a while, I learned that the node on the network that generated the most traffic (by far) was transmitting this traffic using the following pattern:

top_of_loop:

   do this 8 times
      send this ${Protocol} ${Verb}
      wait 10ms
   done

   sleep 6 seconds

goto top_of_loop;

The thing is, if you know anything about this ${Protocol}, it is unreasonable for a single node on the network to be transmitting more than one of these ${Verbs} onto the network every, oh.... 10 seconds (and, if everything works out, the node should STOP transmitting this traffic at this point...). It could be argued that 20 seconds might be an even more reasonable value in this situation. At transmission rates higher than this, this node is basically causing a denial-of-service attack, no matter what its intentions are. This is simply an unreasonable amount of traffic for one node to generate.

It turns out that there is a straightforward way to rate-limit all of this traffic on this network. I don't control this ${organization}'s network, but I thought that before I recommended this configuration to this ${organization} that I'd try to model what it would look like if this rate-limit were put into effect.

The thing is, it turns out that we can model this configuration easily enough. All I had to do to model this configuration change was to modify my post-processing scripts to simulate the rate-limit that I was proposing. Then I just took this modified post-processing script and sent all of the original data that I collected at this site through this. This gives us a very good idea of what effect a per-node 10 second rate limit would have on this network's ${Protocol} traffic.

Here is a graph that shows the overall effect of a 10 second rate-limit:

Here is a graph that shows the overall effect of a 20 second rate limit:


Again, putting some sort of rate-limit rule into effect on this network shouldn't be controversial -- really, there is no reason for any node on the network to be hammering the network with this type of traffic. Discussions are ongoing with ${organization} as to the details of this rate-limit rule.

I'm just happy that I could analyze this data, point out some obvious problems, suggest a solution, and provide a way of simulating the results of my suggestion. I am optimistic that my rate-limiting scheme will go a long way towards making this ${organization}'s network a lot more usable and responsive.

Text Editor Fu

I enjoyed reading "Staying the hell out of insert mode". I have to concur with the author of this article: insert mode is not the most powerful mode in vi.

I don't claim to be some sort of text editing guru or anything. In my career I've met some people who can do some awesome things with their text editor. I always like learning new tricks and techniques.

Getting back to the article that I mentioned above, for me, the reason why insert mode in vi isn't the most powerful mode in vi is because in command mode you can do things like

'c,.!perl -pe 's/foo/bar/'

...and, if you know anything at all about me, you know that I have a thing for Perl one-liners...

I'm not really a heavy-duty vi user though. I'm more of an XEmacs user. In Emacs/XEmacs, you'd type something more like:

C-u M-| perl -pe 's/foo/bar/'

The one person who reads this blog might be wondering "why not use the natural capabilities of the text editor itself to do transformations like this?". To this I pretty much have to shrug and respond "this works for me". It is probably safe to say that a huge percentage of the code I have ever produced in my life has been produced through keystrokes that start with "C-u M-|".

Basically, I am lazy and I would rather let the computer do the typing for me...

Updated graph from silly network attack

As an update to my earlier post regarding silly network attacks, I present an even better graph:

This graph was taken from the same ${organization}'s network.

This graph shows the sum total of a particular type of network traffic. The spike in the graph was caused by a single computer.

Like I said before, I like this type of graph because it lets me quickly focus on and find the offending computer on the network.

Rodale Books == SPAMMERS

I would publicly like to thank Rodale Books (the company that produces Bicycling Magazine, Men's Health, Prevention, etc.) for spamming the crap out of me. Thanks for all of the unsolicited email, folks! Also, thanks for providing no reasonable way for me to get off of your mailing lists.

And actually, let me also thank Active.com for providing Rodale Books with my email address. I am certain that this is what happened, since I used a unique email address when I did business with Active.com.

I have a special message for the folks at Rodale Books: let me be very clear: I will never ever ever ever buy one of your books or magazines.

Idiots.

The anatomy of a silly network attack

It has been a satisfying week at work. I have recently been working with ${organization} on a weird network problem that they've been having. Actually, "working with ${organization}" isn't really right -- they've been complaining and I've been telling the staff that runs ${organization}'s network that I strongly suspected that their network was suffering from a certain type of problem. But the staff at ${organization} won't follow my simple recommendations...not without any evidence of my theory.

So, I decided to put together some proof for my theory. The network problem that ${organization} is suffering from only happens during odd hours, and it was my observation that this problem involves a particular computer protocol. Specifically, this problem involves a particular verb in this computer protocol. Even more specifically, this problem involves too many of these verbs being transmitted onto the network at once. This is sort-of a denial-of-service attack....

In order to prove my theory as to what was going wrong on ${organization}'s network, I put together a simple deep packet analysis tool. Next I hacked together a postprocessor that went through all of the data that was collected and produced a histogram from this. Here's the final product:


Like I said, the problem occurs at odd hours. I thought I had some pretty compelling evidence of my theory at 10:30pm (their time) on March 11th. But then at 3:30am on March 14th my evidence became overwhelming.

For my analysis, a graph like this is invaluable. With this graph, I was able to zero right in on the problematic traffic on the network.

I'm still working with ${organization} on addressing their network problem. I wish I could say that one evil villain caused this problem, but my evidence shows that multiple end-users were involved with this problem and I have to assume that the problem is more of a mis-configuration problem rather than an actual network attack....

bash programmable completion

I am a huge fan of the bash programmable completion project. Just this morning I found myself typing:

scp remote-host:some-long

...and then I hit tab. At this point, the "bash programmable completion" code took over and within a half-second my terminal window was updated with:

scp remote-host:some-long-long-filename

...and if you really understand what went on when I hit that simple tab character, you should be truly impressed.

I love this package! This is one of the neatest, most well-put-together projects I have seen in a long time.