Banksy + Simpsons == Awesome

Why I like Perl, Reason #498

For a ${project} I am working on, I am writing an RPM install. I am quite picky about the code that I write -- I insist that the code must be bombproof and reliable. I also insist that the code play nicely with other things on the target system.

So, one of the things that my RPM install has to do is to modify /etc/sudoers . I could have written code that would have simply overwritten the target system's /etc/sudoers file with my own copy, but this doesn't seem right to me. My reasoning goes like this: this file is not exclusively mine to control, and I want to allow other {things} that use this system to be able to use /etc/sudoers too. Quite frankly, the thought of just blindly overwriting a file like /etc/sudoers makes my skin crawl.

One of the things that my code has to account for is that on the target OS the default version of sudo is old enough so that it won't work with a directory like /etc/sudoers.d . My code is pretty much limited to updating the sudo subsystem via the file /etc/sudoers -- that's it.

Another thing that my code has to account for is this: I anticipate that future versions of the ${project} will have further updates for /etc/sudoers. So, the RPM install is going to have to be flexible enough to be able to deal with a never-modified-by-my-code version of /etc/sudoers, as well as an already-modified-by-my-code version of /etc/sudoers.

I let my mind get creative when coming up with a solution to this problem. Naturally, I used one of my favorite tools to solve this problem: Perl.

Here is the solution I came up with:

   perl -i -ne '

    # clean out whatever is there in the first place
    print if (! (/# XYZ Project BEGIN machine-generated section DO NOT MODIFY/
                  ..
                 /# XYZ Project END machine-generated section DO NOT MODIFY/));

    if (eof) {
      print <<'EOF'
   # XYZ Project BEGIN machine-generated section DO NOT MODIFY
   tomcat ALL= /usr/sbin/dmidecode
   # XYZ Project END machine-generated section DO NOT MODIFY
   EOF
     ;
    } ' /etc/sudoers    

In the next version of the RPM install, if somebody needs to update the stanza in the sudoers file, all they need to do is update the "here-document" here in this code, and they're done. Very easy!

So, there you have it: a clean, reliable, easy-to-understand, one-pass solution to the problem at hand. I'm a big fan of solutions like this.

Goodbye Bloglines

I'm pretty unhappy to hear that Bloglines is shutting its doors. I really liked this service, and I used it quite a lot over the past few years.

I find the rationale for this site's closing to be...strange. I can't argue with Ask.com's claim that traffic to this site is down significantly. However, I do find the explanation for this trend to be weird: apparently I am supposed to understand that "everything is social now" and "RSS is dead". But, I have a hard time accepting this.

I use RSS technology exactly for its designed purpose: to make it easier for me to keep track of a lot of web-content, without me having to actually go out and poll each site for that content. For this purpose, RSS is wonderful.

I'm pretty dubious that I'll ever be able to keep track of the content that I am interested in by somehow interacting with a social network. It's not that I'm not social -- it's just that I have my own set of interests, which I assume are fairly specific to me. I don't think that I ever will find myself in a social network in which I can interact with my "network" and thereby keep track of all of the interesting content that I like to read.

So, now I need to find a new RSS reader. I haven't found one yet that I really like. Bloglines had a pretty reliable site, and this site gave me a nice hierarchical view of the feeds that I subscribed to. I haven't found anything yet that works as well as Bloglines always did.

Sigh.

Java IO Faster Than NIO?

The following article on /. piqued my interest: Java IO Faster Than NIO

I read Paul Tyma's slides intently because the subject of "how do you design a massively scalable server?" is one of my favorite topics. There are so many interesting areas to explore in this area.

I have designed several scalable servers over the years. In one case this was for some telco-class equipment that I was helping to design. On that project I ended up using a combination of threads, /dev/poll (on platforms that had this) and poll() (on other platforms). I was under a large amount of pressure to make things fast/scalable on this project, so I was always tinkering around. In the end, I came up with a fairly high-performance communications subsystem that satisfied the goals that we had. The test harnesses that I wrote to go along with this server seemed to validate this as well.

For work on another scalable server that I designed, I went with Java NIO. This is why the article referenced on /. interested me so much. When I started on this particular project, I had no idea of whether to go with a blocking/threads-based architecture or a pure NIO-based system. In the end, I guessed that NIO might be faster so I went with that. The design that I came up with for this server is based on a lot of research that I've done over the years, and this server appears to be very scalable and fast.

I do have some other random thoughts on this topic...

• One of the scalable servers that I have designed happens to interact with SSL/TLS. I have to tell you -- getting a non-blocking I/O layer to interact correctly and efficiently with a SSL/TLS layer is pretty complicated. I spent a huge amount of brainpower and time getting things right here. Given the information that is presented in Tyma's article, if I were to start designing a scalable Java-based server again, I would definitely consider just using Java IO rather than NIO. This would make the code much more simple and straightforward.
• Speaking of making a high-performance communications subsystem, what's the deal with all of the postings on the web from "programmers" who can't figure out how to use their socket API so they insert calls to sleep() in order to "guarantee" that they can read/write data? I must have seen a dozen postings with advice like this. This boggles my mind! A call to sleep() has no place in a high-performance communications subsystem, and anybody who puts one there is not producing good quality code.
• In Tyma's slides, I especially liked the story of Rob Van Behren. When I got to the sentence that read "[w]hen he was done, he sat back and realized he had written the foundation for a threading package" I felt like I had known this gentleman named Rob for a hundred years...

Around The Big Lake

Bike route 576671 - powered by Bikemap

It was challenging and rewarding ride. Definitely the hottest and hilliest ride I've ever done.

I'm telling you right now, and I am deadly serious about this: there is nothing better than going on a challenging adventure with a great bunch of people.

Thanks go to JP for putting together such an awesome route. Thanks to R.+S. for the water and the great support. S.'s cookies were to die for and were just the right thing after that one particular hill. C. kept on talking about "frappes in Wolfboro" and the idea of this kept me pedaling enthusiastically. R. pulled the paceline into Middleton while I was hanging on the back, trying to transfer every water droplet from my nearly empty bottles into my dehydrated body. T. and I even sprinted at mile 143, but he beat me by a wheel...{sigh}.

This was a great ride!

How is your software project trending?

How is your software project trending? Are you going to make your deadline with time to spare? Is your project going to spend all of its time in development, leaving not enough time for testing? Is your project bogged down?

Can you understand how things are trending in ten seconds or less? Can other people in your organization understand these trends as well?

Do you employ any metrics to track this trend?

...

At the point I am at in my career, in addition to doing technical things, I also manage projects. When I first started managing projects (several jobs ago) the thing that I really wanted was a way to understand, in ten-seconds or less, was how the project was trending. I am somebody who believes that some metrics are very useful, and this is a metric that I really wanted.

I knew that the information that I wanted to understand was ultimately stored in the bug-tracking system that I was using, but I simply could not find any pre-packaged solution that gave me what I wanted.

So, I hacked together my own solution.

My solution basically consists of a "cron" job that queries the bug-tracking system for its data. I run this cronjob every morning at 12:05am.

What I do is as follows: I organize projects by milestones, and I track bugs by the milestone that they need to be addressed by. How all of this works at a given jobsite varies from company to company. But the essential point is that all of the bugs are organized by clear milestones.

Then, every morning at 12:05am, my cronjob runs, querying the bug-tracking system for its information. Generally, I think that looking at the previous 45 days worth of data provides a reasonable view of how a project is going, so my scripts default to using a 45-day window.

And then....my scripts produce graphs...one per milestone that I am interested in. One of the first things I do every morning when I get into work is to look at these graphs. I also publish these graphs so that everybody in the organization can see and understand these. I am, after all, a big believer in transparency.

Having this system run automatically at night is a win for me, because it gives me more time to do actual technical things during the day. I simply do not have time to futz with graphs every day.

I'll show you some examples of these graphs, but before I do, I have to issue the following disclaimer: ALL OF THE DATA THAT WAS USED TO GENERATE THESE GRAPHS IS FAKE.

Seriously, the data is fake, OK? Trust me.

So, here is an example:
So, what we have here is a project in which ~100 bug tickets were written. Some of these tickets might have corresponded to feature requests and some of these tickets might have corresponded to actual bugs in the project itself. The graph here clearly tells how the work towards this milestone went overall: roughly 45 days before "01-apr-2049", work started in earnest. Work on the project started slowly, but then things picked up. If you consider the fact that the number of "Open Bugs" appears to be relatively constant over time and the number of "Bugs Being Tested" appears to be increasing greatly over time, then you can easily deduce two things:

• during this time, the number of new bugs being written against this project/milestone was roughly equal to the number of bugs being "fixed" and sent to SQA for testing.
• SQA was getting a little bit backed up with tickets that they needed to test and verify...but this doesn't seem to have been a big deal because SQA seems to have had the capability to test/verify/close a large number of tickets in a short amount of time.

By interpreting this graph, you can even see pretty clearly that the project came to a pretty deliberate end. This is good. Everybody who looks at this graph should easily be able to understand all of this. This is the whole point of the graph.

Here is an example of another graph:
This is a much different graph than the previous graph. There are two things that I would like to immediately point out about this graph: this graph only includes tickets that have been deemed to be "blockers" to the release and also the window that this graph depicts has been increased to three months.

The addition of the "blockers" criteria is something that I have added to my reporting tools. My thinking is that, at some point in a software release, the whole organization needs to just concentrate on blockers and nothing else. Hence, my reporting tools offer this capability.

I have increased the reporting window here to ninety days to better illustrate the points that I will make next.

There is something very likely wrong with a development effort that produces the preceding graph. The number of open "blocker" bugs never trends towards zero, and neither does the number of "testing" bugs. It should be very easy to understand the following point: unless and until these two lines reach "zero", the organization cannot complete its software release. Even more troubling than this is the fact that the number of "closed" blockers is constantly increasing, which indicates that there is a constant stream of "blocker bugs" being found and fixed...for nearly three months.

How serious is all of this? Well, it depends on THE SCHEDULE, of course. In my experience, three months is a long time to be working on a single milestone, so, if I were to see a graph like this, I'd be pretty concerned.

...

These graphs are a (trivial) invention of mine, and I find them to be useful in my work in managing projects. I hope that others find this technique to be useful!

....

Update:  naturally, soon after I published this, I learned that this is type of graph has an official name:  burn down chart.  I continue to find these charts to be useful in my own planning and management.

Dream for a Dollar

A friend of my wife's sent along the following email:

I'm sending this email to ask each of you to pass this website along to as many people as you know. As most of you know my son Travis has Muscular Dystrophy. He is in need of a new van that will get adaptive equipment from the State of Massachusetts. However the cost of the vehicle is the responsibility of the disabled driver and the state presents only two options of vehicles to choose from, a large industrial van or a Town and Country. These are the only two vehicles the State can adapt to hand controls and lower the floor. His current van is battered and bruised and in constant need of parts replacement. The State has slotted him for a new vehicle so the process has begun.

Dream for a Dollar is just that ... we are asking folks to donate a dollar. Not much. We're hoping his site will go viral and dollar donations will add up. So asking people you know to send this site out to as many people as they know will be a great help.

Please follow Travis' progress, his stories and updates on http://www.dreamforadollar.org

This seemed like a good cause to us, and we know that the need is legitimate.

If you are reading this, perhaps you'd like to donate too?