Engineering Documentation Rant

B. sent me a link to Institutional memory and reverse smuggling.  When I got to this paragraph:

Oh, and as an external consultant, I'm not allowed to know some of the trade secrets in the documents. The internal side of the team needs to handle the sensitive process information, and be careful about how that information crosses boundaries when talking to the external consultants. Unfortunately, the internal team doesn't know what the secrets are, while I do. I even invented a few of them, and have my name on some related patents. Nonetheless, I need to smuggle these trade secrets back into the company, so that the internal side can handle them. They just have to make sure they don't accidentally repeat them back to me.

....I didn't know whether to laugh or to cry.


.....


Here is related and possibly semi-amusing story about some of the work that I used to do at a job in my past:

I used to work at a certain networking company.  I did a fair amount of work with the SNMP protocol and something called "SNMP MIBs".  A "MIB" is like a database schema -- the MIB isn't the management data itself, rather the MIB just describes how the management data is organized.

Well, as I cut my teeth at this company, I had to work with MIBs that were designed in-house.  This was frequently a tedious chore, because our internal MIBs nearly always came without any documentation whatsoever.  The implementation of these MIBs also varied from product family to product family -- probably because the design was poorly understood.  I was at the end-of-the line, trying to make sense out of all of this stuff.  Frequently my job was basically a reverse-engineering job -- I'd have to make sense out of some new MIB, and then I'd have to make sense out of how our own products actually implemented the MIB.  This entailed a lot of work in the lab, as well as a lot of email and phone tag.

Here was the first big problem:  it was totally crazy that I had to reverse engineer this stuff.  Everybody involved with the production of this stuff worked at the same company, during the same time period.  I shouldn't have had to have reverse engineer anything.  All of the engineering staff should have understood what the design was at the time that all of this stuff was produced.  But this simply did not happen.

Vendors aren't the only ones who write MIBs.  Standards organizations write most of the useful MIBs.  Here, take a look at one of my favorite MIBs:  RFC1573.  This is actually a very technical document:  there is a bunch of text in here that describes how the data is structured and how things work.  There is also a machine readable section in this document.

Whenever I had to do my work at this company, it was always a lot easier for me to deal with a MIB written by a standards organization rather than a MIB developed in-house.  Why?  Because somebody bothered to document the design of the MIB.  There is no reverse engineering required to deal with such a MIB.

Eventually, during my career at this company, I volunteered to help create new MIBs.  I had a few reasons for doing this, but one of the most straightforward reasons was that I thought that I could help make the company more efficient by ensuring that newly designed MIBs came with some documentation.  I made it very clear from the outset that I didn't want to spend a huge amount of time ensuring that the MIB was as...perfect...as an IETF MIB.  I just wanted to get the design into the MIB so that there would be no more reverse engineering required.

Here is the second, even crazier part of my story.  I started helping with the design of a certain MIB.  My work was done as a part of some sub-committee work.  I jumped into my work, and I produced a MIB that I think pretty elegantly solved a certain problem that was at hand.  I wrote everything up.  Nobody on the sub-committee besides me really had any input.  At this point, I gave my work to the committee chair for discussion and approval by the full committee.  I fully expected my new MIB to be quickly approved of by the overall committee.

I was so naive.  I did not anticipate what happened next.

Eventually, the committee chair forwarded my new MIB to all of the other committee members.  The fireworks started immediately.  I got at least a dozen confused queries, all in the form of "how does this work?".  I really did not understand these questions.  I tried to explain how things worked, but I always referred the members of the committee back to my original document.  I told them things like "see section 3 of the MIB document ; it gives an example that precisely answers your question".  Eventually, there was so much acrimony  over the design of my new MIB that a meeting was called.  I remember driving to this meeting, thinking to myself "I am traveling to a very hostile meeting, and I do not understand where the hostility is coming from".  I was totally confused.

The meeting itself was everything I expected -- there was lots of hostility and yelling.  I kept on telling people "I don't understand why you are confused about any of this ; this is all clearly explained in the document".  I spent nearly the whole meeting re-explaining things that were clearly described in the document.

After the meeting was over, I followed one of the committee members back to his office to try to understand what had happened.  I asked "why was there so much confusion over this?".  Eventually, I figured out what went wrong.

Take a look at RFC1573 again.  My document had a similar format (but it was a lot shorter).  Do you see how in RFC1573 there is some text, and then in section 6 appears the machine readable part of the document?  Well, the committee chair had never ever seen a MIB before that had any sort of descriptive text in it.  This confused him, so, before he sent out my MIB to the whole committee, he edited the MIB and took out all of the descriptive text.  The reason why the other committee members didn't understand the design of my MIB is because they were all looking at some bastardized version of my document, one that didn't contain any of the actual design.  Let me drive the point home:  what do you think would be easier for a human to understand:  the full text of RFC1573 or else just the text in section 6 of RFC1573?

When I learned of what happened to my document, I got pretty angry.  I had just endured a hostile, two hour meeting, and a lot of the acrimony during the meeting would not have been present if the committee chair had simply provided my document to the other committee members without modification.

So, I walked over to the committee chair's office at this point and asked him why he had done this really stupid thing.  His response?  "I never thought about it"  When he said this I completely believed him.

As I drove home from my meeting, I was pretty wrecked.  After some time, I realized that I worked for a company with some pretty real problems.  We not only had the problem of "the internals of our product is really poorly documented", but, we also had another problem too:  it was basically codified into the workings of this committee that "design documents should not exist".

...

A lot of other things went wrong with my involvement with this committee too.  I soon lost my enthusiasm for working on this committee.  I moved on to other things soon after.

I do continue to believe that a little bit of documentation, in just the right places, has a huge positive effect in the overall efficiency of an organization.

I love mjd's "It came from... the HOLD SPACE!"

I love mjd's It came from... the HOLD SPACE.

This article describes some of the culture that I grew up in.

Slide 18 in particular....wow, that takes me back...  I can remember cutting my teeth on some sed/shell-scripts back when I was younger.  I had work that needed to be done, and these seemed like the logical tools to do it with.

I made progress on this work too.  For example, in one frenetic afternoon/evening I wrote a script that my boss (later) claimed saved the company....a huge sum of money.  It was nice to be able to solve some big problems with these tools.

However, I never did really fall in love with these tools.  The tools themselves had some downsides:  not portable enough ; subject to arbitrary internal buffer limitations ; and, most importantly, for some of the problems I was trying to solve I felt like I was working with my arms tied behind my back.

In mjd's slides, you can see him grapple with a couple of problems in the "sed" section.  When I saw this, I got this awful case of déjà vu...

I have a friend who is 100 times as good of a sed programmer as I am.  I have seen him whip together phenomenal sed programs that accomplish some totally neato and useful work.  I never really got to be a good sed programmer, and I always admired his skills in this area.

However....I never really felt like I needed to put in a ton of time into sharpening my sed skills.  I can remember on one specific day, fighting like crazy to get sed to do The Right Thing in a certain text transformation.  I must have fought for two hours to try to get things right.  But the problem I was running into was a lot like the problem mjd brought up in his article -- what to do about that last line?

Eventually, I decided that it wasn't a good use of my time to spend a huge amount of time with tools like this.  There was this neato new language called Perl....and it looked very similar to what I already knew....but it really didn't have any of the limitations that I regularly found myself dealing with with the old tools.  So, I spent quite a long time with Perl, and that proved to be pretty fruitful.


Nowadays, I pretty regularly work with some pretty huge datasets with my trusted tool:  Perl.   I still find the old tools to be useful, but when the problems get serious, I reach for more powerful tools.

Kardashian Rake

Hi Sears,

Thanks for the new rake.  No, really.

This new rake is even nicer than the one I bought from you a long time ago  (Craftsman...lifetime warranty...etc.).   I used the old one quite a bit and eventually the wooden handle wore out.  I like the new fiberglass handle.

Sears, you probably don't remember me.  I used to frequent your store fairly often.  But your competitors are so nimble and they give me what I want...

Remember that time I bought a dishwasher from you?  I opened one of your credit cards to make the purchase, because this saved me some money.  Wow, did you ever make me pay for that, huh?  You called....and called....and called...trying to get me to sign up for financial services and every other scheme under the sun.  I didn't get a peaceful night at home for at least a week and a half.

Anyways, that was all years ago, and despite you treating me pretty poorly, I've continued to buy some items at your store.  I mostly buy tools, but sometimes I buy some clothes there too.  Clearly my sense of fashion is...simple.

When my old rake broke, I thought to myself "gosh, it seems a little bit unfair to return this rake".  I mean, let's be honest, I did abuse the hell out of this thing for more than a decade.  In the whole scheme of things, it seemed fair to just buy a new rake.

However, at the same time that my rake broke, what did I hear on the radio?  "Sears signs a big deal with the Kardashian family".  Huh?

Isn't this the family who is most famous because of "Kim", who seems to be some vapid woman whose only real accomplishment in life is creating a sex tape?  I'm certainly no prude, but I happen to think that a woman like Kim Kardashian is a horrible example for young women everywhere.  By endorsing people like the Kardashians, Sears seems to be suggesting to young women everywhere "don't make anything out of yourself -- don't go to school -- don't try to be a productive member of society -- just make a sex tape and you'll be all set".

The icing on the cake is that Sears seems to refer to their deal with this family of nitwits as the "Kardashian Kollection".  I guess Sears figures that if you've gone this far down the road to unseemliness, proper spelling is the least of their worries.

So, anyways, the way I figure it, if Sears can send millions of dollars to the Kardashian clan, then they can give me a new rake. too.

Thanks for the new rake, Sears!  Don't worry -- when it breaks, I'll be back!

-k

Bradford Century

Bike route 1110607 - powered by Bikemap 

We did this ride in July.

Where did C. find these hills?  Wow, this was an excellent ride.  The sun baked me like a fried egg on that last big climb.

You can't go wrong going on a great adventure with a great bunch of people.

Politician Skips Chance To Latch Onto Our Paceline

Picture taken from:

http://blogs.wsj.com/washwire/2011/09/30/ron-paul-takes-a-bike-ride-in-n-h-sans-helmet-of-course/



Apparently I passed Ron Paul on my lunchtime bike ride today.  Our group was HAMMERING along.  We were traveling at such a clip that I didn't even recognize him.  Mr. Paul didn't seem to be interested in latching onto the end of our paceline.

Ah....Fall in New Hampshire.  It is getting to be election season...

Amiga Saves the Day

Our eleven-year-old TV died yesterday.  Bummer...

This is the only TV that we had in our house, so, naturally, we started thinking about a replacement.  It seems likely that we're going to inherit one of our family member's cast-off TVs.

In the meanwhile, while we are waiting to arrange for a replacement, I decided to cobble together {something else} for us to watch TV on:

What is this?  I replaced our 11-year old, recently deceased TV with my 24 year-old Amiga monitor.  It still seems to run great!  Thank goodness for composite audio/video!  I suspect that when it comes time to replace my next TV that it won't be nearly as simple to perform this sort-of swap.

I've still got my old Amiga 1000 in a box downstairs  (I didn't need to turn it on for this exercise).  My wife grouses about my "computer museum" occasionally, but yesterday my box of computer junk turned out to be "handy".

Super Taster

So, my wife and I were watching one of our favorite cooking shows on TV the other day: New Scandinavian Cooking hosted by Andreas Viestad. The subject of the show related to apples and sourdough.

So, Mr. Viestad related the following story: one day he made some sourdough for a friend of his. His friend liked the bread, but had picked up an interesting taste sensation in the actual bread itself. His friend asked if the sourdough starter had been created with grapes. Mr. Viestad (who lives in Norway) had no idea: he had obtained his starter over a decade before from a friend of his who lived in England. Intrigued by this small mystery, Mr. Viestad called his friend in England to inquire about the origin of the sourdough starter. It turns out that his friend didn't really know about the origin of the starter either -- she had actually gotten the starter from a friend of her's from California. So, Mr. Viestad contacted this distant friend in California to ask about the origin of the sourdough starter. It turns out that the starter was created in Napa Wine Country, using local grapes.

My wife and I found this story to be really neat! Can you imagine -- being able to pick up the taste of grapes in a slice of bread created with a sourdough starter that was in its Nth generation of existence, originally created over a decade before? Amazing...

Excitement on Horse Corner

So, there we are climbing Horse Corner yesterday, when some dump-truck heading towards the local gravel pit takes a turn in front of us. As the truck takes the turn I think to myself "gosh, that truck is going into that turn pretty fast...".

The truck makes the turn...for the most part. Everything on the truck makes it except for the apparatus on top of dump trucks that swings back and forth -- the thingie that prevents the wind from blowing away the contents in the body. This apparatus can't take the forces involved with the turn, so the whole thing...falls apart.

So, now me and the guy who I'm riding with find ourselves climbing a steep hill, with a heavy 8-foot long steel pipe clattering down the hill straight at us. My fellow cyclist goes right into the ditch, I swing left out into the road, and the pipe clatters between us. A little way down the hill the pipe rolls off into the ditch.

The dump-truck never stops.


I guess that some rides are more memorable than others... (-:



(the attached map was nicely put together by somebody else)

Certificate Authority Model Is Br0ken

I was mildly surprised when I read in Bruce Schneier's Blog that the Comodo Group Issued Bogus SSL Certificates. Here is yet another example of a CA that doesn't even understand why it exists, and what security precautions it should take because of this.

I think that the most prescient observation I have ever seen written about the Internet's current manifestation of CAs was written by Matt Blaze, who wrote:

Commercial certificate authorities protect you from anyone from whom they are unwilling to take money.

Of course, in this particular case, Comodo Group was so eager to take money from people who wanted their stamp of approval that they partnered with various third-parties in order to issue more certs....and they never ensured that these third-parties implemented adequate security measures.

This NYT article also provides pretty good information.

Now a lot of this mess has ended up in the laps of browser manufacturers (Mozilla, Google, Opera, Microsoft, etc.). I feel badly for them, really, I do. I am involved with a project right now that works with things like root certificates, and handling things like certificate revocations is something that I am only beginning to have the time to investigate. This is a complicated area to work in...

Synchronization Bugs

My previous posting reminds me of an amusing story I was reading in the book _Coders at Work_ by Peter Seibel. The story is told by Joshua Bloch, who I consider to be an exceptionally gifted programmer:

To test the code, I wrote a monstrous “basher.” It ran lots of transactions, each of which contained nested transactions, recursively up to some maximum nesting depth. Each of the nested transactions would lock and read several elements of a shared array in ascending order and add something to each element, preserving the invariant that the sum of all the elements in the array was zero. Each subtransaction was either committed or aborted—90 percent commits, 10 percent aborts, or whatever. Multiple threads ran these transactions concurrently and beat on the array for a prolonged period. Since it was a shared-memory facility that I was testing, I ran multiple multithreaded bashers concurrently, each in its own process.

At reasonable concurrency levels, the basher passed with flying colors. But when I really cranked up the concurrency, I found that occasionally, just occasionally, the basher would fail its consistency check. I had no idea what was going on. Of course I assumed it was my fault because I had written all of this new code.

......

I spent a week or so writing painfully thorough unit tests of each component, and all the tests passed. Then I wrote detailed consistency checks for each internal data structure, so I could call the consistency checks after every mutation until a test failed. Finally I caught a low-level consistency check failing—not repeatably, but in a way that allowed me to analyze what was going on. And I came to the inescapable conclusion that my locks weren’t working. I had concurrent read-modify-write sequences taking place in which two transactions locked, read, and wrote the same value and the last write was clobbering the first.

I had written my own lock manager, so of course I suspected it. But the lock manager was passing its unit tests with flying colors. In the end, I determined that what was broken wasn’t the lock manager, but the underlying mutex implementation! This was before the days when operating systems supported threads, so we had to write our own threading package. It turned out that the engineer responsible for the mutex code had accidentally exchanged the labels on the lock and try-lock routines in the assembly code for our Solaris threading implementation. So every time you thought you were calling lock, you were actually calling try-lock, and vice versa. Which means that when there was actual contention—rare in those days—the second thread just sailed into the critical section as if the first thread didn’t have the lock. The funny thing was that that this meant the whole company had been running without mutexes for a couple weeks, and nobody noticed.

I find stories like these from the trenches to be weirdly entertaining.

The problem that Bloch reports here reminds me of a bug that I encountered once during one of my graduate classes. I was taking a parallel programming class in which we were doing a fair amount of coding with various parallel programming environments. One of these environments provided a barrier synchronization primitive. The implementation of this primitive was heavily optimized to be efficient with a large number of processors. Since a large part of the grade in the class was based on our ability to write code that was demonstrably efficient, my code made use of this primitive. The problem was this: I started noticing some weirdness in my benchmarks, so I started inserting more and more self-checks into my code. Eventually I came to the inescapable conclusion that the barrier synchronization primitive had some bug in its implementation. At this point I had no choice but to implement my own barrier synchronization primitive and migrate my code to use my implementation. When I reported my problem I (understandably) got a skeptical reply. I had very little time to look into the details of the problem, so I continued to use my own barrier synchronization primitive instead of the system's. My implementation was definitely slower than the system's, but I simply could not use the system's anymore.

After a couple of weeks of existing in this state, I felt a good deal of vindication when I heard my fellow classmates complain in class that they thought that the barrier synchronization primitive had a bug. At this point the professor who ran the class gave us the code for the system's underlying barrier synchronization primitive. I really wish I could report that I (or one of my classmates) had an "a-ha!" moment with the code at this point...but unfortunately this never happened. I put around three hours into trying to track down the bug in the system's highly optimized barrier sync mechanism, but given the large amount of hours I was putting into $DAYJOB at the time, three hours was all I had to give to finding this problem. As far as I know, nobody ever found the actual problem here. I always felt a little badly about this, but there was nothing I could do about this problem.

Despite this specific incident, I have had a good deal of success in debugging thread/synchronization bugs over the years, and the one thing I can say about this class of problem is this: it is really satisfying to find and fix a problem in this area!

YO DAWG I HEARD YOU LIKE THREADS

I'd probably find this to be funnier if...oh....nevermind....

ascii2keyboardscancode

Here is a short/goofy/useful script that others might find to be useful: ascii2keyboardscancode .

This is a quick hack that I threw together for a specific and cool little project I have been working on. In very high level terms, it is possible to use this script to drive VirtualBox's "headless" interface, using this CLI interface:

    VBoxManage controlvm some-machine keyboardputscancode ...

Using all of this, it is possible to create a new VM after some nightly builds run, use Kickstart to create a new OS install, shutdown the VM, transmorgrify the VM into various VM formats (.ovf, etc.), and otherwise automatically deploy the shiny new VM into a test harness.

The very best part of all of this is that everything happens unattended in the wee hours of the morning, when CPU time is cheap. By the time everybody arrives at work in the morning, all of this tedious work is done.

I love it when a plan comes together!