30 June 2014

Judo In Security Engineering (first story)

One day at {dayjob}, I decided it was time to tell the VP of Engineering about my concerns regarding the product's security.  Specifically, I was concerned about the security of a part of the product that I was helping to design+implement.

We were starting to get some enterprise and government interest in the product, and I wanted to have a sensible security scheme in-place before we suffered any embarrassment.  We still hadn't released the product out into the wild yet ; the time was right to implement the security measures I had in mind.

So, I had a conversation with the VP of Engineering, gave him an overview of my plans, and got him to agree to my plans.  He allocated space in the schedule for me to work on this project, and he even allocated two of my co-worker's time in order to complete the project.

So far, so good....

The simple fact of the matter, however, was that we were working at a fairly crazy pace at {dayjob}, working towards the FCS of this product.  Although the VP had given me some time in the schedule, he didn't give me much....something on the order of maybe 4-6 days, total.  I had to be very strategic.

The security scheme that I had in mind involved some well-known cryptographic tools.  I was not looking to invent anything new here.

So, I sat down with my two co-workers to go over the plan.  This is where things got....interesting.  One of my co-workers {PracticalGuy} was a little more junior than I was, but very smart and very capable.  The other co-worker (MathGuy} was one of the most senior engineers in the company -- far more senior than I was.  As you might be able to tell, {MathGuy} was very mathematical in nature.  Honestly, he seemed to be really good at mathematical analysis, and frequently his analysis was a bit over my head.  {MathGuy} was also very capable.

Anyways, I described the problem to my co-workers, and the plan that I had come up with to address the problem.  Part of my plans involved using some of the crypto tools from {OpenSSL / libcrypto} in our product.  {PracticalGuy} came up to speed on what needed to be done right away.  The interesting part of what happened next had to do with {MathGuy}.  As soon as I said "OpenSSL library" {MathGuy} strongly objected.  I didn't know a huge amount about OpenSSL at the time, but I knew enough to understand that this library could be a bit tough to work with.  However, this really had nothing to do with {MathGuy}'s objections.  He stated "OpenSSL is a big complicated library, and I don't know how every single line of code in this library works.  I'd prefer to write my own crypto code.".

Wow.  I did not see this coming.

I tried to reason with {MathGuy}.  Obviously, the first two things I pointed out to {MathGuy} were (1) the schedule and (2)  the number of man-years involved in the design+implementation of {OpenSSL / libcrypto}.

"I don't care!" stated {MathGuy}.  We argued for a few minutes....it was a fairly crazy argument that went sort-of like this:

MathGuy:  I don't think that it would be very hard to write my own libcrypto.

Me:  do you understand how many man-years have gone into this library?

MathGuy:  I always wanted to write my own Diffie-Hellman.

Me:  we don't need Diffie-Hellman for this project.

MathGuy:   I think it would be better if we used it.

And then {MathGuy} ushered me away, telling me that he was going to get started on the project.  I left this conversation with little confidence that {MathGuy} would contribute to this project in any meaningful way.

A few weeks later, me and {PracticalGuy} started on the project in earnest.  {PracticalGuy} brought a lot of capabilities to this project that I didn't have, and I was truly grateful to have him working with me on this project.

I did try on one occasion to get {MathGuy} back on-track and into a mode where he could actually help us on this project.  I stopped by his cube and tried to get him excited about one aspect of the project that needed attention.  "I've got no time to help you with that problem!" said {MathGuy}, "I'm in the middle of designing my own BigNum library, and I am swamped with issues right now!".

As soon as I heard the phrase "my own BigNum library" I knew I was in a pickle.  I couldn't exactly storm into the VP of Engineering's office and tell him about this crazy situation....after all, {MathGuy} was much more senior than I was, and seemed to have much more political weight as well.  I couldn't get {MathGuy} to work on anything practical either.

So, I did what I could do.  I made sure that {MathGuy} was working on his own isolated branch in source-control.  I also continued to work with{PracticalGuy} on the security project, and we were able to complete the project with great  success.  And...when the VP of Engineering asked me how the project was going and sort-of forced me to account for what {MathGuy} was working on, I told him "well, {MathGuy} has his own ideas about this security project, and he is working on those ideas, but I'm skeptical that those ideas are going to pan out.....I think that he might be looking for something else to work on....could you maybe find some other work for him to work on?".

I never really checked on what was going on out in {MathGuy}'s isolated branch in source-control.  I sort-of assumed that the whole thing fizzled out.  This was simply the best I could do, given the situation....

22 May 2014

My Take on Aaron Maxwell's "Use the Unofficial Bash Strict Mode (Unless You Looove Debugging)"

I enjoyed Aaron Maxwell's Use the Unofficial Bash Strict Mode (Unless You Looove Debugging).  Read it -- seriously.  Mr. Maxwell is clearly somebody who knows what he is talking about.

I'm a shell hacker too.  My fascination with the shell has....evolved over the years.  I no longer reach for the shell as my first tool of choice...I more have the following heuristics in my head when I approach a task:
  • If a program can be written in under a page of code, then a shell script is a perfectly reasonable solution to a problem.
  • If a program can't be solved in under a page or two of code, it is time to think about writing the program in Python/Perl.
  • If a program basically has to fork+exec a bunch of other external programs, even if the program gets to be big, it might be just easier to keep the program implemented as a shell script.
  • But...if a shell-script evolves to be more than five (5) lines of code (or maybe even less...), then the shell script HAS TO be written according to the following template that I've come up with.

My Template

Here is my template for a reasonable shell script:


#!/bin/bash

#################################################
function usage() {

cat <<EOF
Usage:  $0 [OPTIONS]

OPTIONS

  --blah
        Allows you to specify the "blah" value.

EXAMPLE

  $0 --blah 1.2.3.4

EOF

}

########################################################
function cleanup() {
    echo "Cleaning up"
    rm -rf "$TMPFILE"
}

########################################################
function errHandler()  {
    echo $0: some unexpected error happened.  1>&2
    cleanup
    exit 1
}

########################################################
main() {    
    IFS="`printf '\n\t'`"

    set -E    # "set -e" is good.  I think that "set -E" is better.
    set -u   # exit if a reference is made to an undefined variable
    set -o pipefail   # exit if there is a failure somewhere in a pipe
  
    trap errHandler ERR



    # Let's say that somewhere in this program the code needs to
    # use a temporary file.  Well, let's have a consistent name for
    # this file, so we can ensure that it gets removed if this program
    # fails for some reason.
    TMPFILE=/tmp/some-temp-file.$$

    echo "blah" >"$TMPFILE"

  

    # Just for the sake of an example, here we're going to
    # introduce the chance for an error to occur.

    echo "The next operation may or may not fail.  Stay tuned..."
    echo

    [ $(expr $(date '+%s') % 2) -eq 0 ]


    cleanup
    return 0
}

main "${@}"
exit $?



Let me explain why I write shell scripts in this way:

#!/bin/bash

I know that some old-time Unix programmers might scoff at this.  Somebody might say "but the Bourne shell (/bin/sh) is the One True Shell".  My response:   I'll be glad to make my shell script ultra-portable when it needs to be.  Until then, nearly all of my shell-script code simply has to be portable to Linux (and occasional Mac) systems....and these all have Bash available for them.

I'm explicitly using the GNU Bourne-Again SHell here.  I do not care to write code that invokes /bin/sh but actually depends on /bin/sh being the Bourne-Again SHell.  Code like this dies when /bin/sh is actually Dash.  If somebody thinks that I am making a mountain out of a mole-hill here, I would remind them that Dash is the default /bin/sh on Debian/Ubuntu systems, and I don't want code that I produce to die strangely on such platforms.

I have no problems with using the original /bin/sh, "ash", or "dash", and I happily will use these if the situation is appropriate.

usage()

Putting usage information at the top of the file seems to allow me to provide documentation and useful code commentary with a minimum of duplication.

cleanup()

Many programs like this create temporary files, or some other thing that might need to be cleaned up.   I try to put all of my cleanup-related code in this function, as opposed to sprinkling code like this everywhere in the code.

errHandler() / set -E / trap errHandler ERR

This is how this program handles unanticipated errors. 

"set -e" and "set -E" are very similar, but, in my opinion, "set -E" is better because the "exit if something goes wrong" behavior is applied to subshells and functions.  I prefer to have Bash executing in this mode as much as possible -- I don't want this behavior to (silently...) not occur just because my code happens to be running in a function.

Notice how the errHandler() code properly calls cleanup() before it exits.

I can say a lot about Bash scripts that are written in this manner -- they exit if some unexpected error occurs.  Writing code in this manner is sort-of like employing a poor-man's exception-handler.   In my opinion, the following two points are very important to understand about code that executes in this configuration:

  • It does take some more effort to produce code that executes in this context.  The code needs to be written in such a way that every possible non-zero exit value is properly trapped.
  • On the other hand, code that is written in this manner is much more reliable than code that is written in the default mode.  As a programmer who is only interested in producing reliable code, I spend nearly zero time tracking down bizarre problems in code that is written in this style.   If the shell is not configured in this way (the default mode...) then you can pretty much guarantee having to track down a bizarro problem at some point in time.

main()

This is my invention.  I write code in this way because I've worked with hundreds of shell-scripts over the years that started out as some small bit of code...and then grew...and grew....and grew....and eventually became a steaming pile of top-level code plus functions plus more top-level code.  I've worked with some 20000 line shell-scripts in the past in which I had a hard time even figuring out where the code started executing.  Working with code like this causes me to want to wash my eyeballs in bleach afterwards.  I have little patience for this sort of thing: the code that I produce isn't going to suffer from this problem.

Please also notice that this code uses functions to group together related bits of code.  I have seen lots of shell-script code over the years that is just page after page after page of code...with no functions.  Code written in this "style" ignores basic software-engineering practices.  Yuck.

Epilogue

This shell-script template is my personal style.  I don't always write shell-script code (I'm more of a fan of higher-level languages...), but when I do write shell-script code I try to employ every reasonable technique that I can to help ensure that the code is clear and reliable.

In my opinion, the "errHandler() / set -E / trap errHandler ERR" part of this template is critically important.  I can think of several large shell-scripts that I have been called upon over the years to maintain that have benefited greatly from this scheme.  When I first started working with these large programs, the programs were HUGELY UNRELIABLE, regularly spewing errors and leaving systems in many inconsistent states.  It cost LARGE AMOUNTS OF TIME AND MONEY to fix systems that were left in an inconsistent state because of these problems.  After I started maintaining this scripts, it did take me a good deal of effort to update these unreliable programs to be written in my preferred style.  But, after this effort was over, the enhanced reliability has paid HUGE DIVIDENDS......I no longer have to chase boffo problems in these areas.

It is seductively easy to write shell-scripts.  But, the results are a lot better if some basic, common-sense rules are followed.

10 April 2014

Protocol Design

I agree with what Dan Kaminsky wrote:
Protocol design is miserable, nothing is elegant.
I do like working in the area of protocol design+implementation.  There are a lot of interesting problems to work on in this space....problems that fully engage your mind if you care about doing it well.  But...wow...it can be a frustrating area to work in too.  If you are not on top of your game 100% of the time while doing protocol work, then the results always seem to come back to haunt you.

I think I'll make another donation to the OpenSSL folks now....

31 March 2014

Debuggers

Recently, I enjoyed reading Ellen Ullman's book The Bug.  I picked this book up a long time ago and just hadn't gotten around to reading it until now.  Overall, I thought this book was interesting and worth the time that I put into reading it.

Here was one of my favorite sections:

Better fix that bug, Harry had said.  William Harland is acting like he never saw a bug before.  The VCs are watching.  Tomorrow he was supposed to meet with Harry about the schedule.  He could not stop and simply leave a note for himself.  Tomorrow there would be no "Here you are, Ethan."  No message saying, "Investigate pointer indirection."   No, the only note he could leave himself could be the bug report itself, with its message to the testers saying, "Fixed."

Step, step, step.

Some part of him knew that he should get away from the debugger.  He should get away from the machine, stop and think on a yellow pad, a whiteboard.  He wasn't making headway like this.  He kept beating against the same certainties--here, else here, else here.  Writing and sketching might break his thinking patterns, force him into other channels.  But there was something seductive about the debugger:  the way it answered him, tirelessly, consistently.  Such a tight loop:  Step, he said.  Line of code, it answered.  Step, line of code: step, line of code.  It was like the compulsion of playing solitaire:  simple, repetitive, working toward a goal that was sure to be attained in just one more hand, just one more, and then one again.

And so the paradox:  The more the debugger remained the tireless binary companion it was designed to be--answering, answering, answering without hesitation or effort late into the night---the more exhausted and hesitant the human, Ethan Levin, found himself to be.  He was sinking to the debugger's level.  Thinking like it.  Asking only the questions it could answer.  All the while he suffered what the debugger did not have to endure:  the pains of the body, the tingling wrists and fingers, the stiffness in the neck, the aching back, the numb legs.  And worse, the messy wet chemistry of the emotions, the waves of anxiety that washed across him, and then, without warning, the sudden electric spikes of panic.

I think that any decent programmer has a very good idea of what is going on in the character Ethan Levin's head.  Every good programmer knows what it is like to struggle with an elusive bug.

....

With regards to debuggers, I consider these to be a useful tool in my toolbox.  Some programmers who I hold in high regards really dislike them (rather famously Mr. Torvalds).  I respectfully disagree, but I think I understand some of their perspective.  I believe that my occasional use of a debugger is prudent.  For example, in my past I tracked down a stupid bug that caused me to stay at work late one night by strategically using a debugger.  My strategic use of the debugger in this case helped me find the root cause of a bug in minutes.  Without this tool I would have had to have spent another few hours tracking the problem down.

On the other hand, I do recognize that a debugger can be a seductive tool that doesn't always help with solving a problem.   A debugger can certainly get in the way of understanding what the actual problem at hand is.

All of this makes me recall a story in my past...

....

One day at {dayjob} I found myself in the following position:  the software product that our group was producing was in its end-stages of development.  There were maybe another 10 blocking bugs left to resolve before we could declare that the project was done.  It had been a long slow grind to get this project to this point.  The finish line was just becoming visible.  We HAD TO make this upcoming deadline.

Of these 10 blocking bugs, most were straightforward.  However, one of these bugs was elusive and strange and difficult to reproduce.  The bug itself reared its ugly head (on average) only once per week, and always at inopportune times.  A full system restart was the only way to clear the system of the problem.  In the office we referred to this bug simply as "System Haywire".

My role in this project was mostly to write code, but I also had a leadership role in this project too.  The official project leader was unfortunately in the hospital, and so I found myself leading the team towards the finish line.  "Okay, I can do this", I thought.

So, I managed the project, assigning engineers to work on the remaining bugs, and helping out where I could.  But, there was still the matter of "System Haywire".  This problem was frustrating because hardly anybody could reproduce it.  It is tough to fix what you can't reproduce...

One day at one of our status meetings, one of our junior engineers mentioned to me that with his slightly non-standard setup, he was frequently experiencing a weird error that caused him to have to restart.  The more he described what he saw, the more convinced I became that the problem he was experiencing was, in actuality, "System Haywire".  "Oh, is that what this is?" he asked.

After the meeting was over, I went over to this engineer's office and asked him if he could show me "System Haywire".  "Sure", he replied.  Within ten minutes, I was looking at a totally haywire system.

This is where things got strange.

I was happy that we were finally able to reproduce the problem.  This was very good news.  So, I asked my co-worker, the junior engineer, if we could restart the whole system and try it again.  "Okay..." was his response.  So, we restarted.

Now that the system was restarted, I asked my co-worker if he could enable all of the system's debug logging.  "Huh?  How do you do that?" was his reply.  I thought it was a little bit weird that, at this late stage of the project that there was still somebody working on the project who didn't know how to turn on debug logging, but I rolled with this:  I taught my co-worker how to do this.  He went through the steps impatiently.  I also asked my co-worker to enable a protocol trace for the networking protocol that was being employed here.  Again, there was more befuddlement and frustration from my co-worker.  "Please be patient" I implored, "this is going to help us find the System Haywire bug".  My co-worker replied "I don't see how".

We had spent around 15 minutes setting everything up here, with me explaining various steps.  My co-worker was becoming more and more exasperated with me.  Eventually, everything was setup, so we started up the system.  It took a few minutes of futzing around with the system in order to get the system back into the System Haywire state.  My co-worker seemed to be really annoyed with me at this point.

It happened.  System Haywire reared its ugly head.  Yipee!  I started the process of trying to collect the logfiles....

At this point, my co-worker was becoming unglued.  He did not want to help me with getting the logfiles and the protocol trace off of the machine.  I implored "look, we're nearly done here -- all I need are the logfiles.  It'll just take another five minutes".

For a minute or two, my co-worker relented.  We got one of the logfiles back to his local machine.  I asked my co-worker to "load up" the logfile just so we could take a quick look at it.  He shrieked "HOW DO YOU EXPECT ME TO DO THAT?".  I responded "well, load it into your text editor -- it's only a couple of megs big".  He replied "DO YOU MEAN MY IDE?".  "Yeah, your IDE, I guess" I replied.  Something that I couldn't quite put my finger on was confusing me here.....a programmer didn't know how to look at a logfile?  What....?

My co-worker's IDE loaded up the text file and the logfile was on the screen.  Standard logfile sorts of stuff:  [time] [subsystem] [some message].   "Good, we got the output", I said.

My co-worker exploded at me.  A full 30 seconds of vitriol and shouting:  "[expletive] [expletive] [expletive] THIS IS NOT HOW YOU [expletive] DEBUG A [expletive] PROGRAM.  YOU LOAD THE PROGRAM INTO A [expletive] DEBUGGER AND YOU START IT UP AND PUT BREAKPOINTS INTO THE [expletive] PROGRAM.  THAT'S HOW YOU [expletive] DEBUG A [expletive] PROGRAM YOU CLUELESS [expletive] TWIT" was one of the things that he screamed at me.

I tried to reason with my co-worker "Look, this doesn't seem to be the type of problem that a debugger is going to help us with.  There are no memory problems here, and the system isn't crashing -- it is going haywire.

He continued to shout at me.

And this is when the following fact hit me like a ton of bricks:   my co-worker and I came from completely different worlds.  The only way that he knew how to debug a program was from using the debugger in his IDE.   He had no capacity to debug a program in any other way -- by looking at the code, working things through on a whiteboard, looking at a logfile, analyzing a protocol trace, for example.  I was fairly stunned when I made this realization.

But....I still desperately needed those logfiles.  My co-worker was still shouting at me, so I decided to yell back at him.  I told him "LOOK, I DON'T CARE IF YOU ARE UPSET.  I WANT THOSE LOGFILES....NOW.  GIVE ME THOSE LOGFILES AND I'LL GET OUT OF YOUR HAIR.  I'M NOT LEAVING UNTIL I GET THOSE LOGFILES.  RIGHT NOW.".

So, after a few tense minutes in which I watched my angry co-worker's every single move (because that's how important these files were to me), I got the logfiles transported back to my computer for further analysis.  As I left his desk, my co-worker told me that he was going to delete his copies of "those stupid [expletive] logfiles right [expletive] now".

Wow.  I did not need this in my day.

.....

Anyways, after I got back to my desk, I spent the next six hours looking through the logfile, the protocol trace, and thinking my way through the code.  Eventually, I found the problem....well, two problems, actually.  One of the problems was a protocol error.  The other problem was an extremely subtle problem in the code.  The fact that there were two problems at hand here was one of the things that had made "System Haywire" such a beast of a problem to find and reproduce.

The "System Haywire" problem was fixed, and soon after we shipped.

....

I still consider a debugger to be a useful tool to have in one's toolbox, but I think that it is important for a software engineer to have other tools in there too.

17 February 2014

Lily at Trailside

My heart melts when I watch this cute little dog named Lily.

16 February 2014

UNH needs a $25M stadium like I need a hole in my head

I was disappointed to read the following news release from my alma mater:


January 31, 2014
UNH Outlines Plan for New Athletic Complex as Part of Campus Master Plan
DURHAM, N.H. – With a focus on meeting its strategic priorities, the University of New Hampshire is moving forward with fundraising and planning for a new athletic complex that will benefit the university, Durham and the Seacoast region, and the state. Athletic facilities and venues, including a new stadium, were identified as a priority in the campus’ master plan to improve recruitment, student satisfaction, and overall visibility in an increasingly competitive marketplace.  
Expected to cost $25 million and open in late 2015, the facility, which is referred to as the West Stadium, will offer increased and improved seating, state-of-the art broadcast and WiFi capability, concessions, restrooms, and a special student section. The original concrete section of the current East Side facility dates to the 1930s.
....
 The project will only move forward when the university raises $5 million in donations and the final project cost is no more than $25 million. The university plans to seek internal borrowing for the additional $20 million. The cost of this project will be absorbed within the university’s current five-year financial planning model.

I travel through the UNH campus often enough, and when I do, I feel more and more like a stranger in a strange land.

I love the new (but not very new anymore) library on campus.  Same thing about the new engineering buildings.  I understand that one of the main liberal arts buildings is going to be replaced soon too -- cool!  I spent a lot of time in all of these buildings during my studies.

But there has been a lot of other new construction on campus too...stuff that I've never understood.  Like for example, I see a lot of high-end housing.  Suites for students to live in -- much nicer than my first apartment that I rented when I got my first job after college.  The new ice arena is nice, but I voted against it as a student a long time ago.  I've been told that there are classrooms in the new business school that have over-the-top multimedia stuff in them, complete with many LCD screens and a microphone system that automatically tracks who is speaking in the audience (yeah, I'll bet that students learn in this classroom 100x better than I did back in a more austere classroom...).  And now....this new $25 million stadium.

Here is why I am not wild about things like a new stadium:  this doesn't fit into my idea of what a state school should be.  I think that state schools should exist to give students a good education for a reasonable price.  I believe that frills at state schools should be kept to a minimum.

The new (but not very new anymore) library at UNH is great.  I've always looked at the study areas attached to the library, complete with their decent lighting and thought to myself "I would have killed to have such a nice area to study in when I was a student".  Putting money into stuff like this is a reasonable use of money.

But....things like high-end housing options and a brand-new stadium?  In my opinion, these things are not necessary for UNH.  Sure, UNH should have an athletic department, but its scope should be more limited.  This idea to put up a new stadium for a $25 million dollar price-tag is ludicrous for an institution like UNH.  If some student/family/consumer wants all of these things, then, in my opinion they should look to one of the many private schools that offer all of this stuff.


Why do I think this way?  Well, I'll tell you why:  when I was a high-school senior, I had a simple choice to make:  attend UNH or a higher-priced, more "prestigious" university.  And, one thing that weighed heavily on my mind was the complete understanding that whatever choice I made, I could not expect my dear parents or relatives to pay for my choice -- that just wasn't in the cards.  I was able-bodied and I knew how to work (thanks Mom and Dad!) and how to make money -- I knew that the entire cost of my higher-education would be mine to take care of.  So...I considered my options....and I chose UNH.  I worked my tail off to pay for my education and I am happy with the choices that I made.

But...here's the thing....every time I travel through UNH's campus lately, I am struck with the realization that, all things being equal, if I was a high-school senior today, I probably wouldn't be able to afford attending UNH.  And...I think that this is a real shame.