Open Source's Bleeding Heart


How did we get here?

Well, it's been a bit more than a week since Day 0 in the Heartbleed Bug.

What's Heartbleed? No, really.

In case you haven't heard too much already, last week the Heartbleed Bug was publicly discovered in the Heartbeat function of recent versions of OpenSSL.  The request for a heartbeat response could be specially-crafted to have OpenSSL return not a "I'm running correctly!" message, but instead "Here's what I'm thinking about" message - i.e. current memory contents.

Given that OpenSSL is responsible for encoding and decoding all your secrets in conversations with webservers (and a few other types of server), knowing what it is thinking equates to knowing its secrets.

That's a problem, and a meta-problem.  First, you might reveal secrets like usernames, passwords and bank account details.  But, a layer deeper, you might also reveal the secret keys to certificates which are used to encrypt the conversation in the first place - so you might both get secrets, and the means to get more secrets.

It's a disaster.  The worst security bug The Liberator has ever seen.  As I write, there's still thousands of unpatched servers, and teams of script kiddies trawling the Internet looking for secrets off vulnerable machines.

Wherefore Public Disclosure?

It is also, to The Liberator's mind, massively under-covered.  In some cases, the coverage is outright misleading.  On Day 3, Google announced that its users did not have to change their passwords.  But Google - your servers were definitely subject to this bug, and you have NO IDEA if user passwords were stolen.  Yes, you've patched and re-issued the certificates - but you cannot honestly say nothing bad happened - this bug was in the wild for 2 years, after all.

Other organizations just sit schtum, perhaps preferring to 'Let it all blow over.'  Perhaps, if they're silent, people will continue to trust them implicitly, and none of the fish will get spooked.  Take Facebook, for example.  You can see that Facebook uses Open Source technology for its servers, but its OpenSSL is not currently vulnerable.  Does that mean it was never vulnerable, or did they quickly switch to a different version?  It would be great to see a public statement on the topic - like fellow Open Source consumer eBay has done.  Well done, eBay.

Servers running Windows were completely immune - Microsoft has their own SSL stack independent of OpenSSL.  Hotmail is safe.  Unfortunately (how dare The Liberator say this?) most of the web is running GNU/Linux, not Windows.  Hotmail is the only big service I've checked on Windows.  Google, Yahoo!, eBay, Amazon, Facebook, CIBC, RBC, CRA (Where's my Social Insurance Number, now, Canada?), etc, etc are all potentially-susceptible.

A Lesson in Redundancy

It was many years ago that The Liberator designed high-availability systems for part of a service offering by a predominantly-software engineering firm.  The foundation of the availability calculations was often redundancy.  Yes, one unit might have a low reliability, but two of them will have a much higher reliability.

Software, however, does not generally fail by random occurrence.  It doesn't have a bearing buried deep in a fan assembly that can eventually grind to pieces and cause a complete cooling malfunction.  Software has no moving parts.  It doesn't get old or wear out.  When software fails, it is mostly due to a bug.

Bugs have entry conditions.  Heartbleed's is a special request.  At my old company, it was a malformed datestamp broadcast by another machine.  Every single system we had connected crashed simultaneously, and kept crashing that whole day.  On the failure analysis diagram, just because there were two installations of our software on two machines did not create two fully-redundant systems - we had two redundant, independent machines and a single-point-of-failure software installation.

So, if you want real redundancy, don't forget your software.  That will take some real care, particularly for embedded systems.  For example, you won't have software independence in a firewall pair from any particular vendor - they'll both have the same OS and rules engines.  Don't use the same hypervisor for every VM in your organization.  Don't use the same OS for every server you've got.  Don't use the same web browser or SSL decryption engine.  If you want to protect your system and have the best reliability, create some heterogeneity. 

What About Free Software?

Well, here's where The Liberator must fall on his sword.  The Heartbleed Bug definitely affects MYRA's flagship Free Software OS - Red Hat 6.  MYRA's servers were exposed because of this bug - although quickly remediated.  "Many eyes make all bugs shallow," we paraphrase Eric Raymond, famous author of The Cathedral and the Bazaar.  But this bug, this huge security hole, got right through the original upstream community, through the Fedora testing-grounds, through Red Hat QA and went Production for years before eventual discovery.  Yes, Red Hat released the patch within hours on Day 0.  But the bug was already in the wild, and it was really, really serious.  That was all wrong.  The Liberator has soul-searching left to do.

Once More, Software Minimalism

Regardless of other conclusions on the best path for the future, one principle is highlighted - lines of code equals a surface to attack.  More software, more bugs, more maintenance, more vulnerabilities.  The Liberator occasionally sees customers installing GUIs on servers - don't do it!  You don't want another billion lines of code just so you have a checkbox.  If you don't need a feature, don't buy it.  If the feature comes free (or even Free), delete it, disable it, uninstall it.  Adding more useless cruft to your environment will just find you up this same creek some day in the future.  All software has bugs.  The less complexity in your system, the better.  Minimal is elegant, reliable and maintainable.  With bare installations taking up just 300MB of disk space, RHEL is still good at that, at least.