I was talking to an old friend at the recent Mobility Day held at the Microsoft Office near Boston and he brought up an incident that I have seen happen to others. I realized though that it isn’t something talked about often so it seemed like perfect blog fodder.
He told me of working with a large bank in Boston (that doesn’t really narrow the list down) where outsourcing was literally a requirement based on the budget. The code for the bank system was developed by a Russian firm that showed great talent. Unfortunately they also showed great talent for deciet. The code delivered had 3 backdoors in it that would have allowed easy access to account data and possibly to money. After ripping out the offending code after doing a very wise line by line code review the system was deemed safe. How often has this happened without it being caught? The X-Files premise, “Trust No One“ is actually correct. I don’t mean to indicate that only off-shore firms would do this, quite the contrary, but I think the odds go up based on how subject to prosecution the developers would find themselves if discovered.
This also brings up what I think is the biggest fantasy of all. The one that asserts that open source code is inherently more secure than commercial software. We have examples from the last 12 months where some of our selfless open source contributors were not so selfless after all. It should be no secret based on the main subject of my entire blog that I think that security is the place where all the action will be in the next 5 years. This translates to where all the cost will be as well.
My point is that you must truely Trust No One. If you decide to use open source because it is cheaper then you are deluding yourself unless you include the cost of doing a complete, line by line code review before implementing it. The advantage of using commercial / proprietary products is that if you buy it from a company and you make sure it is one that you can sue for enough money to matter if they put in a backdoor, then that is your hedge against the threat. Always ask yourself the question of what is preventing this developer from putting in a backdoor.
Point well taken, I’m sure.
Having been on both sides of the custom software development fence I couldn’t agree more; ‘Trust No One’, as a philisophy is simply prudent.
I do, however, take some exception to your leap from your experience with a custom software development company to targetting open source software in general.
First off, I find the argument to be counter-intuitive. Your mantra of doing line-by-line code checks on the software that you plan to use cannot so easily be used to increase cost estimates on implementing open source software. In a closed source world, sure you’d save the time up-front (due to the fact that a line-by-line code review is simply impossible) but I think you’re deluding yourself to believe that this fact is making this software any safer.
Speaking of costs; last time I checked (post .com era) lawyers were more expensive than programmers. Also, the time it takes, on average, to bring a civil suit against a large corporation (not making any mention of the dreaded ‘class action suit’) outwieghs the time taken to code review on a fairly substantial portion of code. The only difference is you can be relatively sure that your code is clean (given proper review process), the same cannot always be said of the process of the courts.
My point is, I agree that one should, in essence, trust no one. However, we have seen at least as many examples of destructive habits within closed source software as are seen in alternatives. Due dilligence should be a factor in all decisions. I would warn against using open source software found in dark corners of the web ‘site unseen’, but I would also warn the same for closed source.
Thanks for a great article and the opportunity to respond.
Mike Mullen
Open source projects that are in wide use and mature should raise no more alarm bells than their closed source compatriots.
If that bank was going to do a line by line review, what is the point of outsourcing at the first place. If they did not trust that company, then why give them the project. This simply proves that the IT group at that bank is bunch of idiots.
And if this code was not open source, they would have never been able to figure out that there is some security issue with the code. Open source code is not that bad if you know your stuff well.
I can review someone else’s working code in a slight fraction of the time it takes to design, develop and debug an application into working. I am suggesting that you shouldn’t outsource anything beyond those you trust unless you are willing to do a line by line check. Sometimes you should have things written by your own employees checked line by line. Only the paranoid survive is an often proven tenant. That being said, there are objects like operating systems that require too much code for anyone to check line by line. In these cases we fall back on the fact that the provider is an entity vulnerable to lawsuit and therefore that provides our insurance factor. You typically can’t sue or even locate all the contributors to a large open source project which means not doing a line by line check and not having a culpable target to hold responsible means accepting the risk (alot of it in many cases) on yourself.
Thanks
Patrick
I wonder how many companies that outsource either consider the issue of code inspection, or require it as part of the contract. Perhaps as importantly, how many firs consider this on code developed INTERNALLY?
We all know that outsourcing, in general, can bring great benefits – you give things you are not good at to another firm that knows how to do it well, thus saving money for your, and making a profit for them. But the idea that the company you out source to is completely honest is absurd.BUt then, so is the notion that all your employees are honest!
My view is that all user input is evil unless proven otherwise. No matter who the user is!
Thanks for a great post that made me think!