One of the phrases I hear fairly often from the open source community is that “Many eyes make better code” and it confounds me that people actually believe that the quality of code, and thus the quality of the application, is simply a function of how many people view the source code. Before I give the impression that this is a screed against the concept of open source software, allow me to assert that I don’t have an ideological preference one way or the other towards open source, though I find zealotry of any form to be tiring and frustrating. For finished apps its great that the application is free, I like free, but it really delivers no value to me if I can change the source code because frankly I am not going to want to modify the source for my word processor (is that term still in use today, or is it an anachronism from the 80s?). For code that I consume in conjunction with my own code, say a .Net library or a PHP application that I want to use on my website, there is value to me in being able to freely alter the code as well, but only if I don’t have to worry about any restrictive licensing (BSD style license = good, GPL = pain in the ass if used for anything related to work).
That said, I find some of the common open source sentiments silly, the “Many eyes” argument one of the silliest. Essentially what the “Many eyes” argument suggests is that quite a few more people are doing code reviews than you would have in a closed source application, which for the most part is probably true. There are two counterarguments that I haven’t seen addressed. The first is that having many eyes look at source code doesn’t mean a thing by itself, if you don’t qualify the experience and expertise of the many eyes, nor does it mean anything if the analysis isn’t systematic. Five really sharp closed source developers will be more effective reviewing code than one hundred amateur open source developers. They are more likely to be scrutinizing the trouble areas, and they are more likely to spot issues in those areas.
Code reviewing is an art. A reviewer is looking at things line by line, but has to retain knowledge of all the code that impacts a particular line of code. It’s easy to spot use of a banned API, potential overflow conditions, bad arithmetic or logic formation, etc, but in most cases static code analysis tools are faster and better at spotting those sorts of things. Where a human has an edge over the tools is seeing larger code pattern flaws, questionable but legal use of certain API, missing validation, and code traversal issues, but as a qualifier, the humans that are good at spotting those problems are a rare breed. They can focus on many things at once, recall pertinent details from code many pages before, and instinctively spot worrisome code. It isn’t clear to me that an open source project attracts any more of these experts than the equivelent closed source project, at least on average. So I believe that open source can summon more eyes for code review, but I haven’t seen evidence that those many eyes actually were thoroughly scrutinizing the risky areas of most OSS applications, nor have I been convinced that most OSS applications have attracted more skilled eyes.
That point is pretty irrelevant with my second counter argument: code review is not the key to robust applications but rather just one component. They key to robust applications is project organization. Regardless of whether a project is closed or open source, to be secure, to be stable and robust, the entire development process needs to be structured and thorough. During design time features need to be decided upon, experts in given areas (UI, database design, network design) need to spec out their given areas, developers need to create development specs to reflect the design specs, and then security reviews need to be done on the design. Also during design time coding guidelines need to be established and key project, development, and test managers need to be established (even if it is open source, you need specific goto people to oversee things). During development code needs to be reviewed to ensure the coding guidelines are being followed, that the implementation reflects the design and if not (for a good reason) the design documents need to be updated. Testing needs to start as features are initially implemented, not once code complete is called. Testers need to be systematic and need to act as a further sanity check that the implementation matches the intended design. They also need clear areas of test assigned to them to ensure even coverage and no unnecessarily redundant effort at the expense of untested code. Through each phase, design, development, and test, there needs to be security checks. If the first security work is done when the first beta is produced the application is going to be full of vulnerabilities.
A thorough, organized development effort is what makes good code. Some open source projects have this, for example the linux kernel, the apache web server, and arguably the mozilla/firefox teams now, though in this latter case it is very clear this wasn’t always so. Most people forget that when Mozilla 1.0 finally released, despite being several years behind schedule, it was an utter failure, so much so that the Pheonix (then firebird, then firefox) team forked the project just to try and makes something that didn’t suck.
A great deal of open source projects do not get this, they don’t get that they need to be as organized and managed as any closed source project. Its obvious that coding standards either didn’t exist, were incredibly loose, or just weren’t enforced on many open source projects. For example just download any PHP application; phpBB, Wordpress, phpNuke, etc all have very unprofessional code. There are very few comments, if any, in the code, extensive use of nested arithmetic in function calls, little data validation and error checking, use of tertiary operators for complex assignments (they are meant for simple assignments), etc. I can only imagine that if coding standards weren’t followed the rest of the development process similarly lacked hallmarks of a well run project.
There are many closed source applications that similarly lack this organization. Many small companies aren’t organized a whole lot differently that small open source projects. Developers hate the bureaucratic pain in the ass steps that a rigidly run project makes them perform. At small companies it is common for the devs to just not follow those steps, because no one is there to force them to. I commonly hear “Why do I have to write a document describing what my code is going to do when I could just write the code and show you” concerning developer specs. They don’t realize or care that it is a lot easier to read an english description of something and figure out how it works than it is to look at the code, even for the most competent coders. Its also a lot quicker to write and correct an english (or chosen spoken language) outline than to implement everything and make changes after the fact.
Even very large companies aren’t guaranteed to run projects well. I think it is pretty clear that both Adobe and Apple haven’t embraced comprehensive reform of their entire development practices which is necessary to produce secure code (or really, quality code at all) with their given patch record and response. Vista would have been released much earlier if the project wasn’t horribly managed for the first three years, though I think it is a testament to Brian Valentine that he managed to undo all the damage Jim Allchin (whom, incidentally, I believe to be undead) did to the project and release a reasonably sound and secure solution. MS should lament loosing him to Amazon.
If you ask me what makes secure, robust software, it isn’t many eyes or many people working on a project. It isn’t a function of how many people are on the project at all. It comes from a commitment to those who manage a project or company. If executives of a company are absolutely committed to the security and quality of their company’s products, as is the case with the SDL at microsoft, then the company will produce secure, quality software. If the executives waver at all, it will cascade through the whole company. Likewise, on open source projects, the ultimate quality of the project comes down to who is running the project, how much power and say they have, and if they have an equal commitment. Open or closed source, if you are working on software the way you influence security and quality is to influence those in charge to be committed to such goals.