Seriously, the webappsec community is abuzz with the new PCI requirements that allow either code review or application firewalls, and there are people out there debating the merits of both, as if it is a competition.  Are WAFs a band-aid?  Of course they can be, to compensate for crappy input validation, but even if one’s input validation is perfect and protects against all known attacks today having an added layer of defense isn’t likely going to hurt a great deal (though specific situations can cause issues), and is quicker to update when new attacks are developed.  Also, your input validation isn’t perfect.  Even your whitelist input validation isn’t perfect.  It is probably “good enough”, but it isn’t perfect.

Are WAFs a replacement for code review- not ever.  A WAF sucks at finding back doors or easter eggs, or finding pages that *should* have an authorization check at the top but don’t, or a myriad of other issues that can still be used right through the firewall.  Code review will find those things, even if it is a very inefficient and ineffective way of tracking down vulnerabilities in general. 

In truth, none of this is sufficient, but PCI isn’t really about securing customer data, at least the way it is written, but rather a way to avoid liability.  It is a minimum number of steps to say “look, we tried to do something”.  The fact that “Private Networks” are considered secure and data can be sent in the clear over them (even though less than 30% of successful attacks actually come from external networks) shows that it isn’t really concerned with really protecting payment card information.

For companies that want to meet the minimum requirements for PCI, the only question is which solution is cheapest and easiest.  For companies that actually care about protecting their customer’s sensitive information, the question shouldn’t be which to do, but what additionally can be done.  Neither of these solutions are sufficient to ensure that data is protected by themselves.  The first requirement to protect customer data is to get upper management to care, then to put security standards in place, a secure development lifecylce in place, and educate developers on the threats they face and the protections they should employ.

An ideal solution shouldn’t just have a code review, but a feature review when the feature is proposed (with security requirements included), a risk analysis and mitigation strategy during the architecure planning, threat modeling and detail design security review, code review, and both security control testing and pentesting both with and without a WAF, so that if the WAF fails the application is still strong.

That is what PCI should be.  Yes, it is a huge burden on development teams and companies that don’t do those things already, but at the same time, ask yourself if you would trust a company that did the current bare minimum to meet PCI as it stands now with your card information.  I sure don’t, which is why I am glad that my bank now offers onetime card numbers and more and more sites accept paypal. 

~ Joshbw

Anne van Kesteren has a blog post deriding MS for deviating from the proposed standard with the inclusion of XDomainRequest. Your can read the horribly dry standard as well as the design decisions behind it here.  To be blunt, the standard (as well as Mr. van Kesteren) is wrong and MS is right in this case, though if you have read a previous post of mine it should be evident that I think standards bodies generally suck at including security.  I think the W3C has a habit of focusing on what web developers want to do, and not whether they *should* be able to do it (essentially the same problem MS had pre-SDL; give the users what they want, not what is in their best interest).

Key to the proposed standard, ”cookies and authentication information should be sent in the request”, which is a horrible, horrible idea.  Sure, it allows the inclusion of session specific information, information for a specific user, from an external domain in the web page the user is viewing which could have some interesting mash-up results, but it also allows the website making the request to access to that information.  In response headers, which necessarily need to be queriable from JavaScript to see if the request worked, that cookie information is going to be readable as will the potentially sensitive information returned as content.  There is no good reason why my website should have access to any other website’s cookie info.

One could argue that a website can always deny a request from an external source unless it happens to be from a specific domain (hah, as if that would be spoofed), and essentially grant content only to predetermined trusted callers, but there is already a mechanism for that.  If I have an agreement with Amazon to access their user specific data from my website, I can do so through a webservice call in my server side code.  It does require a bit more work, but that extra work will require both parties to think very carefully about what is exposed and what level of access to give. 

If this is implemented in a browser, it will be exploited.  Large websites with a strong security record will likely implement this functionality securely, but there are a whole lot of companies out there without strong security records, with marketing people calling the shots, who will allow unrestricted third party websites to make calls to their protected content.  The security community will figure out how to steal cookie information using it (with the current proposal, it would be sort of trivial).  The idea that “Cookies and authentication information is already sent cross-site for various HTML elements, such as img, script, and form. This means that such a request does not introduce a new attack vector.” as the design decisions for the W3C proposal states, is hopelessly naive. Citing requests made outside of JavaScript, where the browser returns no significant information to potentially malicious scripts, is very different from making said requests and granting access to the response and content inside of script.  That the standards body can’t grok the difference is quite worrisome.

Microsoft is absolutely correct in their implementation of XDomainRequest.  It should only have access to information that the whole world has access to, and it should not pass any sensitive information (like sessionIDs stored in cookies) that the original domain does not already have access to.  Is it restrictive, absolutely, but for the blackhats as well as the web developers.  That is sort of the point.  There are secure ways to retrieve information from another domain.  It does require more work.  It also requires more work for me to have to verify my identity when using my credit card than it would if everyone would just trust me, but I think we can all agree that is for my own good.

For all of you complaining about MS not supporting the standard with this, please stop considering for a moment what you could do with the W3C proposal, and consider what a malicious person could do.

~ Joshbw

Many, many companies rely on vendors to do some level of development for them.  Even Microsoft, with its 50,000 someodd full time development staff (probably larger now) and equally large number of contingent staff, ends up outsourcing certain development projects to third party companies (in the case of MS, the reason for this is myriad; for one shot projects it makes no sense to form a team that will be disbanded in a couple months.  For established teams, due to the corporate structure, it is actually easier to get millions of dollars more in a budget than a couple more full time employees, so vendor development ends up being a practical way of filling a manpower gap).  For non-software companies vendors play an even larger role.

In my experience with vendor developed code, from a security standpoint it has largely been absolute crap, with few exceptions (just ask the TSA with their recent disclosure fiasco because of a vendor developed website).  The implementation is a mess, and during pentesting the hardest part is documenting all of the myriad of flaws found.  The cause of this seems to be two fold.  First, most vendor companies have no security mindset to speak of and second, the hiring company didn’t make security a requirement during negotiations (though it is sad that security needs to be an explicit requirement but such is the sad state of software development).

To increase the quality of vendor developed code I have a couple practices.  To begin with, anyone interfacing with a vendor should have a degree of expertise in doing so or be accompanied by someone who does.  Having a marketing guy with no product development knowledge to speak of being the sole liaison with a particular vendor is a really bad idea.  You don’t send an accountant to interview potential pilots, so why send a marketing person to interview potential developers?  They are not going to know the correct questions to ask when determining which vendor company to employ.

Speaking of which, one of the correct questions to ask is “Describe your secure engineering practices and policies”.  A vendor that knows anything about security will immediately start citing policy. “At design time we do this, at implementation time this, at test time this” and so forth.  If they hesitate, have very vague or uncertain answers, or just start talking BS, your conversation should be done.  What they mean to say is that they have no secure engineering plan.  As a followup question, assuming they answered positively to the first question, ask them about the security training their developers and testers have.  What percentage have had training, who gave the training, what sort of training was it, how often do they receive followup training?

As fair warning, Vendor companies that can adequately answer such questions are few and far between.  Companies have to search these vendors out, and those vendors rightly charge more than their competitors.  This premium is money saved in the long run.  Insecure implementations by vendors typically need to be heavily modified if not out right discarded to fix the vulnerabilities, and if exploited in the interim often times the cost of the exploit  is many many times the cost of competent engineers.  Choosing a security ignorant vendor will simply lead to needing to hire the competent vendor three months after the project is done (or alternatively hire another security incompetent vendor and repeat ad nauseam until you learn your lesson). 

Security standards need to be part of vendor requirements as much as any other development standards should be (you do have dev standards you give vendors, right?).  The level of security you expect in the finished product is a feature as much as any other functionality that you want the vendor to implement.  The standards should be thorough and often time explicit: stored passwords should be hashed, sensitive data protected, input verified, output sanitized, authorization and authentication requirements should be specific, windows client side apps should enable ASLR and NX (if getting binaries instead of source) for C++ as well as pass prefast, .Net client side apps should pass fxcop, etc.  The vendor’s paycheck should be as contigent on these things as the implementation of any other functionality.

Ideally, you would get the vendor to agree to some sort of maintenance contract, and if so, security issues need to be called out along with every other bug.  In fact, ideally you would make the contract contingent on the vendor being responsible for fixing vulnerabilities regardless of whether there is a maintenance contract in place or not.  Vendor companies pick up secure practices real fast after the execution of such a support clause a couple hundred times, destroying any profit they might make many times over.

If you do these things you will go a long way to improving the security of code you receive from a vendor.  In the end they should be held to standards at least as high as any internal developer.  You are paying through the teeth (or what ever the saying is; I am unclear wear exactly you should pay through) for their time.  It might as well be worth it.

~ Joshbw 

One of the phrases I hear fairly often from the open source community is that “Many eyes make better code” and it confounds me that people actually believe that the quality of code, and thus the quality of the application, is simply a function of how many people view the source code.  Before I give the impression that this is a screed against the concept of open source software, allow me to assert that I don’t have an ideological preference one way or the other towards open source, though I find zealotry of any form to be tiring and frustrating.  For finished apps its great that the application is free, I like free, but it really delivers no value to me if I can change the source code because frankly I am not going to want to modify the source for my word processor (is that term still in use today, or is it an anachronism from the 80s?).  For code that I consume in conjunction with my own code, say a .Net library or a PHP application that I want to use on my website, there is value to me in being able to freely alter the code as well, but only if I don’t have to worry about any restrictive licensing (BSD style license = good, GPL = pain in the ass if used for anything related to work).

That said, I find some of the common open source sentiments silly, the “Many eyes” argument one of the silliest.  Essentially what the “Many eyes” argument suggests is that quite a few more people are doing code reviews than you would have in a closed source application, which for the most part is probably true.  There are two counterarguments that I haven’t seen addressed.  The first is that having many eyes look at source code doesn’t mean a thing by itself, if you don’t qualify the experience and expertise of the many eyes, nor does it mean anything if the analysis isn’t systematic.  Five really sharp closed source developers will be more effective reviewing code than one hundred amateur open source developers.  They are more likely to be scrutinizing the trouble areas, and they are more likely to spot issues in those areas.

Code reviewing is an art.  A reviewer is looking at things line by line, but has to retain knowledge of all the code that impacts a particular line of code.  It’s easy to spot use of a banned API, potential overflow conditions, bad arithmetic or logic formation, etc, but in most cases static code analysis tools are faster and better at spotting those sorts of things.  Where a human has an edge over the tools is seeing larger code pattern flaws, questionable but legal use of certain API, missing validation, and code traversal issues, but as a qualifier, the humans that are good at spotting those problems are a rare breed.  They can focus on many things at once, recall pertinent details from code many pages before, and instinctively spot worrisome code.  It isn’t clear to me that an open source project attracts any more of these experts than the equivelent closed source project, at least on average.  So I believe that open source can summon more eyes for code review, but I haven’t seen evidence that those many eyes actually were thoroughly scrutinizing the risky areas of most OSS applications, nor have I been convinced that most OSS applications have attracted more skilled eyes.

That point is pretty irrelevant with my second counter argument: code review is not the key to robust applications but rather just one component.  They key to robust applications is project organization.  Regardless of whether a project is closed or open source, to be secure, to be stable and robust, the entire development process needs to be structured and thorough.  During design time features need to be decided upon, experts in given areas (UI, database design, network design) need to spec out their given areas, developers need to create development specs to reflect the design specs, and then security reviews need to be done on the design.  Also during design time coding guidelines need to be established and key project, development, and test managers need to be established (even if it is open source, you need specific goto people to oversee things).  During development code needs to be reviewed to ensure the coding guidelines are being followed, that the implementation reflects the design and if not (for a good reason) the design documents need to be updated.  Testing needs to start as features are initially implemented, not once code complete is called.  Testers need to be systematic and need to act as a further sanity check that the implementation matches the intended design.  They also need clear areas of test assigned to them to ensure even coverage and no unnecessarily redundant effort at the expense of untested code.  Through each phase, design, development, and test, there needs to be security checks.  If the first security work is done when the first beta is produced the application is going to be full of vulnerabilities.

A thorough, organized development effort is what makes good code.  Some open source projects have this, for example the linux kernel, the apache web server, and arguably the mozilla/firefox teams now, though in this latter case it is very clear this wasn’t always so.  Most people forget that when Mozilla 1.0 finally released, despite being several years behind schedule, it was an utter failure, so much so that the Pheonix (then firebird, then firefox) team forked the project just to try and makes something that didn’t suck.

A great deal of open source projects do not get this, they don’t get that they need to be as organized and managed as any closed source project.  Its obvious that coding standards either didn’t exist, were incredibly loose, or just weren’t enforced on many open source projects. For example just download any PHP application; phpBB, WordPress, phpNuke, etc all have very unprofessional code.  There are very few comments, if any, in the code, extensive use of nested arithmetic in function calls, little data validation and error checking, use of tertiary operators for complex assignments (they are meant for simple assignments), etc.  I can only imagine that if coding standards weren’t followed the rest of the development process similarly lacked hallmarks of a well run project.

There are many closed source applications that similarly lack this organization.  Many small companies aren’t organized a whole lot differently that small open source projects.  Developers hate the bureaucratic pain in the ass steps that a rigidly run project makes them perform. At small companies it is common for the devs to just not follow those steps, because no one is there to force them to.  I commonly hear “Why do I have to write a document describing what my code is going to do when I could just write the code and show you” concerning developer specs.  They don’t realize or care that it is a lot easier to read an english description of something and figure out how it works than it is to look at the code, even for the most competent coders.  Its also a lot quicker to write and correct an english (or chosen spoken language) outline than to implement everything and make changes after the fact.

Even very large companies aren’t guaranteed to run projects well.  I think it is pretty clear that both Adobe and Apple haven’t embraced comprehensive reform of their entire development practices which is necessary to produce secure code (or really, quality code at all) with their given patch record and response.  Vista would have been released much earlier if the project wasn’t horribly managed for the first three years, though I think it is a testament to Brian Valentine that he managed to undo all the damage Jim Allchin (whom, incidentally, I believe to be undead) did to the project and release a reasonably sound and secure solution.  MS should lament loosing him to Amazon.

If you ask me what makes secure, robust software, it isn’t many eyes or many people working on a project.  It isn’t a function of how many people are on the project at all.  It comes from a commitment to those who manage a project or company.  If executives of a company are absolutely committed to the security and quality of their company’s products, as is the case with the SDL at microsoft, then the company will produce secure, quality software.  If the executives waver at all, it will cascade through the whole company.  Likewise, on open source projects, the ultimate quality of the project comes down to who is running the project, how much power and say they have, and if they have an equal commitment.  Open or closed source, if you are working on software the way you influence security and quality is to influence those in charge to be committed to such goals.