At Blackhat this year Mark Dowd of IBM and Alexander Sotirov or VMware presented a paper concerning how to bypass Vista’s Buffer Overflow Countermeasures. The paper is worth a read both because of the vulnerabilities, and because it does a great job detailing how much protection Microsoft has provided to people who can’t write good code. The analysis of the paper by various people on the Internet is mind numbing though (probably this post included).

Before continuing, I think some things need to be established up front. Vista provides protection that no other OS does against poorly written code. Even if the security features were completely worthless Vista would not be more dangerous than any other OS- just more dangerous than Vista from last month. This additional protection was meant to be a stop gap to secure bad code, not an excuse not to fix the bad code. The root cause is that some code sucks to begin with. It is like throwing a WAF on a wide open Internet app. That buffer overflows still exist to the extent they do is just disgraceful to us as an industry. We know exactly what causes buffer overflows and we know how to avoid them. Do not use unchecked buffer copies. Explicitly copy no more data than a buffer can hold (and please, please do not use inline calculations in strncpy, since that may as well be an unchecked buffer copy). Don’t dynamically create the first argument for *printf/*scanf without knowing exactly how many other parameters will need to be provided. And the list goes on. These are all things that static source code analysis tools are remarkably competent at finding. A clever source control system will flat out ban people from checking code in with bad APIs (strcpy and its unchecked brotheren). Finally, fuzz the crap out of all your input points (hey Jim, this is why input validation is important :-p). You will find and kill 99% of your buffer overflows if you do that, and the remaining 1% won’t be very discoverable (bad guys do fuzz testing too you know).

Anyway, there is a reason why MS has had a huge decline in buffer overflows relative to the industry, and it is because they do all of the above. They have too much legacy code and third party giblets to fully get rid of overflows, but all of the extra protection in Vista is primarily to provide some level of protection for the OS because a retarded third party (*cough*quicktime*cough*acrobat*cough*flash*cough) has yet one more buffer overflow that can be used to compromise the box. The protection was not a permanent fix, but simply something to raise the bar to perform exploitation, and I think Microsoft has been reasonably good about presenting it in that light while still trying to sell 3rd party developers on the protection. The biggest flaw in the protections that Vista presents is that applications opt into using them. Programs need to explicitly state they want the protection which is why the paper focuses on exploiting applications that do not have all of the protections (like IE 7 which doesn’t use data execution prevention because the poorly written piece of junk sun java plugin would blow chunks if they did; fortunately IE 8 does use it, as does firefox 3).

Now on to the commentary on the web. Peter Bright of Ars Technica tries to downplay it and many of his points are valid. Bruce Schneier thinks it is a big deal, which is true. A month ago people didn’t know how to bypass these protections but now they do. That is a big deal, but Adrian Kingsley-Hughes of ZDnet is still a moron for saying it renders Vista security next to useless.

So to respond to the commentary, the reason this is a big deal (moreso than Bright believes) but not the end of the world (as Kingsley-Hughes thinks) is that it makes exploits feasible but more costly. Any technological measure that increases the cost of exploiting something will decrease the number of instances that blackhats bother exploiting it. For example, to defeat ASLR the authors advocates hosting a .Net Web Component of sufficient size to dramatically impact how the address space can be randomized, and in order to facilitate downloads they also recommend the webserver hosting the component use gzip compression (which in turn means controlling web server configuration). That’s entirely possible, but if a malware author had the choice between an exploit targetting all of XP that didn’t require that extra work or an exploit for both XP and Vista, I suspect 90% will choose to just go after XP. (the exploit using Java applets is a bit more likely at the moment do to the GIFAR hack recently discussed, since you can very easily get hundreds of sites to host that for you).

On top of that, to bypass the multitude of protections requires some sophistication, which is just going to raise the bar on who crafts the exploits (well, until all of the bypasses are just built into metasploit). Raising the bar lowers the number of attackers capable of compromising the system.

Additionally, the easiest work around for DEP was to target applications that don’t use it, but this will become increasingly difficult, especially as MS enforces DEP to a greater degree. Finally, at least in terms of IE, one feature that was not addressed with the process isolation of the browser in Vista. It remains an open question on whether the IE sandbox in Vista is vulnerable, as it may still contain the attack.

So attacks are now possible, but the exploitability is still more difficult than other platforms. That’s big, but it by no means renders the protection next to useless. As the authors conclude

In this paper we demonstrated that the memory protection mechanisms available in the latest versions of Windows are not always effective when it comes to preventing the exploitation of
memory corruption vulnerabilities in browsers. They raise the bar, but the attacker still has a
good chance of being able to bypass them. Two factors contribute to this problem: the degree to
which the browser state is controlled by the attacker; and the extensible plugin architecture of
modern browsers.

At the end of the day, good OS design cannot completely mitigate bad code. Get rid of buffer overflows and this is all a moot point anyway. Its 2008 people, why aren’t buffer overflows extinct?

~ Joshbw

Ars has their panties in a twist because Kevin Turner dared to suggest that vista is more secure than OS X or Linux. The thing is, from one point of view, Kevin is absolutely right, but from other points of view he is wrong, which just points out the general failing that keeps popping up among security folks: How on earth do you measure security?

When talking about security there isn’t a clean metric.  You have

Vulnerability: Software (for this purpose) weakness that provides a door in
Threat: danger that someone will find the vulnerability
Risk: the likelihood and cost of the vulnerability being exploited
Exposure: time that a vulnerability remains

Companies/orgs love to cherry pick which one they go off of.  Firefox is big on touting their low exposure while being horribly quiet about the number of vulnerabilities.  Microsoft is big about vulnerability (and to some extent exposure, at least on the OS level) count, since they do have a much lower count then competitors that don’t have a decent secure development lifecycle (which would be Apple and most of the OSS projects), but they don’t like mentioning risk because they have more risk than freaking anyone.

The problem is, all of these approaches are at least a little right.  Sure the risk is huge on windows as a whole, but if you install patches quickly your risk is actually pretty low, and with a lower vulnerability count there is less likelihood that someone will release a zero day for a vulnerability that isn’t patched. Mozilla is also right about firefox being more secure, as the fast patch and deployment rate means that zero day isn’t going to remain a concern long (except that they have been a little too quick pushing patches and have regressed things on a number of occassions), but their higher vulnerability count does mean zero days are going to crop up more often.

The only stance that isn’t right is one that throws out vulnerability count as a metric because of “more eyes” (which I have complained about before). Being open source may mean that more good guys have access to code reviews, but it also means more bad guys can get a leg up looking for flaws. The number of known vulnerabilities is ALWAYS a problem regardless of why they are there (incidnetally, again, that more eyes arguement is crap. Code reviews are a horribly inefficient way of finding flaws, and you never know when some poorly trained reviewer will comment out the functionality that actually makes your random function random).

So if all of these are a little right, what is the problem? I think it is really twofold. First, it makes it very hard to make an informed comparison of two platforms in terms of security and best fit for your org. Second, it makes it hard to gauge your own security and communicate that to people in and out of your organization. Both are pretty important, and a common, informative, metric benefits everyone (except those that look bad because of it).

Thinking about it I like:

[Ʃ (CriticalityModifier * Exposure)] / (nTotal * 10)

or, in English, the sum of Criticality of each vulnerability in the past twelve months * the number of days that the vulnerability was unpatched (patches that get rolled back don’t count), then divided by the total number of vulnerabilities times ten, to scale down the number a bit.

The criticality modifier is 0.5 for low, 1 for moderate, 2 for high, 3 for critical. As an example, if a hypothetical product had the following vulnerabilities over the past twelve months:

would give you (3*45 + 3*35 +3*4 +2*20 + 1*45 + 1*50 + 0.5*145 + 0.5*75) / (8 * 10) = 6.15 Security Rating.

This seems like a fair metric, as it measures a level of impact (the criticality, though this isn’t horribly quantitative) of the flaw, the days of exposure to the flaw, and the number of total flaws. What it does not take into account is likelihood, as I can’t come up with a good quasi-quantitive means to measure said likelihood (open to suggestions here).

Thoughts?

~ Joshbw

Perhaps I was a bit misleading in the title as this is a flaw in an internet connected coffee machine (why, oh why, do we need this) that allows, among other things, access to a connected PC.

There are a couple lessons in this. First, no matter how trivial your product, you very likely still need to worry about security if it connects to something that does have value. Second, your system is only as secure as all of the systems that connect to it, unless the system is inherently paranoid about connections. In this web 2.0 world of mashups, in the world of connected enterprise applications, our applications are not living in a bubble. They are connected to multiple other systems, and in most cases there isn’t a whole lot of thought that went into security over these connections. A web application may be horribly paranoid about anything it receives from the user, but then blindly accept data from internal systems without a thought. If any of those systems is flawed and you don’t validate input from them, your application is just as susceptible to attack as it would be if it didn’t validate user input.

This just further cements in my head the need to push threat modeling out to more people. I think the real value of threat modeling isn’t the complex threat trees that get built for risk analysis. I actually find those pretty worthless. I think the data flow diagram is the big thing. It is a convenient visual diagram showing where a process or feature is pulling data from and where it is sending the data to. Any source of data is a source of attack, and the DFD is a useful tool for identifying where further scrutiny is necessary. Any output source is a particular asset that might be attacked through your application. I think every dev spec should require a data flow diagram simply because it gets people thinking.

Finally, the last immediate lesson from this is that it is a scary time to be a client. The OS may be pretty solid but there are a million ways to execute code that seem perfectly valid from the OS point of view. Every connected application, any application that pulls data or opens data originally from a remote source, is a potential open door. If a freaking coffee machine can be a source of attack client security is in a lot of trouble.

~ Joshbw

Seriously, the webappsec community is abuzz with the new PCI requirements that allow either code review or application firewalls, and there are people out there debating the merits of both, as if it is a competition.  Are WAFs a band-aid?  Of course they can be, to compensate for crappy input validation, but even if one’s input validation is perfect and protects against all known attacks today having an added layer of defense isn’t likely going to hurt a great deal (though specific situations can cause issues), and is quicker to update when new attacks are developed.  Also, your input validation isn’t perfect.  Even your whitelist input validation isn’t perfect.  It is probably “good enough”, but it isn’t perfect.

Are WAFs a replacement for code review- not ever.  A WAF sucks at finding back doors or easter eggs, or finding pages that *should* have an authorization check at the top but don’t, or a myriad of other issues that can still be used right through the firewall.  Code review will find those things, even if it is a very inefficient and ineffective way of tracking down vulnerabilities in general. 

In truth, none of this is sufficient, but PCI isn’t really about securing customer data, at least the way it is written, but rather a way to avoid liability.  It is a minimum number of steps to say “look, we tried to do something”.  The fact that “Private Networks” are considered secure and data can be sent in the clear over them (even though less than 30% of successful attacks actually come from external networks) shows that it isn’t really concerned with really protecting payment card information.

For companies that want to meet the minimum requirements for PCI, the only question is which solution is cheapest and easiest.  For companies that actually care about protecting their customer’s sensitive information, the question shouldn’t be which to do, but what additionally can be done.  Neither of these solutions are sufficient to ensure that data is protected by themselves.  The first requirement to protect customer data is to get upper management to care, then to put security standards in place, a secure development lifecylce in place, and educate developers on the threats they face and the protections they should employ.

An ideal solution shouldn’t just have a code review, but a feature review when the feature is proposed (with security requirements included), a risk analysis and mitigation strategy during the architecure planning, threat modeling and detail design security review, code review, and both security control testing and pentesting both with and without a WAF, so that if the WAF fails the application is still strong.

That is what PCI should be.  Yes, it is a huge burden on development teams and companies that don’t do those things already, but at the same time, ask yourself if you would trust a company that did the current bare minimum to meet PCI as it stands now with your card information.  I sure don’t, which is why I am glad that my bank now offers onetime card numbers and more and more sites accept paypal. 

~ Joshbw

Anne van Kesteren has a blog post deriding MS for deviating from the proposed standard with the inclusion of XDomainRequest. Your can read the horribly dry standard as well as the design decisions behind it here.  To be blunt, the standard (as well as Mr. van Kesteren) is wrong and MS is right in this case, though if you have read a previous post of mine it should be evident that I think standards bodies generally suck at including security.  I think the W3C has a habit of focusing on what web developers want to do, and not whether they *should* be able to do it (essentially the same problem MS had pre-SDL; give the users what they want, not what is in their best interest).

Key to the proposed standard, ”cookies and authentication information should be sent in the request”, which is a horrible, horrible idea.  Sure, it allows the inclusion of session specific information, information for a specific user, from an external domain in the web page the user is viewing which could have some interesting mash-up results, but it also allows the website making the request to access to that information.  In response headers, which necessarily need to be queriable from JavaScript to see if the request worked, that cookie information is going to be readable as will the potentially sensitive information returned as content.  There is no good reason why my website should have access to any other website’s cookie info.

One could argue that a website can always deny a request from an external source unless it happens to be from a specific domain (hah, as if that would be spoofed), and essentially grant content only to predetermined trusted callers, but there is already a mechanism for that.  If I have an agreement with Amazon to access their user specific data from my website, I can do so through a webservice call in my server side code.  It does require a bit more work, but that extra work will require both parties to think very carefully about what is exposed and what level of access to give. 

If this is implemented in a browser, it will be exploited.  Large websites with a strong security record will likely implement this functionality securely, but there are a whole lot of companies out there without strong security records, with marketing people calling the shots, who will allow unrestricted third party websites to make calls to their protected content.  The security community will figure out how to steal cookie information using it (with the current proposal, it would be sort of trivial).  The idea that “Cookies and authentication information is already sent cross-site for various HTML elements, such as img, script, and form. This means that such a request does not introduce a new attack vector.” as the design decisions for the W3C proposal states, is hopelessly naive. Citing requests made outside of JavaScript, where the browser returns no significant information to potentially malicious scripts, is very different from making said requests and granting access to the response and content inside of script.  That the standards body can’t grok the difference is quite worrisome.

Microsoft is absolutely correct in their implementation of XDomainRequest.  It should only have access to information that the whole world has access to, and it should not pass any sensitive information (like sessionIDs stored in cookies) that the original domain does not already have access to.  Is it restrictive, absolutely, but for the blackhats as well as the web developers.  That is sort of the point.  There are secure ways to retrieve information from another domain.  It does require more work.  It also requires more work for me to have to verify my identity when using my credit card than it would if everyone would just trust me, but I think we can all agree that is for my own good.

For all of you complaining about MS not supporting the standard with this, please stop considering for a moment what you could do with the W3C proposal, and consider what a malicious person could do.

~ Joshbw

Many, many companies rely on vendors to do some level of development for them.  Even Microsoft, with its 50,000 someodd full time development staff (probably larger now) and equally large number of contingent staff, ends up outsourcing certain development projects to third party companies (in the case of MS, the reason for this is myriad; for one shot projects it makes no sense to form a team that will be disbanded in a couple months.  For established teams, due to the corporate structure, it is actually easier to get millions of dollars more in a budget than a couple more full time employees, so vendor development ends up being a practical way of filling a manpower gap).  For non-software companies vendors play an even larger role.

In my experience with vendor developed code, from a security standpoint it has largely been absolute crap, with few exceptions (just ask the TSA with their recent disclosure fiasco because of a vendor developed website).  The implementation is a mess, and during pentesting the hardest part is documenting all of the myriad of flaws found.  The cause of this seems to be two fold.  First, most vendor companies have no security mindset to speak of and second, the hiring company didn’t make security a requirement during negotiations (though it is sad that security needs to be an explicit requirement but such is the sad state of software development).

To increase the quality of vendor developed code I have a couple practices.  To begin with, anyone interfacing with a vendor should have a degree of expertise in doing so or be accompanied by someone who does.  Having a marketing guy with no product development knowledge to speak of being the sole liaison with a particular vendor is a really bad idea.  You don’t send an accountant to interview potential pilots, so why send a marketing person to interview potential developers?  They are not going to know the correct questions to ask when determining which vendor company to employ.

Speaking of which, one of the correct questions to ask is “Describe your secure engineering practices and policies”.  A vendor that knows anything about security will immediately start citing policy. “At design time we do this, at implementation time this, at test time this” and so forth.  If they hesitate, have very vague or uncertain answers, or just start talking BS, your conversation should be done.  What they mean to say is that they have no secure engineering plan.  As a followup question, assuming they answered positively to the first question, ask them about the security training their developers and testers have.  What percentage have had training, who gave the training, what sort of training was it, how often do they receive followup training?

As fair warning, Vendor companies that can adequately answer such questions are few and far between.  Companies have to search these vendors out, and those vendors rightly charge more than their competitors.  This premium is money saved in the long run.  Insecure implementations by vendors typically need to be heavily modified if not out right discarded to fix the vulnerabilities, and if exploited in the interim often times the cost of the exploit  is many many times the cost of competent engineers.  Choosing a security ignorant vendor will simply lead to needing to hire the competent vendor three months after the project is done (or alternatively hire another security incompetent vendor and repeat ad nauseam until you learn your lesson). 

Security standards need to be part of vendor requirements as much as any other development standards should be (you do have dev standards you give vendors, right?).  The level of security you expect in the finished product is a feature as much as any other functionality that you want the vendor to implement.  The standards should be thorough and often time explicit: stored passwords should be hashed, sensitive data protected, input verified, output sanitized, authorization and authentication requirements should be specific, windows client side apps should enable ASLR and NX (if getting binaries instead of source) for C++ as well as pass prefast, .Net client side apps should pass fxcop, etc.  The vendor’s paycheck should be as contigent on these things as the implementation of any other functionality.

Ideally, you would get the vendor to agree to some sort of maintenance contract, and if so, security issues need to be called out along with every other bug.  In fact, ideally you would make the contract contingent on the vendor being responsible for fixing vulnerabilities regardless of whether there is a maintenance contract in place or not.  Vendor companies pick up secure practices real fast after the execution of such a support clause a couple hundred times, destroying any profit they might make many times over.

If you do these things you will go a long way to improving the security of code you receive from a vendor.  In the end they should be held to standards at least as high as any internal developer.  You are paying through the teeth (or what ever the saying is; I am unclear wear exactly you should pay through) for their time.  It might as well be worth it.

~ Joshbw 

One of the phrases I hear fairly often from the open source community is that “Many eyes make better code” and it confounds me that people actually believe that the quality of code, and thus the quality of the application, is simply a function of how many people view the source code.  Before I give the impression that this is a screed against the concept of open source software, allow me to assert that I don’t have an ideological preference one way or the other towards open source, though I find zealotry of any form to be tiring and frustrating.  For finished apps its great that the application is free, I like free, but it really delivers no value to me if I can change the source code because frankly I am not going to want to modify the source for my word processor (is that term still in use today, or is it an anachronism from the 80s?).  For code that I consume in conjunction with my own code, say a .Net library or a PHP application that I want to use on my website, there is value to me in being able to freely alter the code as well, but only if I don’t have to worry about any restrictive licensing (BSD style license = good, GPL = pain in the ass if used for anything related to work).

That said, I find some of the common open source sentiments silly, the “Many eyes” argument one of the silliest.  Essentially what the “Many eyes” argument suggests is that quite a few more people are doing code reviews than you would have in a closed source application, which for the most part is probably true.  There are two counterarguments that I haven’t seen addressed.  The first is that having many eyes look at source code doesn’t mean a thing by itself, if you don’t qualify the experience and expertise of the many eyes, nor does it mean anything if the analysis isn’t systematic.  Five really sharp closed source developers will be more effective reviewing code than one hundred amateur open source developers.  They are more likely to be scrutinizing the trouble areas, and they are more likely to spot issues in those areas.

Code reviewing is an art.  A reviewer is looking at things line by line, but has to retain knowledge of all the code that impacts a particular line of code.  It’s easy to spot use of a banned API, potential overflow conditions, bad arithmetic or logic formation, etc, but in most cases static code analysis tools are faster and better at spotting those sorts of things.  Where a human has an edge over the tools is seeing larger code pattern flaws, questionable but legal use of certain API, missing validation, and code traversal issues, but as a qualifier, the humans that are good at spotting those problems are a rare breed.  They can focus on many things at once, recall pertinent details from code many pages before, and instinctively spot worrisome code.  It isn’t clear to me that an open source project attracts any more of these experts than the equivelent closed source project, at least on average.  So I believe that open source can summon more eyes for code review, but I haven’t seen evidence that those many eyes actually were thoroughly scrutinizing the risky areas of most OSS applications, nor have I been convinced that most OSS applications have attracted more skilled eyes.

That point is pretty irrelevant with my second counter argument: code review is not the key to robust applications but rather just one component.  They key to robust applications is project organization.  Regardless of whether a project is closed or open source, to be secure, to be stable and robust, the entire development process needs to be structured and thorough.  During design time features need to be decided upon, experts in given areas (UI, database design, network design) need to spec out their given areas, developers need to create development specs to reflect the design specs, and then security reviews need to be done on the design.  Also during design time coding guidelines need to be established and key project, development, and test managers need to be established (even if it is open source, you need specific goto people to oversee things).  During development code needs to be reviewed to ensure the coding guidelines are being followed, that the implementation reflects the design and if not (for a good reason) the design documents need to be updated.  Testing needs to start as features are initially implemented, not once code complete is called.  Testers need to be systematic and need to act as a further sanity check that the implementation matches the intended design.  They also need clear areas of test assigned to them to ensure even coverage and no unnecessarily redundant effort at the expense of untested code.  Through each phase, design, development, and test, there needs to be security checks.  If the first security work is done when the first beta is produced the application is going to be full of vulnerabilities.

A thorough, organized development effort is what makes good code.  Some open source projects have this, for example the linux kernel, the apache web server, and arguably the mozilla/firefox teams now, though in this latter case it is very clear this wasn’t always so.  Most people forget that when Mozilla 1.0 finally released, despite being several years behind schedule, it was an utter failure, so much so that the Pheonix (then firebird, then firefox) team forked the project just to try and makes something that didn’t suck.

A great deal of open source projects do not get this, they don’t get that they need to be as organized and managed as any closed source project.  Its obvious that coding standards either didn’t exist, were incredibly loose, or just weren’t enforced on many open source projects. For example just download any PHP application; phpBB, Wordpress, phpNuke, etc all have very unprofessional code.  There are very few comments, if any, in the code, extensive use of nested arithmetic in function calls, little data validation and error checking, use of tertiary operators for complex assignments (they are meant for simple assignments), etc.  I can only imagine that if coding standards weren’t followed the rest of the development process similarly lacked hallmarks of a well run project.

There are many closed source applications that similarly lack this organization.  Many small companies aren’t organized a whole lot differently that small open source projects.  Developers hate the bureaucratic pain in the ass steps that a rigidly run project makes them perform. At small companies it is common for the devs to just not follow those steps, because no one is there to force them to.  I commonly hear “Why do I have to write a document describing what my code is going to do when I could just write the code and show you” concerning developer specs.  They don’t realize or care that it is a lot easier to read an english description of something and figure out how it works than it is to look at the code, even for the most competent coders.  Its also a lot quicker to write and correct an english (or chosen spoken language) outline than to implement everything and make changes after the fact.

Even very large companies aren’t guaranteed to run projects well.  I think it is pretty clear that both Adobe and Apple haven’t embraced comprehensive reform of their entire development practices which is necessary to produce secure code (or really, quality code at all) with their given patch record and response.  Vista would have been released much earlier if the project wasn’t horribly managed for the first three years, though I think it is a testament to Brian Valentine that he managed to undo all the damage Jim Allchin (whom, incidentally, I believe to be undead) did to the project and release a reasonably sound and secure solution.  MS should lament loosing him to Amazon.

If you ask me what makes secure, robust software, it isn’t many eyes or many people working on a project.  It isn’t a function of how many people are on the project at all.  It comes from a commitment to those who manage a project or company.  If executives of a company are absolutely committed to the security and quality of their company’s products, as is the case with the SDL at microsoft, then the company will produce secure, quality software.  If the executives waver at all, it will cascade through the whole company.  Likewise, on open source projects, the ultimate quality of the project comes down to who is running the project, how much power and say they have, and if they have an equal commitment.  Open or closed source, if you are working on software the way you influence security and quality is to influence those in charge to be committed to such goals.