2008 Pwnie Awards: Last Chance For Nominations

Dave G. | July 14th, 2008 | Filed Under: Industry Punditry

Just as a reminder, today is the last day that you can nominate yourself your peers for a pwnie award. Categories include:

1. Best Server-Side Bug
2. Best Client-Side Bug
3. Mass 0wnage
4. Most Innovative Research
5. Lamest Vendor Response
6. Most Overhyped Bug
7. Best Song
8. Most Epic FAIL
9. Lifetime Achievement Award

   As things stand today, the community clearly thinks that a hacker with the simple moniker of ‘ deserves to win every category. I think we can do better.

   Go here now to give your peers the recognition they deserve… a toy for children ages 5 and up.

   1 Comment

Internationalization of Malware

Wes Brown | July 1st, 2008 | Filed Under: Industry Punditry, Malware

Why am I qualified to write about this?

In a former life, I was a malware analyst.  It was my job to analyze incoming samples and determine if they were false positives, or false negatives.  I also worked on automating the process, and it was a very neat job.  Unfortunately for me, economic realities and the precarious nature of startup companies dictated other career paths.  I had analyzed literally thousands of samples, and took notes on characteristics to help improve the anti-malware product.

Eventually after a while of doing this, I have some observations on where malware is going, and I’m going to share some of them in this post.

Growing Internationalization

In the past, an anti-malware company could focus on English-targeted samples.  But an increasing percentage of malware samples are international in origin and targeting international machines.  I saw numerous cases of Chinese malware targeting Chinese software or hosts.  This was quite a challenge to determine if it was malware or not for several reasons.

Cultural Impact

One of the most fascinating facets of the increasing internationalization of malware is the cultural assumptions around such software.  What is considered malware in the US may be commonly accepted in China or Japan, and this is largely due to the society that it exists in.

Anti-cheating rootkits are very common in games released in these countries.  What is considered to be invasive in the North American or European world is acceptable there.  These anti-cheating rootkits would hook into the kernel space in a very invasive way, and have the behavioral characteristics of malware such as hooking into the keyboard driver.  This made it very difficult from a purely technical standpoint to distinguish them.  These kits were attempting to protect the application from being tampered with while running, i.e. to reduce the incidence of bots, or modifications to the presentation layer to allow people to see through walls.  They would watch for kernel debuggers, or running processes that did specific characteristic behavior.  These very techniques would flag them as malware as many such samples would behave similarly to avoid antivirus or to prevent someone from easily reverse engineering them.

If I applied US standards to these particular samples and declared them a true positive, then we would have many angry international customers when their games no longer worked.  This also applied to extremely intrusive adware.  But these pieces of software could run on US machines as well, so it was a very tricky balance.

Linguistic Barriers

In the past, if I ran into a piece of malware that had foreign language strings in them, I could muddle through them if they were a Latin-derived language.  Spanish or French, I did not have any issues with.  But when it comes to languages that come from an entirely different root such as Chinese or Japaense written in hanzi or kanji, I was losing vital clues.

By looking at the behavior of the sample alone, I would declare it malware.  But what if it was one of the aforementioned game rootkits?  How do I know that the game actually includes it, or if it was indeed a trojan’ed game?  With English language samples, I would simply look at the strings, or use Google.  But I had to muddle through pages in a writing system that I could not easily begin to comprehend.

So, if you want to be a malware analyst, it would be in your best interest to become conversant or fluent in one or more of the non-Roman languages.

Internationalization of Antimalware Tools

As we are dealing more and more with malware samples that are international in scope, it becomes important that the tools themselves are internationalized. With the growth of samples targeted at other languages, the automatic tools that I wrote primarily dealt with ASCII and were becoming inadequate.  String and keyword analysis did not work well.  Tools need to be Unicode and multi-lingual.

Hints for International Malware Analysis

• Pay close attention to the signers of samples, whether they are signed or not.  Once you have verified a signed application, consider it the baseline.
• Once you have multiple samples of what appears to be the same application but has different checksums, pay close attention to file size, and the version, vendor strings.  Interestingly, many trojaned applications do not have the correct version and vendor strings.
• Use entropy to your advantage.  Measure the entropy of the binary segments that you have.  If they have very similar entropy values, and have a minor increment in version, the probability of it being a trojan is much lower.
• Pay close attention to the vendor and version strings of samples.  See if you can get an authoritative version of the application from the vendor’s site and compare it.  Once you have a sample that you can declare as as false positive, all other similar samples are much easier to analyze.
• Take note of what binary packers they are using.  Certain packers have a higher probability of being used by malware.  But there are legitimate use of packers, and some antivirus products will trigger a false positive on a packed application, no matter what.
• Build a library of samples, and understand the cultural context of the country of origin and destination.  Categorizing the sample characteristics by these criteria will help you determine if it is a true or a false positive for that particular market. 

Conclusion

It is becoming more and more important that entire infrastructure of malware analysis, from anti-malware client to backend infrastructure to the analyst herself become multilingual and multicultural.   It is a difficult challenge that is going to crop up more and more in the future.

11 Comments

Web 2.0 Vulnerability Reporting Continued

Dave G. | June 25th, 2008 | Filed Under: Industry Punditry

37signals had an interesting blog post entitled “Ask 37signals: How do you say no?”. The post advises readers on how to respond to feature requests that you are not interested in pursuing (at least in the short term). The short synopsis is to answer in one of two ways:

1. The Hard No. If the feature is not aligned with the direction of the product, just be direct and say so.
2. The Soft No. If it is something you might pursue in the future, but you don’t want to commit, say: thank you for the idea, we will consider it for a future version.

For the Soft No, they gave a quick example of language to use:

“Thanks for sending the suggestion over. While we can’t guarantee we’ll be adding this feature, we can promise you we’ll review it and possibly consider it for a future version.”

About a year ago, I posted a tidbit about reporting a security issue to a vendor who didn’t seem to take the issue seriously. Here is how they responded:

“Hi David, We’re always looking for new ideas and fixes to roll out in future updates but as as rule we don’t comment on possibilities or timeframes.”

As you can probably guess, 37signals was the previously anonymized vendor. After getting the Soft No, I had two of my emails ignored, one on May 21, 2007 and one on June 6, 2008.

Some memorable comments from our readers include:

“What we need is disclosure 2.0 guidelines for web 2.0 software. Dave did the right thing by informing the software company. The software company did the wrong thing by treating this as a bug. Perhaps there should be a “hall of web 2.0 shame” for companies that don’t fix vulnerabilities reported to them in a timely way.” — chrisw

“This company didn’t treat the vuln report as a bug. They treated it as a FEATURE REQUEST.” —tqbf

“Those of us who tend a little more towards the punitive end of the spectrum would tend to say that at this point, you name the company (if not the vuln.)” — Ryan Russell

“So maybe you should “vote with your wallet” and look for an alternative application. If you don’t consider the vulnerability serious enough to not use the application, why should they think it is serious enough to expend resources to fix?” — stacy

“Try contacting their funding source or, better yet, their lawyers. These days, those folks understand the words “risk to brand”, etc.” — Money and Lawyers

Vulnerability reporting should not be handled in the same way that you manage feature requests.

Treat them as defects that get in the way of your users utilizing your software. For example, this flaw isn’t so serious that we wouldn’t use the software at all, but it is serious enough that we had to limit how we used the software. From a technical perspective, it is downright boring and unimpressive. I am not interested in revealing the details of the flaw, I am simply interested in getting it fixed.

I have the utmost respect for 37signals both as a business and as professionals that engage, influence and contribute to our community. The goal here isn’t to put users at risk or to make 37signals look bad, but rather to get a better understanding about how our vendors manage security issues (e.g. Are there other security issues that aren’t getting addressed?), get a timeline of if/when it will be fixed, and hopefully to raise awareness about how organizations can better manage security vulnerabilities.

13 Comments

Rootkits Are Top Of Mind, Bottom Of Pile, Only They Really Aren’t

Dave G. | June 3rd, 2008 | Filed Under: Industry Punditry

Reading various media outlets and blogs recently, it seems that we are losing the war on rootkits. Why do I say that? First of all, for those of you who utilize or manage x86 based systems, security researchers are going to be presenting on rootkits that take advantage of System Management Mode. Cisco guy, don’t feel ignored, cause rootkits are a-comin’ your way too. Core is going to presenting on Cisco rootkits at Blackhat. Of course, attackers breaking into your network and installing rootkits on your Cisco’s would be terrible, but why would they bother when they can just sell you some heavily discounted pre-pwned Cisco gear.

Good news though, AV-Tests.org has just concluded that a small percentage of the tools out there catch most of the existing rootkits and were even able to remove them! From the conclusions section:

Tests of the active rootkit detection and cleaning features of anti-malware products are rather time consuming and require a lot of resources to perform. However, programmers and testers should dedicate more attention to these features, as most AV tools still perform poorly in this area. Without proper anti-rootkit features a protection program may give the user the wrong impression about the status of his PC.

To try and shed some light on this, Matasano convened a Blanel Of Experts, who for no reason what-so-ever I will call:

	Misanthropic Researcher A well-known security researcher who knows a thing or two about rootkits. Why You Care: He has written rootkits. Nuff Said. Favorite Email Client: DEBUG.EXE
	The Pundit Ex Analyst Gone Indie with a technical bend. Why You Care: Has heard it all and seen it all. Possibly has said it all. Lethally Trained In: Sniffing out BS.
	Enterprise Security Drill Sergeant In the trenches at an F1000. Why You Care: Unlike our first two participants, he actually has to deal with this crap. Most Comforting Sound: Machine gun fire.

Should we invest time into rootkit detection?

	As a pundit who’s also on the research side, I always think investing time into advancing a technology is good. Just because we’re facing a losing battle against a sophisticated attacker is no reason to give up, since most attackers are far from sophisticated. For every genius criminal mastermind, there are hundreds of bumbling idiots leaving their truck bumpers (with license plate) chained to the ATM machine they tried to haul off. Now that’s the research answer, and I think we clearly need to continue to advance rootkit detection. The truth is this should be included in our AV suites, and customers shouldn’t be paying more for every new category of malware. On the user side, unless you are one of the lucky few to have rootkit detection included in your existing endpoint security suite for free (with good performance), it’s something that should only be used tactically. When you have reasonable suspicion something is going on, through network security monitoring or some other anomalous behavior, the cash and performance cost of rootkit detection probably isn’t worth it. You might consider it more for exposed or high value systems, but even there you want to do a bunch of testing first. In other words, it’s more an incident response, investigative, and cleanup tool (when you can’t just nuke the system from orbit, which is the safest option). Eventually this functionality will be a standard part of our anti-malware (AV) tools, and probably perform about the same. Which means it will miss the hard stuff, catch the easy stuff, and probably cost too much for the value provided. But anti-malware still provides enough marginal value that we can’t just dump it.
	Yes. If you can’t detect them, how do you know they are there?
	Of course we should spend time and money in rootkit detection. There will always be attacks that you can’t or don’t prevent and it is still useful to detect the traces of the whatever rootkit that the attacker installs. Refusing to spend effort to detect rootkits and relying solely on prevention is naive. Both are complicated and difficult problems, but you get better traction having a variety of efforts on both ends. However, we shouldn’t go trying to detect malware hiding in every possible nook and cranny. It is much better to detect their effects because no matter how stealthy the rootkit abuses SMM, hardware-assisted virtualization, or operating system drivers, it still has to actually *do* something and that may be detectable. An anomaly in process CPU usage accounting, file system usage, or network traffic may give it away.

Should researchers continue to explore where they can hide themselves?

	They should, if that research is used to directly drive product hardening or detection/protection/cleaning features in a security tool. Just finding problems really isn’t good enough anymore, that research and those results should drive improvements. And just finding some new, cool way to hide, publicizing it to show how smart you are, and not working with the vendor or tools vendors to detect or prevent it isn’t really contributing positively. And yes, I think we have a responsibility to not just identify problems, but identify fixes. As a pundit/analyst I know all about pooping on other people’s work, and the importance of making that work better after you wipe the brown stuff off.
	Absolutely. It isn’t an arms race if only one participant is running! You will not stop people from attempting to write better rootkits, and there will always be people working to better detect them. It is to my advantage that this happens “in the open” so I can benefit from all the hard work and competitive spirit of both sides.
	Yes and especially when it proves that existing tools, techniques, and products are insufficient. Someone needs to challenge product vendors to do better and make sure that everyone learns from past mistakes (i.e. relying on signature-based anti-virus for so long).

Will either make things better?

	Not really, but neither does anything else. We’re all dead in the end anyway (except me, I’m going the freeze-dry route). I mean how are you defining “better”. In eternal battles incremental victories are all we can hope for. There’s been crime and bad guys since long before any of us were around, and we sure as heck won’t be the ones putting them out of business. It’s about risk reduction. Yes, you’re totally pwned once someone can get a rootkit on your system, but you can at least minimize that damage under certain circumstances. But the truth is we should take a step back, look at the problem, and try and come up with solutions that fix the root cause. Rootkits aren’t the problem- exploitation of the systems is. Viruses, spyware, remote hacking, none of those are the real problems, they are just the vector. I’d rather see more investment in anti-exploitation technologies than rootkit detection, not that the two are mutually exclusive. It’s a matter of prioritizing the research, not dropping one for the other.
	Both combined will make things better. As time goes on, effectively hiding long-term access tools on systems will become more time consuming and less of a “sure thing”. This is excellent. Long-term access to high-value information processing systems has more value than just a snapshot of the data. This added value attracts a more dangerous type of “threat actor”. My strongest defense is to make attacking my information assets riskier and more expensive for The Bad Guys. An ongoing arms race will continue to make rootkits more complicated and less survivable. This effectively reduces the risk adjusted rate of return an attacker gains from them.
	The rootkit research attack/defense cycle results in much less collateral damage when it is between researchers and defense product architects rather than simply malware authors and product vendors. The improved products that stop researcher’s techniques also often make malware author’s job more difficult. You need both sides working together on the problem, otherwise it is just vendors shadow boxing with “the malware threat” that is only the malware that is detected by current techniques.
	Sweet, I can add myself to this anytime I want. Wish I had done that earlier. Well, allow me to wrap this up. Rootkit detection is a last-ditch strategy, after many other components failed you. It also appears to have a somewhat low success rate. What we should really be doing is focusing more on prevention. It’s simply too late when you are looking around for rootkits (especially if your software can’t find them or delete them). You are already screwed when someone installed it on your machine. Especially, when the attacker is in a position to leverage SMM or install a malicious hypervisor. It’s pretty much game over the second someone compromised your user account. You know the account with privileges to access all of your stuff.

Oh. If you were wondering who was on the blanel, some of the images are clickable.

10 Comments

TrailOfBits Weighs In On Mac OS X Vulnerability Stats

Dave G. | May 30th, 2008 | Filed Under: Industry Punditry

Before I comment. Yes, that’s right. Dino Dai Zovi is now a blogger. What’s next… twitter? I thought he only runs web browsers to find quicktime vulnerabilities?

Ok, enough tomfoolery. Dino writes about Mac OS X vulnerability statistics. Based on the information provided by Apple Security Updates, he shows:

Internal: 44
External: 53
Upstream: 84

Then gives a rule of thumb vis-a-vis internal vulnerability finding:

[vendors] should be finding and fixing at least as many vulnerabilities in shipping products as external researchers are.

I’d actually argue differently. Most vendors should have been finding these bugs before shipping. If you are continuing to find bugs in your product after you shipped it, you probably didn’t invest enough in security prior to release.

These statistics are also really hard for external folks to really know because, organizations patch things silently more often than you’d think. Sometimes it’s intentional, sometimes it gets fixed as a reliability bug or someone just changes the code. And that’s not all:

1. Just because an external party didn’t receive credit, doesn’t mean it was found internally.
2. Just because an external party did receive credit, doesn’t mean it wasn’t found internally.

Here is what I think it really fascinating about these stats:

Almost half of the individual flaws Apple has had to fix were due to code that Apple doesn’t actually own. I do not envy any vendor that has to manage that. It’s a nightmare for everyone.

This also extends past Apple. Everyday, enterprises purchase applications (think enterprise oriented web apps) that ship with third-party code (e.g. OpenSSL, mysql, apache, Oracle). How many of those do you think get updated regularly?

1 Comment

Race To Zero: It’s Not A Contest, It’s A Protest

Dave G. | May 5th, 2008 | Filed Under: Industry Punditry

Race To Zero is an event that pits hacker-types against an array of AV products. Unofficially hosted at DEFCON this year, it has already sparked the ire of the AV community. This makes sense as we all know that there is little they can do to stop researchers from writing malware that will be undetectable (until their next update). From their perspective, it is a waste of time. And that is somewhat true. Especially their time.

This type of event, along with the Consumer Reports test of 2006, runs the risk of wasting the AV community’s time. Which if we all recall, had no negative impact on society (or even AV vendors). Even still, I acknowledge it is a pain in the ass for them. A combination of bad press, plus a bunch of really crappy malware samples that have to documented, analyzed, detected and removed even though they will most likely never, ever impact a person outside of a lab environment.

The idea that the AV company’s are getting free research is pretty ludicrous. All that happens is that they will have to analyze as many of these modified viruses to figure out how to detect them. It is just another day at the office.

Which gets to the heart of the matter:

This contest isn’t a contest. This contest is a protest. It is a protest against the fact that there is simply not enough innovation in the anti-malware space. The problem is getting worse and all of the solutions appear to come from the same tunnel-vision line of thought. The vendors that do this have successful businesses that run just fine. New malware will get fixed with the same old solution.

The take-away isn’t going to be research that will help the AV industry to see emerging techniques. It will be that there has has to be another way. Events like this should inspire someone fresh to come in and build a better mousetrap, and build the next MFE or SYMC.

14 Comments

Coverage: Don’t Believe The Hype

Dave G. | April 22nd, 2008 | Filed Under: Industry Punditry

More and more I hear people discussing coverage in terms of security testing.  I am here to give you some bad news.  You will rarely get a genuine answer on how much Coverage you actually received.  It is dependent on approach, methodology, tools and skill set.  

51% percent of Wikipedia editors agree, the most common forms of coverage testing are:

• Function coverage - Has each function in the program been executed?
• Statement coverage - Has each line of the source code been executed?
• Condition coverage - Has each evaluation point (such as a true/false decision) been executed?
• Path coverage - Has every possible route through a given part of the code been executed?
• Entry/exit coverage - Has every possible call and return of the function been executed?

6 Comments

Cigital Ponders: Is Penetration Testing Security Testing?

Dave G. | April 9th, 2008 | Filed Under: Development, Industry Punditry

My initial response was: duhhhhhhhh

My second response was: uhm yes.

My third response was: Maybe I should read past the subject before I respond.

The short answer is still in the yes category.

The longer answer is that it is a form of security testing.

The blogging (and now featuring blockquotes) answer is:

Some people start “Security Testing” by buying and using a pen-test tool on project. Such tools uncover security vulnerabilities (though they seldom help with root cause analysis or even obtaining double-digit code coverage).

I would argue that uncovering security vulnerabilities is a major motivation for security testing. or Security Testing. Or even “Security Testing”. It is great to talk about Root Cause Analysis and code coverage, but i’ll take single digit code coverage over zero digit code coverage. Pen test tools are what I like to call Least Common Denominator testing. It will find the flaws you can expect an unsophisticated attacker to uncover. It is an excellent place for people who are “just getting started”. Because they don’t have anything in place, and want to hit the ground quickly. Rather than quote the next piece, I will try to sum it up. It is a classic security debate about blackbox/white box and requirements. If you read this blog you have heard it 1000 times and/or you already don’t read my posts.

The point I will nit pick on is:

Because black box tools to a large extent run canned tests they will not satisfy my security testing goal (see previous entry) of having run tests that one traces back to requirements. ‘Requirements that one created as a result of doing risk analysis that determines exactly what behaviors (and their impacts) should be avoided were the software attacked. Arguably, security folk have “cached” this risk analysis and these implicit requirements in the pen testing tool. Fine, this is that small benefit that I mentioned pen tests do provide. And, they DO find bugs. Again, this is at best a degenerate case of security testing in the same way running a fuzz testing tool is a degenerate way of conducting functional testing.

I’m really not sure this is “cached” or arguable. These tools discover flaws that are generally applicable to 99% of the requirements applications that need to get tested. Of course there are exceptions. But those are generally, you know, exceptions. Pen-test tools are useful. They identify a lot of vulnerability quickly and in a cost effective manner. Even if some of the flaws it finds aren’t applicable to a given app, most of them will be. And you can always say “Not going to fix it” for the flaws you don’t care about.Finally, your tests shouldn’t be limited to the tests that map back to requirements. Maybe that is controversial, but honestly, most of the times security requirements are:

1. Poorly documented
2. Vague or Poorly articulated
3. Non existent (Especially if you are in the “just getting started” phase)

When you are specific about security requirements, you are in the battle of staying current against attacks. When you are vague about security requirements, you become test cases become all inclusive.SDLC is great. If you are just getting started, you need to be pragmatic.

My final response: Pen test tools are not pen tests. both are forms of security testing. If the question is, “Is it enough?” or “Do these things work?”, then we have a different conversation.

2 Comments

Controlling ActiveX Controls

Dave G. | March 13th, 2008 | Filed Under: Industry Punditry

So, lots of talk about ActiveX controls these days.  I actually had a larger post in mind, but I am pressed for time, so here are some quick thoughts on why browser accessible ActiveX controls are so frustrating:

1. ActiveX controls aren’t (usually) tied to the websites that installed them.  Meaning, any website can instantiate one and communicate with it.  And by communicate with it, I mean perform memory corruption attacks that lead to remote code execution.
2. They are often written poorly.  Even more poorly than most 3rd party software.  Overflows, arbitrary file access… you name it.  You could probably find an ActiveX control that is actually vulnerable to every bug class.   
3. They persist (and can be difficult to remove(*)).  After they get installed, you forget about it.  Forever.  Long after you have even logged into the website that convinced you to install it.   Just waiting for someone to take advantage of issues 1 and 2 to make you part of their botnet.
4. They can be difficult to update.  Unlike a lot of software, ActiveX controls rarely have auto-update functionality.  As a result, most people that are vulnerable, stay that way. 
5. They are rarely necessary.  The worst part is, ActiveX controls are often add-ons that no one really needed and wouldn’t miss if they disappeared.  A lot of times that I have seen them used, they were mostly there to make a UI feel more Win32 and less webby.  The risk to benefit ratio has rarely been worth it.

(*) By difficult to remove, I am referring to the average user. 

15 Comments

Seven Deadly Pen Test Sins

Dave G. | March 11th, 2008 | Filed Under: Industry Punditry

This is a quick list of sins that I think most people that do this are or have been guilty of in the past:

1. Managing Time.  One of the most common yarns about the difference between pen testing and hackers is that pen testers have a limited amount of time to look for vulnerabilities in a small window of time in the applications’ life.  As a result, time is precious.  Knowing when you are on to something that you can confirm in a reasonable period of time is probably the biggest place where good pen-testers go bad.  It is really easy to turn a pen-test into a research project.  And while you toil away on a tiny sub-section of the application, you may never get to that remote code execution flaw that is lurking elsewhere.  This is also where coverage vs. depth plays a huge role.
   
2. Smugness.  Devaluing findings that customers care about.  Yes, XSS and CSRF are lame findings to people used to exploiting memory corruption or even compared to sql injection and auth bypass.   This also extends to “I found one, let the dev team find the rest of them”. Smugness can also be extended into overconfidence.  And overconfidence equals underestimation.  This all results into missing vulnerabilities that you needed to find.
3. Never understanding the app.  It is easy to just treat an application as a series of inputs, and not bother to understand what the application is actually for or what it is actually doing under the hood.  Good penetration testers are often trying to get into the developers’ heads.
4. Over-automation.  While I am a big fan of utilizing tools to make people more effective, there are two problems with relying on these tools:
   1. They generally create as much work as they eliminate.  False positives in the popular web app scanning tools are still common enough that you waste a lot of time, especially on a small website.  
   2. After you run them, you don’t have any better understanding about the application.  It encourages #3.
5. Sloth.  This raises its ugly head in a couple of ways.  It is usually either in avoiding the difficult parts of the test or conversely the easy parts of the test.  Total human nature.  One is hard and the other is boring.  As a result, usually the team that comes in after this sinner finds serious flaws that are either hard-core or are embarrassing.
6. Stagnation.  Given the difficulties of the job, it is easy to not evolve one’s skillset.  This is compounded by the fact that even in 2008 there still aren’t enough resources out there to keep evolving.   This is also an organizational problem inside of every company.  It is why the evil M word raises its ugly head.  
7. Communication/Soft Skills.  Where it’s a project manager or the customer, you need to understand what the customer cares about, manage their expectations, and oh yeah… if you can’t actually care, at least pretend to.  Lots of people think that doing a good job is simply breaking the app.   That is enough until someone who is as technically savvy as you rolls in and also has that polish (you know, the one that you can’t stand).  

9 Comments