Archive for December, 2006

taossa: A new must-read security blog!

Dave G. | December 28th, 2006 | Filed Under: Uncategorized

Happy holidays matasano readers! In your downtime you might want to be reading The Art of Software Security Assessment blog. It is a group blog with mark dowd, justin schuh and john macdonald. As you know, we think they are smart, and they were generous enough to do some book giveaways.

Comment Bubble No Comments

Things to do in Jersey When You’re Disgruntled

Dave G. | December 21st, 2006 | Filed Under: Industry Punditry

New Jersey doesn’t just grow amazing tomatoes, it also grows the most logic bombingest disgruntled admins in the USA (*). Today, the DOJ announced the arrest and indictment of Yung-Hsun “Andy” Lin. He is the third NJ disgruntled system administrator in 4 years who has tried to attack his employer via a logic bomb.

(*) This not fact checked statement.

The short story is that Mr. Lin wrote a logic bomb that was set to delete lots of business critical data (and more) on his birthday. Originally, the motivation was that he might get laid off. After he survived the layoffs, he didn’t remove the logic bomb, but eventually changed the date to go off one year later. Another admin found the destructive code and alerted the information security team. Let’s look at the timeline (I culled this from the indictment):

Medco Timeline

There are a couple of interesting parts about this timeline.

  1. Start of planning to indictment/arrest was 3 years.
  2. Malicious logic bomb sat in production for over a year undetected/reported.
  3. Detection to indictment was almost 2 years.

Point #2 is relevant for the rootkit discussion.  This logic bomb sat in production for a year, hiding in a shell script, not a hypervisor or kernel memory.

Comment Bubble 7 Comments

Checkpoint Buys Their Way Into Last Place

Thomas Ptacek | December 20th, 2006 | Filed Under: Industry Punditry, Uncategorized

Checkpoint is paying 20MM for last-place IDS/IPS vendor NFR.

Some scattershot punditry:

  • NFR has taken VC rounds bigger than 20MM. At a reasonable 5x trailing valuation, that says NFR made less money last year than niche startups like Lancope and Mazu, not to mention indie consultancies like Neohapsis, despite having a “full stack” decoding IPS implementation.

  • Checkpoint is merging their (failed) Interspect product line with NFR Sentivist. One possible outcome: NFR’s technology becomes the basis for yet-another-NAC-in-a-box product like Consentry

  • Speaking of hardware: will Checkpoint maintain NFR’s hardware OEM? You may not have noticed, because you hadn’t looked at an NFR box in 7 years, but Sentivist now runs on an OEM’d multiprocessor, custom-bus, blade-based appliance that looks suspiciously like SourceFire’s.

    The Checkpoint/Sentivist platform is exactly the hardware platform Steinnon has been nagging them to chase. It’s within Checkpoint’s power to buy the OEM supplier. That would be dumb: Checkpoint needs to think about what tomorrow’s hardware problem is, not what NFR and (perhaps) SourceFire thought it was a year ago.

    NFR’s spec sheet also posts thoroughly disappointing stats for the heat those boxes generate.

    Regardless, Checkpoint claims NFR’s OEM’d hardware as a “core technology” asset.

Much as I dislike the NAC premise, the story I like most here is that the NFR purchase revitalizes Interspect and creates a three-way race between Juniper, Checkpoint, and Cisco. In terms of port density (though not cost), a Checkpoint Sentivist NAC box would lead the majors. I hear Nevis and Consentry aren’t setting the world on fire; maybe they get chopped up between CSCO and JNPR.

Some blog reactions:

  • Alan Shimel calls this a further example of consolidation. I disagree: nobody goes to Checkpoint for the IPS budget item, and I imagine that for the past 2 years, very few people have gone to NFR either. We don’t have fewer IPS offerings, and we don’t have two IPS offerings competing within a single vendor; what we have now is (slightly) enhanced credibility for a languishing (and underrated) also-ran.

  • Richard Bejtlich misses Intrusion in his list of IPS vendors, which I mention so as to point out that INTZ probably had more revenue than NFR in 2005. I strongly disagree with his prediction that Cisco will buy SourceFire; Cisco’s IPS gets a ferociously bad rap among the pundits but, from what I can tell, a lot of love internally. I’ll also point out that MCAF is one of the top 3 IPS vendors.

Comment Bubble 12 Comments

Is Open Source Rootkit Detection Behind The Curve?

Dave G. | December 19th, 2006 | Filed Under: Slashdot Rounddown

I was reading this post about rootkit detection on Linux. They go on to talk about chkrootkit (an NMAP top 100 tool!) and rootkithunter. After doing a couple minutes of research, I was pretty sad. As far as I can tell the majority of the techniques are limited to signature style detection. The more ‘hardcore’ tests:

Every operating system has is advantages and disadvantages like the differences between tools en disk structure. Some parts of an operating system are not available to others, so we can use not all tests every time.

Linux - compare processes in ps against the available files in /proc

FreeBSD - look for differences between the output of netstat and sockstat

It seems like we haven’t progressed much since 1996 on rootkit detection. Or maybe this is just an open source issue. Or maybe I didn’t dig deep enough. Because in 1996 i seem to remember people at least doing:

  1. connect()/bind() to every port and compare that to the output of netstat.
  2. kill() every process with an innocous signal and look for pids that exist but dont appear in the output of ps.

I know there were others, what else am I missing? Yes, I know these techniques aren’t particularly effective against more modern rootkits. Yes, I know, this is a game of constant catch up. But we are further behind than I thought. This isn’t meant to be a dig on the authors or the tools. I am genuinely asking if there are smarter things we can be doing.

In other news, we are really busy, and the holidays are fast approaching, so our apologies for the lack of posting lately.

Comment Bubble 21 Comments

BeanSec IV: Now with more notice!

Dave G. | December 13th, 2006 | Filed Under: Gatherings

Oliver Day and Christofer Hoff have set the next BeanSec for December 20th. Even wiser, they have decided to make the meeting occur on a regular frequency.

We’ve now established a regular schedule for the BeanSec! events; the third Wednesday of each month. So you can mark your calendars now for December 20th at 567 Massachusetts Ave. Upstairs at the Enormous Room from 6-9pm.

For the uninitiated:

BeanSec! is an informal meetup of information security professionals and academics in the Cambridge/Boston area. Unlike other meetings, you will not be expected to pay dues, “join up”, present a zero-day exploit, or defend your dissertation to attend.

The location is the Enormous Room (map) in Cambridge.

Interested in starting one in your town? Send emails to blog@

Comment Bubble 2 Comments

Mystery Vulnerability Theater 3000: Part I

Dave G. | December 12th, 2006 | Filed Under: Reversing

Tom was telling me about a really quick way to do risk assessments on memory trespass/register control vulnerabilities. As it happens, the next day I found myself controlling some registers in an application, and figured I would see where it got me.

The basic idea is that when you find a vulnerability and don’t have the time to confirm exploitability via an exploit, you can mock it up in a debugger by setting breakpoints just before the instruction that causes the exception, and resetting the register you control to a valid memory address (I chose the stack).


sitio:~ root# !FOO
FOO=`ps -auxwww | grep XXXXX | grep -v grep | awk ‘{ print $2 }’` ; gdb -pid $FOO XXXXX
GNU gdb 6.1-20040303 (Apple version gdb-384) (Mon Mar 21 00:05:26 GMT 2005)
Copyright 2004 Free Software Foundation, Inc.
GDB is free software, covered by the GNU General Public License, and you are
welcome to change it and/or distribute copies of it under certain conditions.
Type “show copying” to see the conditions.
There is absolutely no warranty for GDB. Type “show warranty” for details.
This GDB was configured as “powerpc-apple-darwin”…
Attaching to program: `XXXXX’, process 23276.
Reading symbols for shared libraries …………………………………………………………………………………………. done
Reading symbols for shared libraries ……………… done
0×9000ab48 in mach_msg_trap ()
(gdb) cont
Continuing.

We have attached and the process is running as normal. Lets see what happens when we send some bad input:


Program received signal EXC_BAD_ACCESS, Could not access memory.
Reason: KERN_INVALID_ADDRESS at address: 0×62626262
[Switching to process 23276 thread 0×7103]

0×62626262, eh? BBBBadness.


0×0224f864 in SomeRandomFunction ()
(gdb) bt
#0 0×0224f864 in SomeRandomFunction ()
#1 0×02257d2c in TotallyRandomFunction ()
#2 0×022570c0 in ReallyRandomFunction ()
Cannot access memory at address 0×61626364
Cannot access memory at address 0×6162636c

Yah I know, the 0×61626364’s look tempting… just ignore them.


(gdb) disas _SomeRandomFunction
Dump of assembler code for function _SomeRandomFunction:
0×0224f84c <_SomeRandomFunction+0>: mflr r0
0×0224f850 <_SomeRandomFunction+4>: stmw r29,-12(r1)
0×0224f854 <_SomeRandomFunction+8>: stw r0,8(r1)
0×0224f858 <_SomeRandomFunction+12>: mr r29,r3
0×0224f85c <_SomeRandomFunction+16>: stwu r1,-80(r1)
0×0224f860 <_SomeRandomFunction+20>: li r3,0
0×0224f864 <_SomeRandomFunction+24>: lwz r4,0(r29)
0×0224f868 <_SomeRandomFunction+28>: bl 0×22547f4
0×0224f86c <_SomeRandomFunction+32>: lwz r0,88(r1)
0×0224f870 <_SomeRandomFunction+36>: addi r1,r1,80
0×0224f874 <_SomeRandomFunction+40>: mr r4,r29
0×0224f878 <_SomeRandomFunction+44>: mtlr r0
0×0224f87c <_SomeRandomFunction+48>: li r3,0
0×0224f880 <_SomeRandomFunction+52>: lmw r29,-12(r1)
0×0224f884 <_SomeRandomFunction+56>: b 0×22547f4
End of assembler dump.
(gdb) quit
The program is running. Quit anyway (and detach it)? (y or n) y
Detaching from program: `XXXXX’, process 23276 thread 0×7103.

Looks like we grabbed control of a register. But it is control of r29. Since we are lazy, and want to see if we can quickly confirm exploitability, lets try something:


sitio:~ root# FOO=`ps -auxwww | grep XXXXX | grep -v grep | awk ‘{ print $2 }’` ; gdb -pid $FOO XXXXX
GNU gdb 6.1-20040303 (Apple version gdb-384) (Mon Mar 21 00:05:26 GMT 2005)

Attaching to program: `XXXXX’, process 23324.

0×9000ab48 in mach_msg_trap ()
(gdb) break *0×0224f860
Breakpoint 1 at 0×224f860
(gdb) cont
Continuing.
[Switching to process 23324 thread 0×8403]

Same as before… only this time we set our break point to one instruction prior to the invalid access prior to continuing execution. Now we send the same malicious input, and…


Breakpoint 1, 0×0224f860 in SomeRandomFunction ()
(gdb) x/2i 0×0224f860
0×224f860 <_SomeRandomFunction+20>: li r3,0
0×224f864 <_SomeRandomFunction+24>: lwz r4,0(r29)

(gdb) set $r29=0xbffffd30
(gdb) cont
Continuing.

Now that we have hit our breakpoint, we modify the value of r29 back to a valid memory address and continue. Lets see what happens next…


Program received signal EXC_BAD_ACCESS, Could not access memory.
Reason: KERN_INVALID_ADDRESS at address: 0×6262626a
0×02257f6c in SomeWhereElseRandom ()
(gdb) bt
#0 0×02257f6c in SomeWhereElseRandom ()
#1 0×02257d44 in TotallyRandomFunction ()
#2 0×022570c0 in ReallyRandomFunction ()
Cannot access memory at address 0×61626364
Cannot access memory at address 0×6162636c
(gdb)

Oh look, we’ve moved past it. Lets check out our execution window:


XXXXX(23324,0×2842a00) malloc: *** Deallocation of a pointer
not malloced: 0×64676f6c; This could be a double free(), or
free() called with the middle of an allocated block; Try setting
environment variable MallocHelp to see tools to help debug
XXXXX(23324,0×2842a00) malloc: *** Deallocation of a pointer
not malloced: 0xbffffd30; This could be a double free(), or
free() called with the middle of an allocated block; Try setting
environment variable MallocHelp to see tools to help debug

Alas, calls to free() on OS X are validated to make sure they are pointing to currently malloc()ed memory. This doesn’t necessarily mean that what I have found isn’t exploitable. But I am not off to a good start.

What I like about this technique is that you can confirm whether something is exploitable quickly.

What I dislike about this technique is that you can’t confirm something isn’t exploitable. For example, assuming that it was a double free implementation that I could exploit, the application in question tosses my user input through an isalpha(). I wouldn’t have known that by solely relying on this trick.

Live and learn…

Comment Bubble 6 Comments

NYSEC 5 Reminder

Dave G. | December 11th, 2006 | Filed Under: Uncategorized

Tonight is NYSEC 5. A perfect opportunity to drown out your monday with beer and horror stories from other security professionals.

NYSEC 5 will be held (as always) at Pound and Pence [map] in the financial district.  NYSEC convenes upstairs by the pool table.  It starts at 6:30PM and generally goes till 9:30.

Comment Bubble 3 Comments

Dead Giveaways of Thoughtless User Interfaces

Thomas Ptacek | December 9th, 2006 | Filed Under: Development, Uncategorized

Submitted for your consideration:

There are “features” (or design choices, or flourishes) that are acid tests of how thoughtful and in-tune a development team is.

For example: consider an MP3 player that sorts a playlist by artist in alphabetical order. Acid test: is the article “the” disregarded, or do all the bands that start with “the” clump together?

A security product example: when timestamps are displayed, are they displayed in ctime() format, like “Thu Nov 24 18:22:48 1986”? Even when the event happened 20 minutes ago, or yesterday, or when the problem domain doesn’t care about seconds precision?

Another example: when traffic stats are displayed, are they presented in bytes-precision, as if the UI just did “sprintf(output, “%Lu”, counter)”, even in the (overwhelmingly common) cases where I just want to compare KB, or MB, or GB numbers?

Talk amongst yourselves.

Comment Bubble 6 Comments

Matasano Security Recommendation #001: Avoid Agents

Thomas Ptacek | December 8th, 2006 | Filed Under: Defenses, Uncategorized

Alan Shimel at StillSecure advocates for a variety of agent-based security systems. Rather than walk into the swamp of a security product marketing debate, I’m going to use his post as an opportunity to make an unambiguous recommendation.

Summary of Recommendations

Enterprise security teams should seek to minimize their exposure to endpoint agent vulnerabilities, by:

  1. Minimizing the number of machines that run agent software.

  2. Minimizing the number of different agents supported in the enterprise as a whole.

Background

Endpoint agents are programs that run silently in the background, usually as Windows Services or Unix daemons, which communicate back to a central management system. Well known examples include:

  • Systems Management (BMC Patrol, CA Unicenter, Microsoft MOM)

  • Antivirus (McAfee, Symantec)

  • Patch Management (Novell ZenWorks, SDS, BigFix)

  • Data Leakage Prevention

Agent-based architectures are a severe security risk. Risk is amplified as more agent-based products are deployed. In enterprises with pervasive agent deployments, attacks on agents are more threatening than attacks on underlying operating systems.

Rationale

Vulnerabilities

Agent-based architectures share a tendency towards unusually complex and hazardous attack surfaces:

  • Listening Network Services on Agents

  • Listening Network Services on Management Servers

  • Client of Agent Service on Management Server

  • Confidentiality and Integrity of Agent/Server Protocol

  • Web Application on Management Server

  • Javascript on Browser Client of Management Server

  • Listening Network Services for Management Clients on Management Server

  • Middleware Frameworks and RPC

  • Client of Management Server Service on Agent

  • Display Logic for Agent-Sourced Data on Management Client

  • Confidentiality and Integrity of Client/Server Protocol

  • Databases

Often, the complexity of the attack surface in agent-based solutions is concealed by turn-key installation and management features. Servers may use embedded-but-listening database servers such as MSDE/Jet, or tunnel DCOM or CORBA in HTTP POSTs.

Virtually every one of these attack surface features is recognized by the industry as a difficult security challenge independently. Deployed in concert, safety concerns are multiplied.

Threats

The deployment circumstances of agent-based products make them an obvious and lucrative target for attackers:

  • Agents tend to be installed en-masse. Attacks that offer uniform compromise of all installed agents provide attacks with thousands of hijacked machines.

  • Even in the absence of an exploit that compromises agent software directly, it is impractical to ensure the security of thousands of endpoints. But every machine running an agent must be secured if the management components are to be shielded from attacks.

  • In a majority of surveyed agent-based systems, compromise of a single management server allows code execution on every agent, exposing the enterprise to a single point of failure.

  • Agent implementations are often substantially homogenous, even across operating systems, enabling uniformly effective attacks against desktops, Windows servers, and Unix servers.

  • Workstations of management operators are high-value IT targets, and compromised agents can inject poisonous data to exploit a myriad of clientside and XSS-style attacks to hijack their machines.

Empirical Evidence

Matasano has surveyed a broad array of different agent implementations, and conducted in-depth penetration tests of more than $4Bn/USD of shipping product and more than ten different vendors. In only one case did a vendor survive a penetration test without a “game-over” vulnerability that would have transformed a deployment into a latent botnet; that vendor had spent tens of millions of dollars over the past year to institute a security development lifecycle and had repeatedly audited its agent-based product.

In almost every other case, we found no evidence to suggest that these high-risk products had ever endured any outside security testing. Classes of vulnerabilities uncovered included:

  • The full gamut of C/C++ runtime vulnerabilities, including simple stack overflows.

  • Metacharacter bugs, including protocols with Unix backtic command execution.

  • Undocumented and unauthenticated protocol commands that enabled command execution or agent reconfiguration.

  • Untested custom encryption protocols, including unprotected key exchanges, insecure cipher modes, replay attacks and crypto bypass attacks.

  • Trivial XSS and SQL injection vulnerabilities that could be triggered from any installed agent.

Matasano presented an overview of these findings at the Black Hat Briefings in August of 2006.

Mitigating Factors

Enterprise security teams required to support agent-based software should prioritize the following objectives for 2006/2007:

  1. Move high-value assets off of servers that must run management agents.

  2. Move high-risk agents, such as laptops and Windows desktops, to seperate management domains with seperate management servers and seperate operations teams.

  3. Obtain detailed documentation for all the protocols used by agents, including the security countermeasures employed by those protocols.

  4. Aggressively filter agent protocols, with host-pair specificity, at every network security border where such filtering is practicable.

  5. Isolate agent management deployments to individual network segments or security domains.

  6. Enable SSL protocol options and institute SSL client certificate access control to ensure all protocol participants at least bear a signed certificate.

  7. Downselect candidate agent management applications to those that have undergone documented third-party security testing.

Conclusion

Agent-based architectures are incredibly convenient and can be a significant cost-saver for IT operations teams. Security teams should assume they already support an unweildy variety of agent software platforms, and should assume that they work under substantial organization pressure to support more agents. It’s therefore critical that security teams crystallize a strategy to minimize the security impact these risky products present.

In all circumstances, enterprises should seek to minimize the number of agent installations within their enterprise.

In all circumstances, enterprises should seek to minimize the number of different agent-based vendors their enterprises must support.

Agent-based software should be treated as a high-risk target for attacks. Agent software warrants intensive security testing and analysis and rigorous access control.

Comment Bubble 28 Comments

Join Our Team

Dave G. | December 8th, 2006 | Filed Under: Matasano

Join Us!
Matasano is expanding. Are you interested in:

  • Presenting new research at conferences like Black Hat and RSA?
  • Shipping commercial software to enterprise security teams in
    C and Ruby/Rails?
  • Reverse engineering network protocols and embedded operating
    systems?
  • Break and secure complex, interesting products
    used by millions of people?
Apply Directly To The Forehead!
You’re reading this on our blog. We write about what we do. Does this
stuff interest you? Then we want to talk to you about joining our
team. Our ideal new team member can:

  • Code (in C and Ruby or Python)
  • Consult (on engagements with F-100 companies)
  • Write (as one of the voices on this blog)
  • Speak (at security conferences)
  • Conduct independent research (in pertinent security topics)
Join Us!
Matasano is a small, close-knit team of security professionals based
in New York and Chicago, combining cutting edge product development
with advanced security consulting services. Work on next generation
security technologies! Help major vendors in the computer industry
improve their security! Antagonize security snake oil vendors!
Interested? careers@matasano.com

Comment Bubble No Comments

Who We Are

Matasano is a team of internationally respected security experts who have led security efforts at @stake, Microsoft, ISS, Secure Computing, Arbor Networks, Secure Networks, Bloomberg, Sandia Labs, and others. Read more about our team and how we can help you today.