Network Patching Is Not An Alternative To Third-Party Patching, Chris.

Thomas Ptacek | September 27th, 2006 | Filed Under: Defenses, Industry Punditry, Uncategorized

Chris Hoff writes (ostensibly) about third-party patching, employing a paeon to Bleu Lane so wrong headed that I’m left to wonder whether pod people have taken over his brain.

Bleu Lane, for those of you who don’t keep track of these things, sells something they call “inline patching”. The idea is, well, I’ll let their pictures explain it for you:

[before]

[after]

You see what they did there? The box takes in shellcode, and then, by “emulating the functionality of a patch”, spits out valid traffic. Wow. That’s amazing. Now, somebody please tell me why that’s any improvement over taking in shellcode, and then, by “emulating the functionality of an attack signature”, spitting out nothing?

Yes, the problem with Bleu Lane Patch Point is that it doesn’t seem to be anything more than a slick marketing story over conventional IPS. Even the message is derivative; ISS used it for RealSecure four years ago. But unless you fund a marketing team millions of dollars to lie by repeated assertion, nobody believes that this is the equivalent of patching.

But that rant is actually a digression from my actual problem with Hoff’s post. Because even if it worked, Bleu Lane has nothing to do with third-party patches.

The power of a third-party patch is that by preempting the vendor, vulnerabilities can be disclosed without waiting. Real third-party patches are inherently “zero-day” patches; what to do with a zero-day patch is a decision you’ll have to make days before an IPS engineer figures out what to do with it.

I’m willing to believe that Chris’ bacon has truly been preserved by a PatchPoint appliance (though I’ll point out that if he bought one two years ago, he apparently had a beta: PatchPoint launched in September ‘05). He’d be telling one of the first stories of someone getting actual protective value out of an IPS, but I’m willing to believe it. I’m also willing to believe that Chris’ interests are profoundly aligned with promoting network-based solutions to security problems, and that they tend to be aligned with promoting network vendors as well.

Am I wrong about PatchPoint? Let me know.

Viewing 25 Comments

    • ^
    • v
    If you have a choice, why would you want to be forced only to block when it comes to "suspicious" traffic? Why force yourself into blocking decisions only when it comes to false positives and negatives?
    The vendor patch corrects (defangs) all traffic... why not put such a capability in place as soon as possible until your team can patch? Yes, Chris was an early customer of Blue Lane. He was a case study before he went to his new venture. Does everyone have to have ISS pedigree to be truly objective about the capabilities of a patch proxy?
    • ^
    • v
    Greg, I'm a big skeptic about Blue Lane. I could just list the reasons here, but if Blue Lane people read us (they should!), why don't we give you equal time to respond? Email interview. Or moderated debate (I nominate Peter Lindstrom from Spire Security as moderator).

    Gotta say, "ISS pedigree", pretty funny. =)
    • ^
    • v
    I have to go host BeanSec! so the beatings will have to continue tomorrow when I sober up. I'll respond via my blog because...well, I can't put up cute pictures in my response here.

    I think you actually implied I was a network-peddling-ho, didn't you? ;)

    ...and then you bring out the big guns...Lindstrom...he already moderated a panel about this -- I was on it. So was Mary Ann Davidson and a host of others.

    Talk later.

    Hoff
    • ^
    • v
    Since I'm a vendor (and even more jaded that former customer Chris Hoff lol) I'll let a few paragraphs from Andreas Antonoupolis' NWW newsletter address the advatnatge of correction over the "all you need is block" song your composing for us.

    >>>One solution to the patching conundrum is an inline “patch proxy” or “network patch” appliance. A patch-proxy device changes the network traffic, applying a transformation to the packets that is functionally equivalent to a patch.

    For example, if a vulnerability is found in a piece of code that does not properly limit the length of a string or memory access call (a.k.a. a “buffer overrun”), the patch proxy can truncate the equivalent string in all packets. In most cases the truncation will not have any effect because the string will not exceed the server’s buffer limitations - that is, the traffic is legitimate and well formed.

    Some packets will be transformed to truncate the data because the packet contains an exploit for the buffer overrun. But this would also apply to packets which contain long strings sent due to misconfiguration rather than malicious intent. Regardless of the intent of the sender, all the packets undergo a transformation that is functionally equivalent to the patch. The patching proxy thereby patches the packets in the network so IT doesn’t have to patch the servers.

    There are two main advantages to this approach. First, a patch proxy can “front” many servers, providing patching services for all of them. IT administrators can either leave the servers un-patched permanently or more likely can postpone patching until the patch has been thoroughly tested.

    In the meantime, the patch proxy provides inline protection so that the un-patched servers are not vulnerable.

    The other key advantage is that there are no false positives or false negatives. The patch proxy is not looking for malicious traffic and does not decide whether a packet is malicious or not. Instead the patch proxy simply transforms all packets with the functional equivalent of a patch. By applying a transform to all packets, dangerous packets are “defanged.”
    • ^
    • v
    Here is a link to the article I just quoted:

    http://www.networkworld.com/cgi-bin/mailto/x.cgi
    • ^
    • v
    I've responded on my blog.

    I like their product. It solves a serious problem.

    I have ZERO interest in Blue Lane other than I think it's an overlooked answer (from a different perspective) to the patching problem -- third party or otherwise.

    I'm over the fact that you're essentially accusing me of shilling their product despite the fact that the only relationship I have with the company is that I was a customer @ my former job. Nice.

    I took it more personally than I should have because I had too many Bourbons. Oh well. My skin was thinner than it usually is.

    Pod people notwithstanding, I understand you're allergic to their approach, but this is not an IPS...

    Your post is lacking tech. It's a wonderfully inflammatory read, but you've not demonstrated why it's a bad thing to do.

    I'm VERY interested in hearing why you think it is and I know I'll appreciate your insight, even if I disagree with you given your background.

    Hoff
    • ^
    • v
    I spent some time with the BlueLane folks many moons ago and was very skeptical at first, but it does what it reports to do (scalability, accuracy, centralized administration, reporting capabilities, ease of use, and all the enterprise functionality folks need require due diligence against their own unique requirements)

    I would assume that everyone will agree on 3 things:

    1. Organizations simply cannot patch some elements of their environment. They cannot patch for business reasons, logistical reasons or because a patch is not available or a valid solution. Even best of breed organizations can take weeks to months to deploy patches, even if they are not careful about ensuring it doesn't break something (anyone remember NTSP6? No? Well it was replaced by NTSP6a - see it broke lotus notes). Oracle databases are a good example of an enterprise element that is notoriously difficult to patch for many, many reasons.

    2. An attack-facing IPS will potentially block legitimate traffic, and not block malicious traffic - they are good at blocking known bad things, reasonable at allowing known good things, but most traffic sits in that grey area and most enterprises only turn on the known bad blocking stuff, essentially they only use about 10-20% of the IPS's purported functionality. So they are great at protecting you against the blaster worm but struggle when it comes to targeted, zero-day, or recently introduced exploits, when the IPS doesn't yet have a signature and the organization doesn't buy the marketing hype about their anomaly detection capabilities and just let it block stuff.

    3. If it works, an inline patch proxy that does not block traffic but allows for connections to flow, business to continue, and transactions to be completed while reducing the potential for someone to exploit the end-points is a real viable option.

    They face 2 main problems though; one is that they sit between 2 mature and well-understood markets in IPS and Patch Management so people will look to put them into one of those buckets, like most companies marketing is key to their adoption - not slimy, snake oil marketing (not that they have that) but really focusing on how to properly position themselves in the greater security eco-system and overcoming the significant skepticism most technical folks would justifiably have. Second is logistical, decomposing patches and writing their signatures with a fast-enough turn-around time to be preventative in an increasingly hostile environment requires some pretty serious logistics, not to mention some cooperation with large ISV's like Microsoft, and Oracle.

    Bottom line: The solution is worth a look. Due diligence is mandatory, especially since it is an inline device most likely being placed in front of some critical infrastructure, but I would recommend folks take a look and make up their own minds.

    Disclaimer: I am an analyst (pundit as Ptacek calls us) who placed BlueLane in a cool vendor note for Gartner in 2006 (I also placed Voltage and Core Security - so Tom and I agree on those) so my opinion is tainted with punditry :-)
    • ^
    • v
    I'll pass on the moderator bit. This is more of a one-sided attack as far as I can see.
    • ^
    • v
    I have similar problems with Blue Lane and wrote up my problems with it in both a blog entry[1] and my MSc thesis[2].

    [1] http://singe.rucus.net/blog/archives/693-Blue-L...
    [2] http://singe.rucus.net/masters/thesis/

    Given that I quoted Thomas, I think this neatly completes the karma circle.
    • ^
    • v
    Dominic and Thomas:

    IMHO the most brilliant "marketing" has been the creation of the "virtual IPS patch" that performs so few patch corrections that it would give PT Barnum the woolies. Even calling it the equivalent of a patch based on signatures is a misnomer. If I didn't have so much respect for the two of you I would suggest that you scrutinize all those false alarms you're getting from Blue Lane's cyberspace buzz a little more closely... as you're practising the very kind of "signature" driven detection that you've lamented.

    The patch proxy actually makes the corrections of the patch by using app logic. It's the equivalent of understanding context before taking any actions. For example, it doesn't block (or bash) something merely because of a resemblence to something else.

    I think that's the deeper lesson here.

    And last but not least: thank you Chris for being a great former customer; thanks Amrit, Andreas, Richard, Art, Steven, Jim, John, Terry, Tim, Andrew, Zeus, Eric, Peter and the rest of you for taking the time to approach us, study us and share your qualified opinions in person, papers and/or the press versus merely slapping Blue Lane with a false block from the blogsphere!
    • ^
    • v
    I have to stick my nose in just cuz I smell a good brawl coming on.

    So, yes there is a strong similarity between the typical IPS and virtual or in-line network based "patching". If this is a semantics flame war I am out of here.

    Most of the IPS vendors I have talked to including Reflex, Tippingpoint and TopLayer deploy IPS signatures that are generic when a new vulnerability/patch is announced. Thus they defend a system before it is patched.

    But, as I understand it, by focusing on virtual patching BlueLane *manages* the patch process. By specifically identifying which patches are deployed to defend which assets you can work with your internal patch schedule to become systematic and therefore save money while still being protected. I like that.

    -Stiennon
    • ^
    • v
    I wish we could stop calling these things "patches". They aren't that.
    • ^
    • v
    That is a fair comment.

    Blue Lane offers a network proxy for a patch... or an inline patch proxy. They apply the application logic of the vendor patch in the network without having the operational risks associated with patching too fast (or reboots, etc). You also don't have tuning, false positives or false negatives. You have streams of defanged traffic. You correct the traffic, just like the patch does.

    Tell me Thomas that you do understand the advantage of correcting ALL traffic on the fly versus being forced into a binary "to block or not to block" decision based on sigs, anomalies or any other means of determining user intent?

    The computational advantage of traffic correction in the network for a known set of software vulnerabilities... versus determining the INTENT of traffic from a growing set of end points, evasions and types of possible attacks against an increasing complex network seems intutively obvious. One is using higher intelligence (app logic corrections) on a limited set of parameters while the other is using growing lists of patterns, anomalies and known bad guy lists against... the collective force of human ingenuity (hactivism).

    Do you think the computational requirements between these two very different approaches are similar?
    • ^
    • v
    I'd rather reserve my sharpest critique for after I've gotten to know the technology better, but: what you say is an advantage (traffic is scrubbed instead of being dropped) I say is a disadvantage (a black box in the middle of the network is corrupting traffic, rather than doing something that network and security engineers can understand.

    In fairness: I can be convinced that this is a good approach. But my starting position is, it offers new risks with few compensating rewards.

    My problem with the word "patch" is that what you do is less than a patch; for instance, there are numerous bugs which, to my mind, seem extremely hard to "patch" with an inline box. How do you handle race conditions?
    • ^
    • v
    Excellent question. We can handle race conditions in the same way that a server patch can. As you know there are a variety of race conditions and responses, so we can discuss in more detail during our briefing. Keep in mind a patch proxy is a proxy for the patch...

    Have a great weekend and thanks for giving us an opportunity to talk about Blue Lane.
    • ^
    • v
    Thanks for seeing this as an opportunity to talk to people about your product. Very two-point-oh-eey of you!

    =)
    • ^
    • v
    I'm a bit late here, but I'm travelling in Europe, so I don't check my personal emails to often :-)

    As somebody who used to architect various inline protection methods I would have to side with Thomas.
    This is nothing new. It's just a marketing spin for something that has already existed with a couple of different bells and names :-)

    You can do something in only so many ways... This can be also called a network proxy that uses "vulnerability signatures" (sounds familiar doesn't it :-))) ) instead of traditional attack signatures. Even the term "virtual patch" is nothing new ISS has beem using it forever (going to their website can easily confirm it).

    Partially rewriting content instead of blocking the whole flow is nothing new either.

    I'm curious how they deal with SSL, SSH, and IPSec traffic. I very much suspect that they don't deal with it :-)

    Having said that it doesn't mean that this is useless, but it definitely nothing groundbreaking, new, or perfect as the marketing claims it to be (but that's their job :-))) ).
    • ^
    • v
    BTW- we've briefed numerous analysts and security experts (including Thomas) and most have come to a different conclusion (about how robust Blue Lane's correction capabilities are compared to the status quo)... and our freedom from the signature scramble game. If you get curious enough to dig deeper we would be happy to brief you as well.

    Sincerely
    Greg
    • ^
    • v
    Greg, it may sound a little bit over the board, but if Blue Lane can provide sufficient technical information there's a good chance that I can prove that the core technology is nothing new (which I understand is not in your interests, of course). It's very likely it's an implementation variation on something that has already been tried. I have spent a significant amount of time researching the existing detection and prevention technologies in addition to developing new ones and based on the publicly available information about Blue Lane's technology this is not something completely new and ground breaking. In no way am I comparing it to string/regex signature based systems that try to detect bad traffic, which are, for some reason, considered to be the only mainstream solutions out there; however, I'm yet to see a single feature that doesn't exist in some other product.

    As far as signatures go... the term "signature" can mean pretty much anything. Anything could be called a signature... even Blue Lane virtual patches.

    As for virtual patches... the idea if flawed from the very beginning. It may work for simple cases; however, the concept fails for complex scenarios and bugs. A patch is useless without the exact environment it's meant to be applied in. In other words for the patch to do its job properly you need the complete environment, which means that unless you completely reproduce the vulnerable application there's a good chance that the virtual patch is not going to do its job.

    You said you briefed numerous analysts and security experts... The question is what was the extent and the goal of the briefings and what kind of comparative analysis has been done. I honestly don't see anything new in Blue Lane's technology even though it doesn't fall into the so called traditional IPS category.
    • ^
    • v
    Kyle:

    If you get curious enough to dig deeper let me know. Drop me a line, introduce yourself and we'll give you better data in which to evaluate Blue Lane. The offer still stands.

    Sincerely,
    G
    • ^
    • v
    Blue Lane CEO Jeff Palmer interviewed in 11/23/06 Network Computing FYI

    http://www.networkcomputing.com/showArticle.jht...
    • ^
    • v
    Two emails and no response what so ever... I guess they changed their mind. Good for them :-)
    • ^
    • v
    http://www.infoworld.com/article/06/11/30/49TCb...

    Roger Grimes reviews Blue Lane PatchPoint.
    • ^
    • v
    Kyle:

    Just saw your post. Who are you emailing? Thomas knows how to reach me... or send an email to info@bluelane.com and we'll get it. My apologies for any confusion... I'll look for an email from Kyle C. Quest...

    Greg
    • ^
    • v
    http://hackreport.net/2006/12/08/changing-the-g...

    Martin Hack report on Blue Lane Dec 8 2006

Trackbacks

close Reblog this comment
Powered by Disqus · Learn more
blog comments powered by Disqus