Joel Snyder Follows Up. Matasano Provides The Missing Subtext.

Thomas Ptacek | February 14th, 2007 | Filed Under: Industry Punditry, Uncategorized

Context. Thousand-Dollar-Joel follows up; from Slashdot.

I’m sure that if they’re serious about actually showing that the statistics are useful then we can find 10 random sites who are willing to be ‘ethically hacked.’ […]

…But IDG doesn’t have any of those random sites. And neither does “Opus One”, my IT Consultancy. Or any of my clients…

The astonishing thing is that most people who will read this press release just don’t get it, and the depths of their not getting it are even more astonishing…

… as you will see in just a few grafs…

I am challenging the conclusion, not the data. I believe that they think that they have found vulnerabilities. I suspect they have found a lot of lousy code. No surprise here. 70%, sure. I’ll bite off on that number. I’m not arguing with that. […]

… at least, not anymore, now that I’ve been called on it…

But there is a huge difference between turning a vulnerability into a breach. Let me give you an example. A lot of Cross-Site Scripting attacks let you steal cookies. So they probably found those. But the question is: when you have a cookie, what can you do with it? Can you steal important data? Can you turn that cookie into a breach?

… I’m asking because I have no idea…

Good web sites that use them also tie cookies to your IP address, which means that if you steal my cookie, you got nothing but crumbs.

… I’m saying this because I’ve never looked at a mainstream web application, or noticed that when I plug my ThinkPad in to my home network after coming home from work, many of my web app logins magically continue to work, as if by magic, magically…

… also I think IP addresses count as authentication…

So the point is not that there are these vulnerabilities, but that they have done nothing to show whether these vulnerabilities are truly breachable and able to get an attacker real useful data.

… I, on the other hand, have. That I have not been sued is testament to the fact that I wasn’t able to carry out these attacks.

Same for things like directory listing. You can do that to my web site. Is that a security problem? No, in fact, I turned it on specifically. If I didn’t want people to read it, I wouldn’t have put it on the friggin’ web server.

… I use the word “same” here in the sense of, “not the same at all”; directory listing vulnerabilities weren’t included in the Acunetix number, which was 50% SQL Injection and 43% cross-site scripting. I have also never found anything useful in a WEB-INF directory.

Is a web site that’s susceptible to an SQL injection attack hackable? Depends on where you get to inject the code […]

… on the planet where I come from…

I’m sure that someone who put their mind to it could take a web site like, say, slashdot, and inject some SQL. Then they might be able to … well, they could read all those posts that are on the web site. Except they wouldn’t be nicely formatted, but real men write HTML with vi anyway.

… yes, I really said this…

Maybe they could store or corrupt data with the injection, and maybe they couldn’t.

… it would, of course, depend almost entirely on whether the attacker could spell the word “UPDATE” or read the word “password”…

Maybe (and this is most likely) they could cause the script to blow up. Is that “hacking” a web site? Hell, I get script explosion errors from web sites WITHOUT hacking them.

Is being able to view a script a security vulnerability? it depends. It depends on the web site. The script. The webmaster’s intentions [, …]

… whether there were debugging methods hidden in the script, whether passwords or authentication information were stored in the script, whether the script took arguments that weren’t obvious from the HTML source code or the request traffic, whether it relied on vulnerable third-party components, whether I cared about my intellectual property, and a whole host of other concerns I didn’t think to write mostly because I only know these things in my fictionalized portrayal on this blog…

So the point is not that they’ve found a lot of theoretical issues, but whether they’ve actually found security issues. And the only way, in my mind, to see whether they have is to see if the issues can be exploited. If they can, I’ll pay up. If they can’t be exploited, then all they’ve done is made long lists of things that don’t matter from a security point of view […]

… and there is no set of circumstances in which I am paying these people $1,000.

20 Comments so far

  • RSnake

    February 14th, 2007 8:11 pm

    Hahah! Hilarious. Joel totally destroyed his own reputation as a security expert with his own two hands. Wow, just wow.

  • one.miguel

    February 14th, 2007 8:19 pm

    oh.my.god.

  • Ryan Russell

    February 14th, 2007 8:34 pm

    Yes, the IP address thing caught my eye, too. Anyone sitting behind a any-to-many NAT will beat you soundly if you try to make your cookies work like that.

  • Nate

    February 14th, 2007 9:42 pm

    Blog wars… BAD!
    Valentine’s Day… GOOD!

  • Marcin

    February 15th, 2007 2:07 am

    beautiful. I cannot wait to see how this all pans out. hehehe :D

  • dre

    February 15th, 2007 3:59 am

    Is it just me or was one.miguel FP on this?

    Then RSnake’s comment comes in later… with an obviously improper (or fake) date.

    Leaving me to wonder if either Tom is getting cute with his SQL skills -or- Tom (and the rest of us) are somewhat in awe of RSnake’s.

    Actually, no, he would have used a POST and manipulated a parameter, such as a date field, by forcing Tom himself to run an HTML iframe that silently slips past NoScript’s protections and turns Javascript on to inject super-secret zero-day XSS with a CSRF twist.

  • dre

    February 15th, 2007 4:08 am

    …and because Tom already had access to Joel Snyder’s weblife (via a similar submarine XSS he found in Acunetix’s website - using an 84 wrapped line bash shell and a few small C progrms in his little blackbag) RSnake followed him in and deleted Snyder’s GMail account, called to cancel all his credit cards, and pulled a little Zalewski magic that writes zeroes to his whole disk. And Joel doesn’t have backups.

  • dre

    February 15th, 2007 4:35 am

    Ryan: I was thinking the same thing about IP adress-based authentication. When you say “cookies”, don’t you instead mean, “since that doesn’t work out, most use HTTP referers” instead?

    Also - would you consider such authentication bypasses (1) to fall under your definiton of tranitive trust that you spoke about in your blog?

    1) Shostack recently wrote on his blog, EmergentChaos, that “There are three types of authentication:
    1. Something you’ve lost
    2. Something you’ve forgotten, and
    3. Something you used to be”.

    In the case of IP address-based authentication, it also means “Something that includes accessing one network from another network without user authentication”. A fairly ambiguous statement, and quite similar to the same-origin problems (and Telnet vulns) that we’ve seen lately.

    IP addess-based authentication, according to work submitted to OWASP by McGraw’s SecureSoftware - didn’t they get bought by Fortify by the way? - is listed on a list of vuls which also includes DNS-based authentication and “Using HTTP Referer fields/tags for authentication” in the same vulnerability category.

  • Ryan Russell

    February 15th, 2007 5:33 am

    When I say “cookies” in this case, I mean the strings that the web server hands your browser that your browser is then supposed to hand back to the matching site or domain on subsequent requests.

    Using a referer string to, say, only allow you to connect if you were referred from loggedin.example.com would be extra additional stupid, and allow for the netcat hacking.

    So what Tom is getting at is back in the day, if you connected to a site from 1.2.3.4 and logged in, you’d get back a cookie that was a function of username+1.2.3.4. So as long as you handed over that cookie, and connected from 1.2.3.4, you’d still get the secret pages. However, if you went home and connected from 5.6.7.8 and handed over the 1.2.3.4 cookie, the server would send you back to the login page to get a new cookie.

    Sounds smart, right? And it’s not a “security” problem, it fails closed.

    Problem is, there are LOTS of times when your IP changes between connections with a stateless protocol. DHCP. Roaming wireless. Going Home. NAT with address pool. NAT with route changes. NAT load balancing across interfaces with different IPs. Your host deciding that its other interface has a better route. And on…

    With the end result being that every time you login, it turns around and boots you back to the login page to login again.

    So if you write your cookie system to do then, then I add you to my list of people who have to die.

    (Cookie authentication takes something you’re going to forget and turns it into something you’re going to lose.)

    The end result being that the world has been forced to use portable cookies. That means that the Wall of Sheep guys really ought to also post the cookies of people who access websites in the clear. The cookie is as good as a password, at least for a window of time. Also why Amazon makes you provide your password over and over again, even though it clearly knows your username the entire time.

    The transitive trust thing I mentioned means that you trust my blog not to hack you; I trust technorati.com not to hack/deface my blog; therefore you trust technorati.com to not hack you. Technorati probably trusts someone else not to hack them, and it boils down to you trust the world not to hack you. In their favor (though I’m too lazy to look hard) I don’t see any Web 2.0 confetti on Matasano’s log.

    Did I get most of the dozen or so points you raised? I didn’t know there was going to be a security blog quiz…I didn’t do my OWASP homework…;)

  • Ryan Russell

    February 15th, 2007 5:33 am

    Dude, look how long of a comment you made me make…

  • PaulM

    February 15th, 2007 11:05 am

    As much as I was unimpressed with Acunetix’s original article, I am rooting for them now. I hope they are able to milk this mess for a lot of free press, and I hope Joel pays up. Maybe he could present them with a check at OWASP Milan in May?

    PS - So, Ryan, does that novella-sized comment make you the Blue Bore? ;-)

  • Ryan Russell

    February 15th, 2007 11:48 am

    I’m in my 25th year of using that handle. I’ve heard ‘em all.

  • ToddH

    February 15th, 2007 1:20 pm

    I have a lot of time for Joel Snyder, who is a long time security veteran and was fighing the good fight back when many of todays self-described pundits were still in high school. It’s clear why Snyder and IDG is frustrated with fluffy hype like the Acunetix ’survey’ that started this discussion. Thomas, from what you posted it seems that you don’t disagree on that point.

    However, if Matasano has a beef with the proposed methodology then perhaps you should document a better approach and make a name for yourselves at the same time. The discourse above may cut it on this blog. To my eyes it reads more like a spectator yelling from the sidelines than a subject matter expert with an interest in exposing some actual data. Are you part of the problem (hype), part of the solution (clearing the hype) or just a bystander?

    And remember - in a battle between the computing press and a vendor, it’s pretty easy to pick the loser.

  • Thomas Ptacek

    February 15th, 2007 1:33 pm

    There’s “right” and there’s “wrong”, “correct” and “incorrect”.

    It is “wrong” to suggest that the only way to draw a conclusion about the state of security in web application development is to hack into a random 10 sites and draw credit card numbers.

    It is “incorrect” to suggest that SQL Injection “often doesn’t matter”, because “you can’t always change information” and “it’s harder to read the data in the SQL dump than in the web app anyways”.

    It is “incorrect” to suggest that “most” good webapps are resistant to XSS attacks because “cookies are tied to IP addresses”.

    I’m not sure what you mean by this “hype” stuff, or “fighting the good fight”, or “who’s going to win, vendor or press”. “Bystander”? “Spectator”? What do you mean? I’m telling you that Joel Snyder and his editor at Network World are out in the weeds shouting crazy talk at the top of their lungs, and giving examples. You can tell me I’m wrong, and I’ll listen — but you haven’t done that.

    Waiting eagerly for clarification.

  • ToddH

    February 15th, 2007 2:13 pm

    I’m not disagreeing with your technical conclusions. What I’m saying is that your method of framing the discussion is somewhat ineffective outside the echo chamber that is the security blogosphere.

    If you believe Joel and Network World are wrong on the technical details, then spell it out in a detailed response and send it to Paul. Your response to my post is actually clearer than your original tongue in cheek blog entry.

    A real response to this would go back to the Acunetix survey, address each item in the discussion (with references for those who aren’t experts in the field) and then suggest an alternative methodology to achieve what IDG set out to do in the first place. That elevates the discussion so many people can benefit from your knowledge.

  • Thomas Ptacek

    February 15th, 2007 2:56 pm

    You’re implying that you understand my technical conclusions (you’re firmly inside the “echo chamber” right now). This wouldn’t surprise me; they aren’t difficult or obscure conclusions to draw. With that said, what’s your take on the conclusion I’m drawing about Joel’s qualifications?

  • Andrew Jaquith

    February 15th, 2007 3:50 pm

    Two things: first, your reply to Joel is one of the funnier things I’ve read in a while. I loved the “I have also never found anything useful in a WEB-INF directory” quip.

    Acunetix gets points for adroitly re-directing scrutiny towards Paul’s employer, neatly skirting the messy issue of hammering out contracts with lots of anonymous (and not so anonymous) website owners.

  • Chris E

    February 15th, 2007 8:19 pm

    ToddH, just because Joel may be a “long time security veteran” doesn’t give him immediate credibility in web application security topics. Regardless of how scientific (or not) the Acunetix survey was, Joel’s comments on Slashdot read like someone who has seen the OWASP Top 10 list but doesn’t really understand how they work.

    Oh, and just in case you missed it, Matasano guys, documenting a methodology to settle this silly dispute might help you “make a name for yourselves!” Don’t miss this golden opportunity! :P

  • dre

    February 16th, 2007 10:17 pm

    Ryan, you said, “I don’t see any Web 2.0 confetti on Matasano’s log. Did I get most of the dozen or so points you raised?”

    First, I wanted to thank you for your post and dealing with my overly inquisitive nature.

    Secondly, yes, you did answer my questions - but I was looking more for an interpretive analysis and yours was more predictable and pedantic than I usually prefer. It was, however, a most interesting read - way above par for what I expected.

    Lastly, the reason you don’t see Web 2.0 confetti on the matasano blog is because Tom is a security freak. He rewrote WordPress from scratch, then tweaked Akismet and mod-security so much that the creators of said software would immediately decide to patch-in upon seeing his changes/rulesets. At leas, that’s my uneducated guess.

    @ Tom: you wouldn’t happen to want to share your WordPress, Akismet, and mod-security configs, especially if they are truly out of the ordinary? Say, in a blog post about them?

  • Ryan Russell

    February 17th, 2007 1:43 am

    dre: Well, you appeared to be asking some fairly basic questions, it’s difficult to know ahead of time what level of abstraction to hand you back.

    • Leave a reply