Random Thoughts On OWASP

Dave G. | July 6th, 2007 | Filed Under: Industry Punditry

I was reading Mark Curphey’s blog, when I came across a post on OWASP’s spring update. I happened to be at New York’s OWASP meeting and saw Jeff deliver the slides. The slide that was the most contentious is the one that Mark also noticed:

Picture 7.jpg

The implication is that if you run every security tool against your codebase, you will at best cover 45% of the known classes of vulnerabilities. This slide is based on a slidedeck provided by the CWE folks. It seems dubious to just talk about numbers without talking about what the remaining 55% is.

Don’t get me wrong, as a company that does services, a statistic that says that tools can’t replace us should be good for business. But the truth is, you probably don’t need to care about 100% of the attack classes out there. And i can’t think of an application that has to worry about every attack class. Attack classes are limited by Operating System, Application Language and Environment, and Implementation Details.

I am kind of shocked that I haven’t seen the code analysis space respond to this more publicly, or that these statistics remained buried in slidedecks and not marketing material.

Anyways, believe it or not, not even what I wanted to speak about. The slide that interested me was:

Picture 6.jpg

10,000 people on mailing lists, ~100 memberships (I combined corp and individual) seems like a bad ratio. Also, I think more than 28 organizations use OWASP resources regularly (currently, we aren’t one of them!). While it is amazine that they do so much without any full time personnel, I think the lack of employees really hurts OWASP. If I ran things (Oh how easy it is to be a backseat driver, especially on a blog!), I think I would raise the employee count to one. That person’s role would exclusively be membership drives focused on corporate memberships. I would also restructure pricing (because i am not just a backseat driver, but also an armchair quarterback), and increase prices for for larger companies. The whole pricing model seems really wack (that is in MBA terminology). Some examples:

  • A pre-funded startup product company would pay as much as the world’s largest software company would. This is not only disadvantageous to the little guy, but OWASP is leaving money on the table that can help them grow.

  • A startup product company would more than a Final Four consulting company would.

  • As it stands right now, a 10 person consulting company pays more than a Fortune 10 company does. In case you are wondering, a Fortune 10 company has more than 10 security personnel, and their total security budget is going to be slightly larger than a 10 person security consultancy.

What would I do after I made these changes? I would build OWASPWORLD, the world’s first application security theme park. Of course, the rides would only work 45% of the time.

Have a good weekend, Interweb.

Disclaimer: Neither Matasano nor myself are paying members of OWASP. While you could call me self-serving or hypocritical in the comments, I would really prefer to see a list of application security themed rides.

Viewing 10 Comments

    • ^
    • v
    Well, the first and most obvious is the Buffer Overflow waterfall ride ...
    • ^
    • v
    You know that OWASP is going to make you pay them to become a member or remove this OWASP-related content from your website. You're making money from their good name!

    you probably don’t need to care about 100% of the attack classes out there

    I was thinking that very important logic flaws probably involve the other 55%. However, this is speculation based on my real world experience ; no different than their speculation. Or yours.

    Could it be that the numbers Jeff Williams referred to were coming from bh-eu-07-chess-kureha-ppt-apr19.pdf ?

    I'll save any additional humor for the end of the thread, as I'm hoping this is just the beginning.
    • ^
    • v
    I'm glad you and Mark noticed anyway! Glad to see some discussion.

    Totally agree that for a given application, not all vulnerability types will apply. It's getting the right 45% that's hard. Do you think the tools are finding the 45% that's most relevant to your application? Or the ones that just happen to be easier for them to find? Anyway, I thought that MITRE's study was well done and the results were surprising.

    I really appreciate the thoughts about OWASP structure, and encourage you to get involved. I'd like everyone to know that membership is completely optional (think public radio) and all the money goes directly to support OWASP projects.

    I agree that OWASP should have a director focused on fundraising - a topic I've spoken about many times. But once we finally got some money, we decided to plough it back into research grants instead. So far we've awarded something like $150,000 in application security research grants and have already started seeing the benefit. I'm sure we'll get a director someday. We did fund an intern for the summer who's already doing great work!

    The membership categories levels are set where they are because we decided to target a small number of large organizations who use OWASP materials. Also to try to minimize the likelihood of OWASP-abuse by product and service vendors. Sure there are hundreds of non-member organizations who use our stuff, and that's fine. This approach doesn't require a lot of effort on our (volunteer) part and has been fairly successful. Personally, I like the fact that we're not out soliciting money all the time. We've got enough to keep doing interesting stuff with some great people.
    • ^
    • v
    Dave

    Granted we don't have fulltime employees, but we do have a core group of leaders, me included, who have been involved since the start and act the same as employees.

    How do you see employees making OWASP better? So far it's community driven, we also have people who spend a large chunk of time on it, working on project management and also pushing the foundation.

    What is the difference between these people and an employee?
    • ^
    • v
    First of all ,thanks for the post ,really interesting .I agree with you OWASP SUCKS ,they need at least One employee .Otherwise it*s clear they sucks.

    Ending :I would really prefer to see a list of application security themed rides.

    Thanks and ByeBye.
    • ^
    • v
    @Jeff:

    Re: Statistics: Without knowing more about the specifics of what's being detected and what isn't its hard to tell (I haven't seen any of the details).

    Re: Pricing: I am sure you are way closer to the pricing sensitivities around OWASP memberships, but from the outside it seems like like it punishes the smaller company. It makes sense that you dont want to have OWASP's brand abused.

    I don't actually know what OWASPs goals are or how much money it would need to accomplish them. But I did notice that your last slide asked people to become members, so I assumed it was important :)

    @Daniel:

    If I had to hazard a guess, I would say that you guys like application security, not growing OWASPs membership. I think having someone's time 100% dedicated to OWASP and specifically to membership drive would help you out tremendously. Having someone who is focused around that, let's you guys do what you do best. Jeff's example of public radio is a great one. There are people that volunteer and people that work at NPR.

    ---

    Just to be clear, the goal of the post was not to criticize OWASP, just some armchair quarterbacking at the end of a workday. The only exception is where I was actively criticizing is the pricing structure.
    • ^
    • v
    Dave,

    I'm not an MBA either, but I often times run into them in the wild so pardon me if I have a small correction. I checked several MBA blogs and the correct spelling of the term is 'WHACK', not 'wack'. I guess we missed out not finishing out our MBA.
    • ^
    • v
    MBA's don't blog; they're too busy trying to make money 358 days out of the year so that they can take their wife and kids to ride "Metacharacter Injection Mountain" during the summer on a nice weekend like this one.

    I personally prefer Lego World, mostly because of the built-in stack protection.
    • ^
    • v
    I was at the NY OWASP meeting and I brought this up during my presentation when I talked about classes of things you can look for with binary analysis.

    The exact classes covered by the tool are what is important. A percentage of CWE is meaningless since that weighs all vulnerability classes equally when they are surely not. You need to look at the distribution of the vulnerabilities that are actually looked for and found in the wild.

    CWE publishes this data using the CVE data:
    http://cwe.mitre.org/documents/vuln-trends/inde...

    1282 of XSS or 18.5% of all vulnerabilities found and publicly disclosed in 2006. This means they are prevalent AND people are able to find them.

    One (1) instance of of type-check vulnerabilities were found and disclosed in 2006.

    Both of these are weighted equally in the percentage pie chart where one is clearly at least 100 times more important to find.

    By my calculations the top 25 of most prevalent vulnerability classes account for 70% of reported vulnerabilities.

    I would love a tool that found all of these. It would be finding less than 5% of all vulnerability classes yet it would be finding all the important vulnerabilities.

    The truth is many of these can't be found with an automated tool. But the point is all vulnerability classes are not created equal. Since the pie chart treats them all as equal it is meaningless.

    -Chris
    • ^
    • v
    OWASP has now hired its first employee by the way, Alison McNamee. Alison's job (among many other things) is going to be to help boost OWASP's membership, both corporate and individual. She's also going to support the members, work on the conferences, and many other things. I agree that getting larger corporations to chip in more is appropriate for OWASP and we are working on that. We are also thinking about getting a full time director, but it will probably take a while longer before we have the funding to do so.

Trackbacks

close Reblog this comment
Powered by Disqus · Learn more
blog comments powered by Disqus