The path to PenTestConsole

Mike Tracy | March 27th, 2008 | Filed Under: Uncategorized

I recently gave a talk at OWASP Chicago called “Web app penetration testing with scripting languages”. Based on feedback since the talk (and dancing between the raindrops of a pretty hectic work schedule), I decided to try a series of blog posts cementing the ideas surrounding the development of PenTestConsole. To Mike, Mike, Al, Walter and the other 5 people I can’t remember right now, your tarball will be in the mail soon… I hope.

What is PenTestConsole?

In simplest terms, PenTestConsole is a set of Ruby libraries (and add-ons to existing libraries) that provide a scripting framework for web application testing. It allows you to work from an irb (Ruby shell) prompt or write scripts to automate tasks that would otherwise consume precious engagement time.

And Now… Some Context

Coming into this business from a QA background, I realize that testing is testing. In the QA world, positive testing is simply testing to show the product works (test to pass). Negative testing shows the product is broken (test to fail). One of my big lessons learned over the years is that you can’t positively test something unless you know what it’s supposed to do. Proving a negative is easy… it’s proving a positive that’s the hard part. These concepts hold equally true for application penetration testing and even though line between “positive” and “negative” blurs to a disturbing gray, the necessary piece of the puzzle is business context.

The problems that plague QA guys are a subset of the problems that hinder application pen testers. Not enough information, too little time and getting started in the wrong part of the development cycle just to name a few. The problems we face as security consultants are compounded by the breadth of expertise required to work any engagement that comes our way. As much as I would love to spend time pontificating about methodology, technology and software development practices (Niaaaagra Falls), I have work to do and that’s the reason for PenTestConsole.

By the time I get to an application it’s either deep into QA cycles or actually in production and I’m testing the production instance in an off-hours gig where I’m hamstrung by limits on what data I can enter into the system or how destructive I can be on a shift schedule that is grounds for divorce in 32 states (Slooooowly I turn). The key thing is that I am looking at an application that fills some business gap for my customer. Nirvana comes when I can use business justification to inform my assessment of security posture. I need to be able to find all the bread and butter issues (XSS check, session problems check, SQL|XML|XPath|Command|… injection check, CSRF check, error handling check…) and find the more complex and subtle (read fun) vulnerabilities that the customer needs to know about.

There are lots of tools out there to help me ranging from from proxies to fuzzers to scanners. Every one I have used has solved some particular problem during an engagement. My big challenge is that the more advanced or subtle functionality is not as accessible or obvious as the basic stuff. I need to spend cycles figuring out how the application under test works rather than ferreting out how my tool works. Even when I do figure it out, I keep running into situations where I need to code something specific because the app uses some new authentication mechanism or postbacks or includes HaveAFunTimeReadingMe.asp or you_need_to_be_on_meth_to_read_this.js (Step by step). Doing this over and over made me decide to write a framework that makes this job faster and less frustrating.

Coming next are the whys and hows of PenTestConsole’s development as well as some examples of how it works. Stay tuned.

Viewing 25 Comments

    • ^
    • v
    Hi Mike,
    Is your pentesting framework going to be available for general consumption? I'm extremely interested in a framework like this, as I prefer Ruby over all of the other scripting languages for my security work.

    That being said, I'm looking forward to future posts about the framework :)
    • ^
    • v
    I would really like to give you feedback on your presentation and/or source code if you want to shoot them over to me. Additional comments below.

    as much as I would love to spend time pontificating about methodology, technology and software development practices (Niaaaagra Falls), I have work to do and that’s the reason for PenTestConsole

    I highly recommend the book, Automated Defect Prevention. Especially if you like pretty graphs and Fishbone diagrams. This book also surprisingly mentions OWASP...

    I need to be able to find all the bread and butter issues (XSS check ...)

    It's usually not possible to find all of the XSS without using white box methods (i.e. unit testing or code inspection).
    • ^
    • v
    Ferreting out how your tool works?

    Hasn't Internet Security Subverter already been written?
    • ^
    • v
    dre and lee: I am currently in the process of cleaning things up a bit then sending it out to interested parties. It's still deeply in development so ymmv.

    dre: I think you might be taking my use of the word 'all' a bit to literally. Whitebox testing is a different animal entirely. I don't get to see source code on the vast majority of engagements I work on. PenTestConsole is designed to help with black and really dark shades of graybox testing.
    • ^
    • v
    chris: I need to learn better how to disambiguate. I happen to be wearing one of their t-shirts right this second.
    • ^
    • v
    mike: i think your readers might take the word, "all" a bit too literally, hence my clarification.
    • ^
    • v
    Mike: In that case, I'd like to sign up as an "interested party" if possible :)
    • ^
    • v
    Hi Mike!,

    I believe what you are referring to as a 'PentestConsole' is metasploit or CORE IMPACT (disclaimer: I work for Core :)). That's why they are called 'pentesting frameworks'. So your idea is good, but it has been done already. Of course, everything needs improvement, but something is already there.

    Both have an scripting language you can use to manipulate existing modules, combining functionality, both have different tools you can use during your pentest, etc.

    For example, CORE IMPACT is not only about exploits, you have other tools, like the SMB/RPC library, modules to inject code, you can install an agent and run a python console and run your scripts on the remote machine to do whatever you want, etc.

    So I think there's no need to 'create' a new framework (well, define 'need' :), I know), I think the best thing to do is for anyone to write modules for metasploit (because it is a free product, for now) or for CORE IMPACT if you are willing to pay for it. That's what we do.

    Of course, creating a new framework might be totally worth it, I'm just saying that the idea of a 'pentestconsole' is already here.

    Bye!
    • ^
    • v
    Hernan: If I understand his intentions correctly from the post, it seems to me that 'PenTestConsole' is a Ruby framework specifically for web app testing -- not building exploits in general.

    I would think building web app testing tools on top of a framework more meant for memory corruption and similar attacks would be cumbersome and overkill. Perhaps the name 'PenTestConsole' is a misnomer, but I think the idea might be more original than you think. Then again, I have not seen/used the product, and have almost no experience with CORE (Metasploit on the other hand...)
    • ^
    • v
    If this thread is collecting "interested party" email addresses, count me in!
    • ^
    • v
    Hi Vitaly!,

    The name 'pentestconsole' is kind of bad,I think we both agree on that. So I recommend to change the name because I think it kills any innovation the name was trying to suggest :). I apologize also because I originally commented on this post because it was very strange to me to see the idea of a 'pentestconsole' as something new, my bad.


    The idea of putting together a bunch of different tools to create a 'framework' or a 'toolkit' so that you can go to the same 'app' to do your testing, can't be considered original, come on! :).

    Then, specifically, creating a 'framework' to do 'testing' (web app or whatever app) using a 'scripting language' is NOT original, come on part 2! :).

    Saying that because of the little time you have to do pentests/'any other thing' having things 'automated' or having scripts, or having frameworks to help you is helpful is also nothing new, come on part 3.

    CORE IMPACT already allows you to do that (with the console and everything), metasploit too, and i'm sure Canvas does the same thing.
    You can open a python/irb console whenever you want, and do your stuff..

    if you use paros, burp, etc, you can script stuff/create plugins, etc.

    For a more 'humble' project, you can check out my 'uhooker' and 'proxy_hooker' (http://oss.coresecurity.com/projects/uhooker.ht...),
    you can hook 'stuff' and script everything from python.

    CORE IMPACT in its latest version is beginning to add web app testing, and since everything is backed by python, almost any functionality can be scripted.. so there you go.. the same goes for almost any other pentesting framework like canvas, metasploit, etc.

    So, I think there's no big innovation here in principle, anyways having said that, I think that after changing the name :) this is a very good project because it could facilitate what you can already do with burp, paros and other tools, add new very useful features,and it could clearly do a better job at helping you do webapps pentests than canvas,impact,metasploit,burp,paros and other tools are doing now.

    SO, to make my point clear, I LIKE the project :),
    and I'd like to see it, i'm just saying that I think some of the ideas proposed are not an innovation, but still can be VERY useful.

    Thanks!,
    bye!


    Bye!
    • ^
    • v
    Vitaly: Thanks for the support and I think you might get where I'm
    coming from... stay tuned!

    Hernan: I agree "PenTestConsole" is a horrific name.
    WebAppPenTestConsole! No. Not really. I really need a name.

    Things will hopefully become clearer as I get through the series. The
    idea behind this that might be considered original (or at least
    innovative) is having a command line interface (shell) to perform common webapp pentesting tasks as well as a scripting framework for automating them.

    My purpose isn't really comparable to metasploit et al. What I'm working
    on is useful to me. It's an interesting learning experience to write
    and my experiences (and the code) might be helpful to others trying to solve a similar problem.
    • ^
    • v
    Hi Mike!!,

    Check out selenium (http://selenium.openqa.org/), among others. scriptable from python and other languages (java, .net, etc). is not strictly for security testing though.

    Anyways, I'm not convinced yet, not that it matters :) and you haven't presented the tool yet, so I'll shut up and wait. :)


    Thanks, bye!!!
    • ^
    • v
    My suggestion for a name for the console was WWMD? (What would Mike do?). Obviously, what Mike would do is not listen to me. :(
    • ^
    • v
    @ Hernan:

    If you like Selenium, then you'll probably like Canoo WebTest more, which is also not strictly for security testing. There is another project called WebDriver which has come a long way recently that is also worth a look as an alternative to Selenium.

    I also suggest a test case submission framework such as FitNesse (with HtmlFixture). These can be used during the requirements phase to build the test cases (while xmlbasedsrs can be used to build ab|use case scenarios, diagrams, et al). If you don't have a lot of experience with test cases, I suggest building at least a Test Case Outline (TCO) for use in scripted testing that can be combined with exploratory testing (which builds a test charter).

    There are also additional ways to test or inspect code inside of an IDE or before integration (and not just unit testing) that I tend to talk about a lot. The problem with tools such as Core Impact and PenTestConsole is that they are built to test too late in the lifecycle. Although PenTestConsole appears to be lightweight enough (in a similar way as UTScapy) to be used as a functional testing tool (either before, during, of after integration).

    Core Impact is more focused on network penetration-testing, while PenTestConsole is focused on software penetration-testing. In my mind, Core Impact would also be a poor choice for acceptance testing, especially at it's lack of proper fit and enormously expensive cost.
    • ^
    • v
    Andre: "There are also additional ways to test or inspect code inside of an IDE or before integration (and not just unit testing) that I tend to talk about a lot. The problem with tools such as Core Impact and PenTestConsole is that they are built to test too late in the lifecycle."

    That is certainly true, but I think context is important here. Many pen-test customers will not use static analysis or whatever else to analyze the code they put out on their sites. A pen-tester's job is to test 'late in the lifecycle' (i.e black box testing) if a real attack is to be simulated -- and this is where frameworks like these become useful.
    • ^
    • v
    What ever happened to the old Matasano of a few months ago, when people posted interesting things worth reading, and not marketing propaganda?
    • ^
    • v
    Marketing propaganda? That implies we'd actually waste our time selling something like this which is obviously not the case.
    • ^
    • v
    sigsegv: I'm offended not that you think we'd use the blog for marketing, but that you think that if we wanted to market something, it'd be PenTestConsole. =)
    • ^
    • v
    Mike: I really like the idea working from within irb and being able to automate certain webapp pentesting tasks. I have some ideas of things I would like to see/contribute. Please put me on your tarball list.
    • ^
    • v
    Mike: No, marketing does not always need to imply that one is selling something. It can be self-promotion of a product, which is what this is. Yes, I *do* think it's cool, but I'd rather read about the features in either a release log, or a post with a link to a tarball at the bottom so I can read the code afterwards. :)

    Thomas: I was merely voicing my opinion that I feel your technical posts are more interesting than this type of thing. What ever happened to your promise for "this old vulnerability: SSH CRC compensator" along with several others? ;)
    • ^
    • v
    No april fools day post ????? were's your sense of humor ??? not k00l
    • ^
    • v
    Mike,

    I'm also interested in the development of your web app testing console (WACT?). We're spending a great deal of time with similar problems (and generally resorting to one-offs and already-written tools). A set of libraries for things like brute forcing urls / forms, fuzzing, injection, etc. would be extremely useful.

    Can you keep me in the loop too?
    • ^
    • v
    Is this meant to be similar to w3af http://w3af.sourceforge.net/?
    I haven't used it yet and I prefer ruby over python, but seems to be very similar to what you are proposing.
    • ^
    • v
    Nice blog again.. I really enjoy reading your blog.. Very useful.. hope you will post another useful ideas..

Trackbacks

close Reblog this comment
Powered by Disqus · Learn more
blog comments powered by Disqus