Pedram Amini’s Amazing PaiMei: Win32 Reverse Engineering in Python

Thomas Ptacek | August 10th, 2006 | Filed Under: Defenses, Reversing

0

I spent the day investigating a secret protocol used by a popular enterprise management application, using Pedram Amini’s PaiMei tools. I can now safely report that PaiMei is going to change your life. In particular, if you have spent more than 5 minutes in IDA Pro, windbg, or OllyDbg in the last few months, you have to install this tool.

1

PaiMei (“white eyebrow”, head priest of White Lotus Clan from Kill Bill 2) is a “framework” for building reverse engineering tools. At its core is pydbg, a debugging engine, using Python’s ctype FFI and the native Windows debug libraries. As pycap/libpcap is to packets, pydbg is to Win32 debugging: it enables you to write debuggers in Python while hiding most of the Win32 trivia from you.

This is not a new idea. I first heard of it in 1997, with Dave Hanson and Jeff Korn’s “Deet”, which essentially implemented a cross-platform debugger in Tcl/Tk (with a small C kernel). Deet and pydbg are scriptable debuggers, which allow you to debug by writing small programs to play out expiriments. I’ve written about my frail efforts to do this using gdb macros. Gdb is venerable. Pydbg is the future.

2

PaiMei would be cool enough if it was just a Python debugging API. But it’s more than that.

Pedram and his team have written GUI-driven “modules” on top of PaiMei. There are a couple, and more in the works, but the one that has my attention today is PAIMEI pstalker (“pstalker” from here on).

Pstalker is the PaiMei reimplementation of “process stalking”. Process stalking is a weaponized variant of “hit tracing”, a technique for profiling control flow in a program under a debugger. The idea is simple: you identify all the functions (or bblocks) in a program and set a breakpoint for each. Setting breakpoints is cheap; handling them is expensive. So a hit tracer records the first time a breakpoint is hit, and then removes the breakpoint. You’re left with a trace that tells you which functions, under a given scenario, are “touched” by program execution.

3

Process stalking, and pstalker in particular, takes this a few steps farther.

First, pstalker uses IDA Pro to discover functions and bblocks, so it benefits from all of IDA’s analysis capabilities. The PaiMei console imports “PIDA” files generated by a plugin to IDA that Pedram distributes and uses them to set breakpoints for hit traces.

Second, pstalker records processor state at each breakpoint, including register contents and memory dumps. This gives you a better shot at reconstructing what was actually happening at any given point in the trace.

Third, pstalker saves hit traces to an SQL database (MySQL currently, but Pedram, please support sqlite; the standalone database is a huge pain). This is nice in that it gets rid of the million little files Pedram’s old process stalking tool used, but, more importantly, it allows for hit trace diffs.

4

The results of one trace can be used to filter subsequent traces. This is huge (in fairness: it’s something that other people, notably Halvar [I believe], have been working on). Did I mention that this is huge? Because this is huge. Here’s why:

  1. Take a complicated Windows server. Use IDA to analyze all its dependent DLLs. Generate PIDA files for them and load them into PaiMei. target selection screen

  2. Using PaiMei’s “tag” feature, create a bucket for an initial hit trace. Call it “background-noise”. Click “use tag for stalking”

  3. Attach to the server, while the network is running and no clients are connecting. Go get a cup of coffee while the server traces. Stop the trace, thus saving it to a database. process selection screen

  4. Create a new tag. Turn stalking on for it. Take “background noise” and turn “use tag as filter” on. Attach to the server again, but this time, connect to it. Stop the trace. You’ll have a (much) smaller set of hits to look at, and they isolate the code that handles connections. data display screen

  5. Repeat the process (you can use multiple tags to filter). Single step a client, byte-by-byte or word-by-word (perhaps using PDB). Throw random crap at it. Vary a header, byte by byte. Each run, filtered by the output of the previous runs, isolates the specific code used by the server to handle the events you generated.

What have we accomplished? Well, IDA Pro and 30 DLLs probably gives us somewhere in the vicinity of 30,000 functions to look at. For a variety of cases, pstalker reduces that to around, say, TEN.

How am I going to reverse anything ever again without this?

5

The PaiMei code is also really, really clean. It’s Python, and even though Python is the language of choice for over-the-hill hackers, it’s much better than Perl. Dug Song made me learn it at Arbor. I had no trouble reading Pedram’s code, and the core library code is well documented.

By way of example, I wanted timestamps on all my recorded hits. This would allow me to create one bucket for all the function hits across several runs; by waiting a few seconds between each run (where each run, say, varies a byte of the header in my protocol to try to uncover what it does), I can tell which hits belong to which runs, without stopping and starting PaiMei for each run.

PaiMei already tracked timestamps (it has a “code coverage” class to handle the bookkeeping details of a hit trace); they just weren’t exposed in the UI. It took me all of 10 minutes to fix that.

7

I do have a wish list.

  1. Can I have stack traces for each hit? I know they’re somewhat redundant, but I can graph them to visualize control flow (in particular, to identify event and “parse” loops).

  2. Symbols. Pedram acknowledges this in his presentation. It didn’t slow me down much not to have them, but it feels weird.

  3. I need to be able to click on a hit and see the assembly for it (if there’s a way to click on something and have it pop up in IDA, so much the better).

  4. Yeah, I need this for non-Windows targets. Remote debugging is apparently coming, which will help. I don’t imagine Pedram’s working on SPARC support (X86 and Win32 has eaten its way pretty thoroughly through the code). Also,

  5. A pony. Please?

8

There is a metric crap-ton of potential in scriptable debuggers. For example, systems that use bizarre crypto that I’m having a hard time cracking (or for that matter, ephemeral SSL I can’t MITM); the program that intercepts cleartext read/writes and logs them is a script away.

I’m pretty arrogant, and I usually look at other people’s tools and frameworks and force myself to write my own. I’m going to have a hard time justifying that with PaiMei.

PaiMei seems like a huge accomplishment for Pedram, cooler than 95% of what I saw at Black Hat this year. Congratulations, and thank you.

Viewing 6 Comments

    • ^
    • v
    A plan9 reference from newsham? I know its cliche, been done, old hat, but its relevant. Really. I swear!

    http://cm.bell-labs.com/sys/doc/acidpaper.html

    Mixing scripting languages and debuggers can be very powerful, especially if you get the abstractions right.
    • ^
    • v
    Getting the abstractions right is a lot easier when once you've mapped the low-level native debugger implementation into a reasonable language, which Python almost is.

    Acid is cool, though.

    When are we getting our Haskell debugger library?
    • ^
    • v
    Getting off topic here, but if you're writing functional code, do you really need a debugger? You'd just write some rules saying how things should behave and have it automatically verified a la quickcheck: http://www.cs.chalmers.se/~rjmh/QuickCheck/

    Oh, you meant a debugger framework written in haskell, didnt you!? ;-) Yah, I'd stick to python for something like that. No need for strong typing and lazy evaluation there, really...
    • ^
    • v
    Wow. I am both flattered by the compliments and appreciative of the detailed post. Thank you on both counts. I like the relative time column you added, send me the patch and I'll check it into SVN ;-)

    With regards to the wish list:

    0.
    The PIDA file format is obtuse. I'm moving that to SQL and when I do I'll look into adding SQLLite support across the board. The default will still be MySQL. The nice thing about the central server is that the whole team's work is accessible from a single location.

    1.
    We can hack this together pretty quick. Many of my structures have a .ext member for filling in whatever you want. I can add that to the code coverage class to generically support whatever additions people want. The process stalker functionality is a class on it's own, so you should feasibly be able to write a small command line script with the added modification.

    2.
    Luis Miras is working on a Python symbol parser. I will definitely be adding that. At least for function names however, you can load the symbols in IDA pre-PIDA generation. But I hear you on the pain ;-)

    3.
    Have you tried the uDraw sync yet? uDraw rocks. Install it, launch it in '-server' mode, then connect PaiMei to it through the connections menu. Next, synchronize your loaded tag and each click in the hit window pulls up the relevant portion of the graph in uDraw. It also works in the other direction, click on a node in uDraw and the closest hit will be highlighted (with the register inspection data) in the hit list. You can also export the hit data into IDA, which I don't find as useful.

    4.
    Yeah it's entirely 32bit Windows right now. There is someone who is writing a Linux PyDbg, when that is done I will rename PyDbg to PyDbg-win32 and write an abstraction class on top of both. Hopefully as the entire framework matures more people will be contributing. Of all the open source software I've written, this one has gotten the most community feedback and support, so it's not out of the question.

    5.
    http://www.moondanceranch.com/images/pics/pony-...

    I use this framework all the time and have every intention of vigorously maintaining it. So all feedback is greatly appreciated and will not be ignored. On a final note, thanks again for the kind words. I caught the PDB talk at BlackHat and it's definitely an awesome tool.

    Cheers

    -pedram
    • ^
    • v
    OMG PONIES*&#(!!
    • ^
    • v
    For what it's worth:

    On "recording stack traces", I just added it, sort of (it "works" to a definition of "works" that includes "poorly tested, and some of the data looks bad").

    I'll mail you the diff. Why am I writing this here? Because it took me less than 30 minutes to "add" this functionality (caveat: adding it so it works might take longer), and I've barely read Pedram's code.

    PaiMei rocks.

Trackbacks

close Reblog this comment
Powered by Disqus · Learn more
blog comments powered by Disqus