Pedram Amini’s Amazing PaiMei: Win32 Reverse Engineering in Python

Thomas Ptacek | August 10th, 2006 | Filed Under: Defenses, Reversing

0

I spent the day investigating a secret protocol used by a popular enterprise management application, using Pedram Amini’s PaiMei tools. I can now safely report that PaiMei is going to change your life. In particular, if you have spent more than 5 minutes in IDA Pro, windbg, or OllyDbg in the last few months, you have to install this tool.

1

PaiMei (“white eyebrow”, head priest of White Lotus Clan from Kill Bill 2) is a “framework” for building reverse engineering tools. At its core is pydbg, a debugging engine, using Python’s ctype FFI and the native Windows debug libraries. As pycap/libpcap is to packets, pydbg is to Win32 debugging: it enables you to write debuggers in Python while hiding most of the Win32 trivia from you.

This is not a new idea. I first heard of it in 1997, with Dave Hanson and Jeff Korn’s “Deet”, which essentially implemented a cross-platform debugger in Tcl/Tk (with a small C kernel). Deet and pydbg are scriptable debuggers, which allow you to debug by writing small programs to play out expiriments. I’ve written about my frail efforts to do this using gdb macros. Gdb is venerable. Pydbg is the future.

2

PaiMei would be cool enough if it was just a Python debugging API. But it’s more than that.

Pedram and his team have written GUI-driven “modules” on top of PaiMei. There are a couple, and more in the works, but the one that has my attention today is PAIMEI pstalker (“pstalker” from here on).

Pstalker is the PaiMei reimplementation of “process stalking”. Process stalking is a weaponized variant of “hit tracing”, a technique for profiling control flow in a program under a debugger. The idea is simple: you identify all the functions (or bblocks) in a program and set a breakpoint for each. Setting breakpoints is cheap; handling them is expensive. So a hit tracer records the first time a breakpoint is hit, and then removes the breakpoint. You’re left with a trace that tells you which functions, under a given scenario, are “touched” by program execution.

3

Process stalking, and pstalker in particular, takes this a few steps farther.

First, pstalker uses IDA Pro to discover functions and bblocks, so it benefits from all of IDA’s analysis capabilities. The PaiMei console imports “PIDA” files generated by a plugin to IDA that Pedram distributes and uses them to set breakpoints for hit traces.

Second, pstalker records processor state at each breakpoint, including register contents and memory dumps. This gives you a better shot at reconstructing what was actually happening at any given point in the trace.

Third, pstalker saves hit traces to an SQL database (MySQL currently, but Pedram, please support sqlite; the standalone database is a huge pain). This is nice in that it gets rid of the million little files Pedram’s old process stalking tool used, but, more importantly, it allows for hit trace diffs.

4

The results of one trace can be used to filter subsequent traces. This is huge (in fairness: it’s something that other people, notably Halvar [I believe], have been working on). Did I mention that this is huge? Because this is huge. Here’s why:

1. Take a complicated Windows server. Use IDA to analyze all its dependent DLLs. Generate PIDA files for them and load them into PaiMei.

2. Using PaiMei’s “tag” feature, create a bucket for an initial hit trace. Call it “background-noise”. Click “use tag for stalking”

3. Attach to the server, while the network is running and no clients are connecting. Go get a cup of coffee while the server traces. Stop the trace, thus saving it to a database.

4. Create a new tag. Turn stalking on for it. Take “background noise” and turn “use tag as filter” on. Attach to the server again, but this time, connect to it. Stop the trace. You’ll have a (much) smaller set of hits to look at, and they isolate the code that handles connections.

5. Repeat the process (you can use multiple tags to filter). Single step a client, byte-by-byte or word-by-word (perhaps using PDB). Throw random crap at it. Vary a header, byte by byte. Each run, filtered by the output of the previous runs, isolates the specific code used by the server to handle the events you generated.

What have we accomplished? Well, IDA Pro and 30 DLLs probably gives us somewhere in the vicinity of 30,000 functions to look at. For a variety of cases, pstalker reduces that to around, say, TEN.

How am I going to reverse anything ever again without this?

5

The PaiMei code is also really, really clean. It’s Python, and even though Python is the language of choice for over-the-hill hackers, it’s much better than Perl. Dug Song made me learn it at Arbor. I had no trouble reading Pedram’s code, and the core library code is well documented.

By way of example, I wanted timestamps on all my recorded hits. This would allow me to create one bucket for all the function hits across several runs; by waiting a few seconds between each run (where each run, say, varies a byte of the header in my protocol to try to uncover what it does), I can tell which hits belong to which runs, without stopping and starting PaiMei for each run.

PaiMei already tracked timestamps (it has a “code coverage” class to handle the bookkeeping details of a hit trace); they just weren’t exposed in the UI. It took me all of 10 minutes to fix that.

7

I do have a wish list.

1. Can I have stack traces for each hit? I know they’re somewhat redundant, but I can graph them to visualize control flow (in particular, to identify event and “parse” loops).

2. Symbols. Pedram acknowledges this in his presentation. It didn’t slow me down much not to have them, but it feels weird.

3. I need to be able to click on a hit and see the assembly for it (if there’s a way to click on something and have it pop up in IDA, so much the better).

4. Yeah, I need this for non-Windows targets. Remote debugging is apparently coming, which will help. I don’t imagine Pedram’s working on SPARC support (X86 and Win32 has eaten its way pretty thoroughly through the code). Also,

5. A pony. Please?

8

There is a metric crap-ton of potential in scriptable debuggers. For example, systems that use bizarre crypto that I’m having a hard time cracking (or for that matter, ephemeral SSL I can’t MITM); the program that intercepts cleartext read/writes and logs them is a script away.

I’m pretty arrogant, and I usually look at other people’s tools and frameworks and force myself to write my own. I’m going to have a hard time justifying that with PaiMei.

PaiMei seems like a huge accomplishment for Pedram, cooler than 95% of what I saw at Black Hat this year. Congratulations, and thank you.