Is Open Source Rootkit Detection Behind The Curve?

Dave G. | December 19th, 2006 | Filed Under: Slashdot Rounddown

I was reading this post about rootkit detection on Linux. They go on to talk about chkrootkit (an NMAP top 100 tool!) and rootkithunter. After doing a couple minutes of research, I was pretty sad. As far as I can tell the majority of the techniques are limited to signature style detection. The more ‘hardcore’ tests:

Every operating system has is advantages and disadvantages like the differences between tools en disk structure. Some parts of an operating system are not available to others, so we can use not all tests every time.

Linux - compare processes in ps against the available files in /proc

FreeBSD - look for differences between the output of netstat and sockstat

It seems like we haven’t progressed much since 1996 on rootkit detection. Or maybe this is just an open source issue. Or maybe I didn’t dig deep enough. Because in 1996 i seem to remember people at least doing:

  1. connect()/bind() to every port and compare that to the output of netstat.
  2. kill() every process with an innocous signal and look for pids that exist but dont appear in the output of ps.

I know there were others, what else am I missing? Yes, I know these techniques aren’t particularly effective against more modern rootkits. Yes, I know, this is a game of constant catch up. But we are further behind than I thought. This isn’t meant to be a dig on the authors or the tools. I am genuinely asking if there are smarter things we can be doing.

In other news, we are really busy, and the holidays are fast approaching, so our apologies for the lack of posting lately.

Viewing 20 Comments

    • ^
    • v
    Some of the better signature analysis systems have moved into the kernel. I know this is also old hat (I think the first RSA signature checking in the kernel I knew about was available for FreeBSD circa 1998).

    In this regard, the latest in signature checking would probably be DigSig/DSI and StJude/StMichael. I found out about StJude from the AWL Press book, "Wi-Foo".

    Other methods for rootkit detection should certainly include analysis of system calls. The XenAccess project, and some of the stuff from the HoneyNet project (UberLogger comes to mind) would be useful, as would strace, truss, ktrace, LTTV, etc.

    I'm sure there are other Unix/Linux rootkit detection and analysis methods that are new and innovative, but possibly lacking running code. The only one I can think of off the top of my head is the Kernel Timing stuff from KD-Team.

    As a final note, I follow the logic of the grsecurity team, who claim that LSM and SELinux are backdoors waiting to happen. Remove them (and AppArmor if you run SuSE). Lock down everything with PaX, use smart permissions on your files, and smart filesystem layout for noexec/etc options - these techniques provide answers to the problems LSM, SELinux, and AppArmor try to solve without incurring overhead or a false sense of security.
    • ^
    • v
    Rootkit detection today is fundamentally broken. What people need is the ability to do tripwire like delta analysis not on directory trees but on seperate disk images of the same machine from different times.

    Nothing running on the machine itself will ever provide anything except a false sense of security.
    • ^
    • v
    Just a thought: maybe one reason there has not been as much progress as you expected to find is that detecting malicious network activity is the easier way to find compromised systems, especially in a large enterprise. Affected systems are usually just wiped and rebuilt. Going through the process of detecting a rootkit on a machine would indicate that there is already something amiss with the machine to begin with. At that point, unless there is some forensic value in finding out what happened, the system is just rebuilt.
    • ^
    • v
    People who think that network based detection of rootkits/backdoors is a better way (easier) then a host based detection apparently don't know much about covert channels…

    People who would like to focus on disk/file system analysis miss the fact the a rootkit/backdoor doesn't need to be persistent…

    There are also rootkits/backdoors which do not create any objects in the OS, like processes nor threads, not even handles… (and I’m not only talking about blue pill here) - so a supper hidden process detector won't help much either...

    Stealth malware detection should be based on a systematic integrity verification of the operating system (including volatile memory). Currently this is impossible to implement effectively (on any general purpose OS)...

    joanna.
    • ^
    • v
    I have to agree with Josh Daymont's comment. The most logical way to verify a system's integrity is to power it off, mount its filesystem on another (trusted) machine, and scan the disk for anomalies.
    • ^
    • v
    • ^
    • v
    In response to Dre, please explain why SELinux is a backdoor waiting to happen? Or link me to where I can read supposed claims.

    Also rootkits aren't a problem if your OS doesnt suck and your users arent given any privileges. Or at the very least those privileges are are kept to a strict minimum with SELinux with execshield or PAX just incase a vulnerability comes along. Of course denying loadable kernel modules, disabling core dumps and all that other good stuff are a good idea as well.
    • ^
    • v
    Hi,

    There are loads of ARKs and Hidden Finder Apps available for Windows, probably a lot more than many people are aware of. One of the best is RKU which is about to be updated yet again to RC4 any day now.

    For Linux along with chkrootkit and rootkithunter there is also Zeppoo which is Free too.

    All these and stacks more such Apps and info/links etc, are to be found in here - http://forum.sysinternals.com/forum_posts.asp?T... - Keep checking for the latest updates to these.

    Happy hunting,

    Spanner

    SpannerITWks
    • ^
    • v
    josh:

    I will go ahead and assume the tools work on some rootkits.

    joanna:

    There is no doubt that the techniques being discussed can be defeated with more sophisticated rootkits. However, right now, rootkits that sophisticated are alien technology compared to the sticks and stones of current open source rootkit detectors.

    all:

    Don't get me wrong. I understand that against a serious attacker it is getting closer and closer to impossible to detect a rootkit. But not all attackers are serious and networks can still have a large enough attack surface that every machine will get patched soon enough.

    Fully understanding that this won't catch a lot of attackers, how many does it need to detect to have been worthwhile to run :)
    • ^
    • v
    "People who think that network based detection of rootkits/backdoors is a better way (easier) then a host based detection apparently don’t know much about covert channels…"

    joanna: I guess this was directed at my comment. Easier does not mean better. I agree that if you could do systematic checking of files and memory, that is the "better way." But as you pointed out, it's currently impossible to implement effectively.

    What I had in mind was that if you have say, 20,000 systems to look after, watching network traffic is the better approach than having 20,000 rootkit detection agents running around your enterprise. Yes, covert channels are difficult (if not impossible) to detect with signature-based alerts, but any system that is compromised will undoubtedly leak some kind of anomalous network behavior. Not only between the attacker and the victim, but with other hosts as well. Unless an attacker compromises THE system she is after and then doesn't do anything else except install the invisible rootkit and then capture keystrokes and slowly steal files from that one system, there will be something amiss. Detecting this "something" is more practical at the network level than the system level, especially for a large enterprise. And you don't need to detect the covert channel directly, either.

    I understand your approach (and I've read your papers - nice work ;-)), but the comment I posted above was directed at the practicality of detecting a compromised system versus the "better way" which is impossible, as you stated.
    • ^
    • v
    DaveG, how do you know that sophisticated rootktis are "alien technology"? After all, you admit that we lack effective mechanism for their detection, so how do you how many of them are installed in the wild?;)

    General note: people who believe that prevention technology is a good solution against system compromises miss two important things:

    1) *any* kernel prevention (OS sandboxing) technology, I can think of, has been at least once bypassed. That applies to: PaX, grsecurity /dev/(k)mem patch, BSD's securelevel, SELinux, Windows IPD and, of course, Vista driver's signing policy ;) Add here also all the bugs in kernel drivers...

    2) even a perfect prevention technology will not protect you against malicious admin (or smart social engineering on your admin).

    joanna.
    • ^
    • v
    Re: Joanna, A (malicious) admin already has legal, trusted access to the box, assuming you asked him/her to be the admin (or yourself in that case). If you cant trust the person you hired to admin your box (or yourself in that case) then you have bigger problems then a rootkit.
    • ^
    • v
    If you hire a maid to take care about your apartment that doesn't mean you need to *fully* trust her or do you? What you need is a way to verify that she didn't do anything wrong.

    The point is: to be able to verify that a system is not subverted. Currently we can't do that and that's really disturbing.
    • ^
    • v
    • ^
    • v
    I would not hire a maid I did not know. This is why organizations do background checks on people who will have access to systems or confidential company information. They want to measure how likely that person is to go overboard, and sabotage that data or compromise it. Because they know that person has complete trusted access to it. But I do agree there is no good way to know, for certain, if a machine has been compromised or not. Its a 'risk' thing when your dealing with the human factor.
    • ^
    • v
    ChrisR: I'll have more on this later, as I read about it once and can't find the link now. check the wikipedia entries for LSM, etc

    elad: nice blog post. thank you for stealing my anti-rootkit playbook - albeit you did turn it into quite a work of art. I think you trust bsd securelevels too much, but joanna called me out on grsecurity having been patched for security once, as well.

    you missed a few points, which I can get into somewhat here. first, you didn't talk about StJude. secondly, which is mostly my fault because I missed this point in my first post - you didn't discuss virtualization.

    I was in shock when joanna wrote redpill, and I went into a 3 day coma after reading about bluepill. but immediately, I started to think of ways of stopping bluepill. my gut reaction told me, "well what if I sit in ring0 and watch for this constantly?". then I recalled virtual machine introspection, which allows access to the memory of other vm's across the system. hence, XenAccess.

    I have more (which will hopefully bring me back to the ChrisR discussion as well), but it will have to wait until I verify a few things. I'll try to make a guest post here if it's really that important. I'll also address the covert channel issue that joanna is simply dead right about.
    • ^
    • v
    first, let me help you with why lsm is a backdoor waiting to happen. I believe it was addressed in a thread we had on tech-security@ some 4 months ago:

    http://mail-index.netbsd.org/tech-security/2006...

    (I suggest you read through the entire thread, it may give you a better view of the bigger picture of what I'm trying to do.)

    this post, and the comments in it, are referenced from my own post. two aspects of rootkit detection were (like I told joanna already ;) intentionally left out: hardware-based detection and virtualization. stjude and stmichael, are, mind you, an abomination. they are equally hilarious to the claim in the dsi page that "dsi can stop buffer overflow attacks", while the footnote says "well not really, we can stop execution of /bin/sh".

    furthermore, it is interesting to read that I was stealing your "anti-rootkit playbook", whereas I'm more or less coordinating the very development of everything I wrote about, for some 2 years now:

    http://www.securityfocus.com/infocus/1878

    but anyway. two notes:

    bsd securelevels: in practice, they are badly implemented everywhere; specifically the raw disk access policy. at the moment, there's no way to enforce that policy correctly, but I'm slowly fixing that, along a bunch of other things, in netbsd.

    grsecurity/PaX having security issues: anyone that is pointing out security issues in software should probably dedicate 5 minutes to read this:

    http://www.cs.columbia.edu/~smb/papers/acm-pred...

    (in other words, are you seriously suggesting that anything that had a security hole should not be used? or is that a much broader question, involving the design, authors, development process, etc.?)

    and I can't stress how much I'm waiting to hear what you have to say about covert channels. :)

    merry xmas!
    • ^
    • v
    Hey Elad - nobody says (at least myself) that something "that had a security hole should not be used!" In fact I highly respect the PaX work and I'm also a big supporter for non-blacklisting-based prevention technologies.

    I only want to stress one thing: we can not rely on *prevention* only - we also need *detection* to complement prevention.

    And I believe that the proper approach to detection is a *systematic integrity verification* of the operating systems (and applications), understood much broader then the integrity verification implemented in e.g. Tripwire.

    Also, I do not believe in efficient network-based compromise detection, because we have advanced forms of covert channels which simply can not be detected in practice.

    This doesn't mean that I think that all sorts of traffic monitoring is pointless - it surly is not - just it's not a way to fight stealth malware...

    marry xmas || happy holidays!
    j.
    • ^
    • v
    First of all, sorry for being late to this thread...

    I think the tool you mentioned that does the connect/bind+kill stuff is rootcheck (now part of ossec). It basically does three things to detect anomalies in the system (that may indicate the presence of a rootkit):

    1-Attempts to bind to every TCP and UDP port. If it can't bind the port (port is used), we check if netstat is reporting it.

    2-Attempt to kill(0), getsid and getpgid every process (from 1 to maxpid). We compare the output of these three system calls with ps and proc (where available).

    3-Compare the output of stat st_nlink with the count from readdir.

    I know these techniques can be evaded, but they are sucessfull against most of the public known unix-based rootkits (99% still based on system call redirection). Rootcheck/ossec also has the rootkits signatures stuff...

    In addition to that, OSSEC also does file integrity checking and log analysis to complete its HIDS tasks...

    Thanks,

    Daniel
    • ^
    • v
    Just correting my previous post. Ossec/rootcheck does 4 things... The 4th one is:

    4- Attempt to read every file in the system and compares the size read with the one from stat.

    Thanks,

    Daniel B. Cid

Trackbacks

close Reblog this comment
Powered by Disqus · Learn more
blog comments powered by Disqus