Thoughts on Ten Years of qmail Security

Thomas Ptacek | November 2nd, 2007 | Filed Under: Defenses, Uncategorized

Ahh, qmail. What is it about Daniel J. Bernstein’s full-featured Sendmail replacement that we can’t shut up about? Oh yeah, maybe it’s the nine years that have gone by since its last major release in 1998 which saw not one exploitable vulnerability. qmail is one of the top 5 Internet mail servers. Not one other project of similar standing shares this track record. It’s actually kind of spooky.

So, we’ve written about qmail before. And, if Georgi Guninski reads our blog, our qmail writing could conceivably cost me $500. What excuse can we make to keep talking about it?

How about a just-published retrospective on qmail and software security by DJB himself?

DJB isn’t really talking about qmail; he’s using it to illustrate lessons learned about software security. Here’s an example: three answers for eliminating security vulnerabilities from software:

  1. Eliminate bugs from code. As Theo said, “we assume all bugs are vulnerabilities”. Kill bugs, close vulnerabilities. How do we kill bugs? Engineering has wrestled with that for decades. Some answers: formalized software engineering. Automated QA and coverage testing. Unit testing. Code review. Up-front security design.

    It works. qmail won by replacing 80’s era C library interfaces with new libraries that made it hard to write incorrect code. Microsoft won by beating the living crap out of their code and release schedule with security-focused QA.

  2. Eliminate code. More code? More potential bugs. Minimizing code footprint reduces exposure to bugs. “Complexity is the enemy of security”. How do we eliminate code? Bernstein’s best answer is reuse: expend effort so your programs exploits your platform, instead of reinventing the wheel.

    It works. qmail won by taking advantage of somewhat obscure Unix features like programmable resource limits. The major enterprise web stacks like J2EE and .NET allow web apps to eliminate code by building on well-tested session, database, and templating libraries.

  3. Eliminate trusted code. If you have to spend lines of code to deliver a feature, at least keep the code out of the line of fire. How do we minimize trusted code? Bernstein grapples with this question, arriving at two answers: use the operating system to compartmentalize code, so that vulnerabilities can’t do real damage, and run sensitive code in interpreters, trading speed for security.

    It works. Minimizing trusted code (or what consultants would call “the attack surface”) in qmail is the signature architectural trait of qmail. But you get the impression that Bernstein is least satisfied with qmail’s execution here. At the same time, it’s impossible to miss the wins other projects see with this approach: the vast majority of deployed enterprise software is innoculated against memory corruption vulnerabilities because they’re run in high-level languages like C# and Java.

Bernstein has a few grenades to toss. I disagree with his take on “least privilege”; revoking program rights to operating system resources like files and sockets isn’t a distraction. Yes, there’s a (subtle) difference between sandboxing a broken chunk of code and designing it properly to begin with; and yes, it’s true that access control schemes are plagued with “game-over-equivalence” problems (“you can read everyone’s mail, but at least you didn’t get root”). But I found his argument mushy, seeming to contradict his second “minimize the TCB” strategy, and Saltzer’s principals still seem vital to me.

There’s a conspiracy theory among those who have audited qmail’s idiosyncratic codebase that Bernstein didn’t actually write it in C, but rather in a language he compiled down to C. Here he in on the subject:

When I wrote qmail I rejected many languages as being much more painful than C for the end user to compile and use. I was inexplicably blind to the possibility of writing code in a better language and then using an automated translator to convert the code into C as a distribution language.

Of course, now that he’s spending his time writing compiler backends, maybe the new plan is to ship the entire language stack along with his applications.

Bernstein had the benefit of hindsight with buffer overflows, but had to weather integer vulnerabilities along with everyone else. qmail fared remarkably well (the only known integer overflow in qmail isn’t exploitable on any real deployment). The jury’s still out on what the next major bug class will be (our money has been on uninitialized stack variables). We may eke out a few more qmail bugs over the next 10 years. But relying on it seems like a safe bet.

Viewing 32 Comments

    • ^
    • v
    I love qmail. Now if only my web host would let me use it. Guess that I'm gonna have to get my own box sooner rather than later...

    Funny how qmail used the concept of developing on a secure platform/framework then. Now, everyone recommends doing so for the same reasons. Why reinvent the wheel and at the same, reintroduce problems that allow developers to make the same mistakes over and over again.
    • ^
    • v
    You know to be fair, it should be pointed out that while qmail base might be this good, do you think all the third party patches that everyone uses to get the functionality they need out of it are also as good?
    • ^
    • v
    What is it about Daniel J. Bernstein’s full-featured Sendmail replacement that we can’t shut up about?

    The unreadable homegrown install infrastructure? God forbid DJB would sully his code by shipping it with an automake script.

    ... Sorry, just still bitter from that one time I had to try to grok that code because it was breaking on some system. I do admire DJB's design and coding practices, and I ran tinydns for the couple of years that I ran a name server at all. (I also ran qmail as my local MTA until I got bitten once too many times by the qmail-OSX fail-to-start race condition.)
    • ^
    • v
    You either love DJB's packaging, or you utterly loathe it. Or you admire it. Or you're ambivalent towards it.
    • ^
    • v
    Automake? hahahaha, laughable to say the least. None of the versions are backwards compatible, automake/autoconf and others have given me and many other developers so many problems in the past that when it came to looking at qmail it was like a godsend.

    The fact that it can conditionally compile certain things based on checking the output of the programs is awesome. Also, compile time is shorter and faster than using any configure programs. For example, on FreeBSD compiling the new modular Xorg 7.2, I bet more time is spent running configure scripts than is used to actually compile the individual modules.

    DJB's homegrown install infrastructure is better than any other I have seen!
    • ^
    • v
    @jf: Yes, I agree -- what "full-featured" means has changed over the years. qmail is abandonware (really, really good abandonware).

    @Matt: Dude, most automake scripts are probably bigger than the qmail source code. Reducing code size is crucial.

    As great a triumph as qmail is, it's as much a lesson in what not to do as it is in what to do. That is: Unless you are a super genius, do not write C code that receives input from across a privilege boundary; and no, despite what you think, you are not a super genius.

    No, really, you are not a super genius.

    You too, in the back. I can see you fidgeting.

    Take the lessons of qmail with you into Managed Code Land, and you might stand a chance.
    • ^
    • v
    If you ever build a large package and feel like your compiler is god-aweful slow, try rewriting a minimal makefile for it. Apache's build just flies when you ditch the autoconfig mega build system.
    • ^
    • v
    qmail misses the point.

    The real security problems with email are 'soft' security problems: spam, viruses, phishing and the like.

    10 years ago you could set up qmail and walk away from it.

    A few years later, there would be a virus outbreak and your /var partition would be out of space in an hour or two. Somebody would use your mail server as an open relay, so you couldn't get mail out to AOL users. Your users connecting via dial-up and DSL would find that they couldn't connect via port 25.

    Around 2002, e-mail became a homeland security problem, and qmail didn't catch up. Dan Bernstein has the crazy idea that software is like a painting -- you finish it, get it perfect, and then it's done. He wouldn't maintain it, and he licensed it under terms that wouldn't let others maintain it.

    Yes, you can install about 20 patches to get a qmail system that can survive today's environment, or you can use sendmail and postfix. Postfix has a similar architecture to qmail and has similar security and configurability benefits. Sendmail is a bear, but at least they keep it up to date, which you can't say for qmail.
    • ^
    • v
    If you want more features, you can always trade security for them and run something other than qmail. You might, for instance, consider Exchange.

    For my part, the feature I care most about is that my mail gets delivered, without losing messages or the machine the mail server is installed on.
    • ^
    • v
    For the past several years every time the qmailsecurity discussion reappeared (generally elicited by Tom) I've felt compelled to make the same comment: The secure UNIX MTA discussion is not about Sendmail vs. qmail, it is about any of those two measuring up to the Postfix standards. Perhaps Postfix's virtuosity is not so well publicized because the project and its author has a much more humble and practical approach to security, yet he managed to keep an equally impressive track record and kept the project alive and up-to-date with current technologies and third-party tools (TLS, IPv6, milter, etc.) even if that meant that the project needed to transcend the original author's persona.
    While DJB talked about security backed up by the impressive track record of qmail, Wietse coded and his code speaks for itself because it is actually readable and maintainable :)
    • ^
    • v
    Ivan: I disagree, for two reasons.

    1.

    qmail has a better security track record than Postfix. I found a vulnerability in Postfix (when it was VMailer).

    2.

    Both qmail and Postfix derive their security from architectural decisions, and virtually every one of those architectural decisions originated (years prior to Postfix) in qmail.

    I'll argue that qmail made a contribution to the field that Postfix merely popularized.
    • ^
    • v
    Also: Venema was a real dick about Bernstein in public.
    • ^
    • v
    The fact that you found A vulnerability does not invalidate Postfix's security track record does it?. So what? George Guiniski found A vulnerability in qmail as well.
    As for the architectural decisions, well I wouldn't credit qmail for them, they are the basic foundations for writing secure code and originated long before qmail (or Postfix) existed. DJB and Venema just took them to heart and followed those principles in their design. Both of them are laudable efforts. I am just trying to point out that while qmail is often touted as the ultimate effort in securing UNIX MTAs, Postfix delivers more functionality, has an equally impressive track record (give or take one bug), is still a live project and doesn't make so much fuzz about it. Incidentally I think the personality traits of the creators of each package are clearly reflected in source code and the evolution of each project. I don't want to offend either author, their contribution to the infosececurity community is immense.
    BTW, being a "real dick" in public is hardly a valid argument in any discussion about security and even then Venema is probably the one I'd place last in the top-100 ranking of people that made meaningful contributions to infosec and have been "real dicks in public" at some point in time.
    • ^
    • v
    Also: did you pay Guninski? Did DJB pay? Is it fair to say that DJB initiated the trend towards commercialization of vulnerability information with his $500 bounty on qmail bugs?
    • ^
    • v
    no no, its not about trading features for security, its about if a product doesnt meet the required feature set then its not even an option, and if that option is to apply third-party patches that devolve the security standing to at least questionable then what are we actually carrying on about?

    re: guninski

    a bugs a bug, it doesnt matter that you suggest everyone wrap themselves in a security blanket, the fact of the matter is that the code could potentially do something incorrect, thats a bug in it. New Linux kernels no longer have a limitation on the number or size of arguments passed to a function, if that resulted in an argc overflow and your sudo got owned, would you call that a bug or that sudo should've had a reasonable expectation of environmental sanity?
    • ^
    • v
    s/function/program/
    • ^
    • v
    Ivan: from what I can tell, no exploitable bug was ever found in qmail. Nobody has ever cited a deployment of qmail that could in practice have been vulnerable to the particular integer overflow Guninski found; it's a data-driven integer error, not a counter-based error.

    The same cannot be said of Postfix.

    I strongly disagree with your take on the novelty of the qmail architecture, and cite all mainstream software, and in particular all MTAs, released prior to qmail to defend my argument. Nothing (that anybody used) worked that way before qmail.

    It's not even a mainstream design --- even among competent developers --- today. Compare DjbDNS to all other DNS servers.
    • ^
    • v
    I've offered to pay Georgi. Has hasn't taken me up on it. I appreciate that he hasn't, because it would sting. But I agree that he earned the $500.
    • ^
    • v
    jf: regarding "a bug is a bug" --- this isn't a semantic argument. There was a very specific qmail security guarantee, with very specific rules about what did and did not qualify. It was not like Knuth's TeX guarantee. There is no "no bugs allowed" guarantee; just "no exploitable security vulnerabilities".
    • ^
    • v
    Also: what has Venema done besides his qmail clone? Bernstein went on to replace BIND, then contributed a practical remote attack against AES, taught a famous (and constructive) class on Unix software security, designed and implemented a "finalist" stream cipher for ECRYPT/ESTREAM.

    I mention this to rebut your implied argument that DJB sat around and bitched while Venema kept right on coding. Bernstein clearly disagrees with you about the importance of SMTP TLS, IPv6, and milters. I'm glad he rescued me from BIND rather than wasting time on IPv6, and hope he'll take out SSH next instead of wasting time on DNSSEC.
    • ^
    • v
    tom: you can't have it both ways, *you* personally qualified Guninski's finding as a bona fide vulnerability worth the $500 prize, it is not fair to diminish its importance not that it suits you to do so. The bug count argument is settled, don't go there.
    Regarding TLS, IPv6 and milters: DjB may disagree with me and the rest of the modern Internet about their importance but guess what? we don't care, we expect MTAs to inter-operate with us, TLS zealots, IPv6 crazies and the spam filtering fundamentalists. This is the DNSSEC argument, only now you're on the other side of the fence.
    I won't address the "who's got the bigger contribution" argument because it doesn't lead anywhere (specially when neither of us had contributed a fraction of a fraction of what Bernstein or Venema had). The point of my original comment was to point out that Postfix is still out there, it has an equally impressive track record, it is actively maintained and up to date with current technologies, its based on the same architectural principles of qmail and has not been given nearly the same credit from the security community.
    Incidentally, I didn't know BIND has been replaced! OMG! when did that happen? Did I miss the party?
    • ^
    • v
    In this case, I can have it both ways: I agree that the $500 is there if Georgi really wants it, but disagree that it has any impact on the real-world security of qmail. You can't say the same thing about Postfix.

    To argue that BIND hasn't been replaced, you'd have to argue that you'd rather deploy BIND9 than DJBDNS. I can see lots of arguments for Postfix over qmail. I can't see many for BIND.
    • ^
    • v
    okay i don't disagree with your point, djb did in fact list a very specific set of requirements to meet supported configuration, imho that's a bit of a cop out- i mean from what I remember even running on x86_64 is unsupported (although my memory may just be way off here). Guninski's bug didn't fall into that stringent set of requirements, fair enough.

    I still think however that it should've been patched.
    • ^
    • v
    also, i just saw this: The jury’s still out on what the next major bug class will be (our money has been on uninitialized stack variables).

    astute observation, id say that or synchronization issues, there's still a lot of both out there. Interesting method to get uninit variables in win32 applications is to watch out for functions that return 3 possible return values- error, okay, and buffer not big enough.

    For instance, in the shell api the maximum path length is 260, however when using unicode api's its possible to have a path that is something like 34,000 wide characters leaving a slight disparity between the two. This means that a lot of the shell api will either return that third error value which is a positive value greater than zero and leave the buffer untouched- a condition that isnt explicitly check for in a lot of instances, or the path gets truncated which in turn leads to other interesting problems.
    • ^
    • v
    You are attempting to invert the burden of proof. I don't need to prove that BIND hasn't been replaced by DjBDNS, you need to prove the opposite.
    Name one OS that ships DjBDNS as its nameserver daemon. There, that's the pro-BIND argument: "It comes with the OS, I just need to configure it and I'm done"
    • ^
    • v
    You're right. "Replace BIND" is hyperbolic. I stand by the remainder of my argument.
    • ^
    • v
    Let's not have a dick-swinging contest by proxy between djb and Wietse.

    I disagree with the assertion that Postfix's architecture clearly owes much to qmail. The fact is that the coders in question are both of academic mien, and were immersed intellectually in environments that could easily have led them to similar conclusions.

    It seems pretty obvious that djb and Wietse both "get it", and have for ages.

    It doesn't all have to be vi vs. Emacs.
    • ^
    • v
    Wietse isn't an academic. Let's be clear about things. Bernstein is a cite leader in his field and a tenured professor; Venema worked at a University.

    Judging people by their contributions isn't "---- swinging" (I apologize for bringing that word into the discussion, let's kill it). What's Venema's long-term contribution? Specifically: what do I know now that I didn't know 15 years ago because of Venema?

    Ivan, I can trace far more of my knowledge to you than to Venema.
    • ^
    • v
    Tom, thanks for that completely undeserved compliment. I don't think you are serious tho, but if you insist in saying so then I would reply that I can trace large portions of my interest in security to Venema's contributions, so there is a transitive relation that applies here :)
    From my perspective the conjunction of tcp_wrappers, SATAN, Venema's replacement to portmapper/rpcbind, TCT and Postfix contributed quite a bit to my interest in and understanding of practical information security matters. Anyway, I reiterate that my comment was not intended to elicit a swinging-by-proxy contest between Bernstein vs. Venema, both of whom would laugh about this entire blog post if they ever read it, but to indicate that in terms of security Postfix is at least on an equal standing with qmail
    • ^
    • v
    I'm not the only one I know who deliver that comment about your contributions and mean it.

    I never used SATAN, and in my scanner-developing days was never inspired by it. I was, however, inspired by Dan Farmer's "Improving the Security of your Unix System Break Breaking In To It", which I bring up as an example of a clear contribution.

    I used wrappers and portmap3 extensively. I liked them. But what do I know now that I didn't know 15 years ago because of libwrap?

    It's not insulting, offensive, or juvenile to judge that one person has made a greater contribution to software security than another. Probably both have made a greater contribution than I have. Sure, Postfix is as secure now as qmail has always been. That doesn't make their contribution equal: qmail appears to have taught Venema how to write Postfix.
    • ^
    • v
    Thomas,

    "soft security" has everything to do with keeping my mail server safe and delivering messages. If a virus outbreak causes my /var/ partition to fill, that's a denial of service. If virus-infected computers are hammering my server so hard that the CPU is pegged or are making the head of my hard drive be in too many places at once, mail won't get delivered.

    From my viewpoint as a sysadmin, it's all about how much time I spend dealing with exceptional events.

    I got into qmail in the 90's. I loved the speed, the lack of buffer overflows, the modularity and the simplicity of configuration over sendmail.

    I got out of qmail around the time netsky and mydoom turned up -- I switched to Postfix.

    Was Postfix influenced by qmail? Yes! That's why I liked it so much. qmail ~was~ the best mail server of it's time when it first came out. If djb had kept maintaining it, or had allowed other people to maintain it, I might still be using qmail today.
    • ^
    • v
    Love Qmail. However, making sure it fits 'perfectly' within OS handling, is frusterating. Nothing new to BSD, etc, but wish ALL assumptions and details were spelled out so that the little details fit perfectly, without having to be DJB or a top developer.
    True of any program, just that lots to know on BSD.

Trackbacks

close Reblog this comment
Powered by Disqus · Learn more
blog comments powered by Disqus