Reminiscences about Lua and Mosquito
Wes Brown | June 2nd, 2008 | Filed Under: Development, Navel Gazing
When Ephemeral Security first came up with the concept of the injectable virtual machine over coffee, we had an aggressive schedule to meet. We needed a proof of concept prototype running within a month in order to meet the submission deadline for Defcon 13.
A perfect language for the task was a glimmer in our eyes, but we did not have the time to implement one. So we profiled various lightweight environments, and Lua looked quite promising. It was small, it was easily extensible, and it was portable across many architectures.
Ephemeral Security settled on LibTomCrypt as our cryptography library, as we considered it the gold standard for such software. It was small, it was efficient, it was well-understood, and it was far easier to use than OpenSSL. OpenSSL does indeed have cipher functions, but it is strongly oriented towards using them in the context of SSL transactions.
The other major component was LuaSockets. We judged that the combination of Lua, LuaSockets, and LibTomCrypt was enough for a proof of concept. We made the deadline for Defcon submissions with an early prototype, and we had a later version ready for demonstration at the actual presentation.
We hated Lua. It wasn’t Mosquito Lisp. We found Lua to be inferior to a language that existed only in our heads at the time. Although, when we needed a prototype that could hobble across the stage in a month, it was the fastest path from idea to novelty. And it was also the slowest path from novelty to tool.
Lua seemed to make simple things complicated, and the misuse of tables for both arrays and objects made things harder than they should have been. Lua also did not have solid and reliable primitives for I/O. It was really awkward to work with. From our perspective, it had a poor debugger, poor I/O, poor architecture, and a poor C API. It was a good extension language, but it was difficult to extend. Our cryptography code was solid, but affiliation could not be gotten to work reliably on top of LuaSockets.
But it did the job, and we had a very small environment suitable for second stage injection as a proof of concept. It was a sickening irony that the projector that Defcon provided did not work with my Powerbook, and we had to borrow a laptop from the audience to do our presentation. So we never did get to do a demo of the Lua-based MOSREF in at Defcon 13.
We learned several lessons from that particular adventure:
- Always make sure the laptop works with the projector provided ahead of time
- We needed an I/O oriented virtual machine and language
- Concepts that are simple in idea can be devilishly complex when we get to details
The next year, we did much better with Mosquito Lisp, thanks to what we learned from using Lua.
For those of you curious about it, here are the presentation slides, and the source code.
EDIT: ‘We’ adjusted to Ephemeral Security in reference to LibTomCrypt to clarify that is is not a statement or belief held by Matasano Security.
Vyacheslav Egorov
June 2nd, 2008 10:42 pmLuaSocket - not LuaSockets
From my perspective Lua is the most elegant language I’ve ever encountered. Brilliant C API, well thought architecture, responsive community… And I hardly need a debugger — nontrivial code slip from my fingers bugfree.
In fact it’s just a matter of philosophy — you can not write a good code until you accept philosophy of the language you are using.
Thomas Ptacek
June 2nd, 2008 11:17 pmSpeaking in my official capacity on behalf of Matasano Security, LLC, I’d like to point out that it is not Matasano policy to consider LibTomCrypt the gold standard for anything.
Mosquito Lisp at Pensieri di un lunatico minore
June 2nd, 2008 11:19 pm[…] I was perusing an interesting posting by the gang at Matasano, I came across a reference to Mosquito Lisp, which is a varient of Scheme designed specifically for […]
Wes Brown
June 2nd, 2008 11:42 pm@Vyacheslav: Noted, thank you for the correction. It had been a long time since I’d touched LuaSocket. On a certain level, I agree with you about Lua. The problem is that Lua’s philosophy is to be as vanilla as possible. It is a very nice language if you need to plug control logic into something greater.
But we had very different goals, and that was to make a small thing rather than a big thing. And we found that Lua’s philiosophy and design goals worked against us in that regards.
@Thomas Ptacek: Indeed, indeed! It was the opinion of Ephemeral Security, not of Matasano Security, LLC. Thanks for clarifying this.
Vyacheslav Egorov
June 3rd, 2008 4:51 am@Wes: Lua is indeed suitable for development of standalone applications. But yes, sometimes I think that Lua definitely lacks some libraries and features, so I have to create my own bindings, dive deep into internals of old rotten ones and write some metalua macroses…
Roberto Ierusalimschy wrote in PiL “Lua gives you the power; you build the mechanisms.”. It’s painful to reinvent the wheel, but in the same time it makes certain things simpler. And once you have one, you never need to rebuild it again.
“Ideal Language for All Programmers and All Tasks”. There is no such thing. So if certain language is not suitable for your task, just do not use it. No one enforces you… And if you persist: never complain. It was your choice, and your fault. Not the fault of the language. That is how I do things.
// No offense. Sorry, if it looks like one…
drrr
June 3rd, 2008 9:25 amIs Mosquito Lisp still being developed? If so, where can we find current code.
Wes Brown
June 3rd, 2008 9:33 am@Vyacheslav: That’s exactly my point. There is no perfect language for every task, and Lua turned out to be the wrong language for what we wanted to use it for. We were attracted to its relative maturity and its size. But other aspects made it quite unsuitable. It was indeed our choice to pick Lua.
If you note, we did not use Lua next time — we reinvented the wheel with Mosquito Lisp using what we learned. No offense was taken.
@drrr: Sort of. Mosquito Lisp as a security-specific language isn’t being developed anymore. But its descendant, intended to be more general purpose, lives on as WaspVM. Mosquito Lisp was designed for a particular goal and it didn’t quite meet what I expected.
If you want a network-oriented streaming Lisp1 that’s small, and portable across multiple architectures, check out WaspVM at http://waspvm.blogspot.com/. Please note that I am not in any way involved with this project, nor is Matasano.
I’m currently working on another project along the vein of injectable virtual machines, and I’ll reveal it when I think it is ready.
brl
June 3rd, 2008 9:36 amIs Matasano working on an injectable virtual machine project? If so, which VM will you use?
Wes Brown
June 3rd, 2008 9:52 am@brl: It’s one of my projects, yes. And I hate being mysterious, but I don’t want to reveal what’s behind the curtain without having something more concrete.
brl
June 3rd, 2008 10:17 amIt’s one of our projects too. We ended up choosing the Java VM after a false start with Ruby.
Charlie Cleveland
June 3rd, 2008 12:35 pmThere is a pretty solid graphical Lua debugger available here (just like MSVC):
http://www.unknownworlds.com/decoda
rikitan
June 4th, 2008 4:53 am@wes: can you elaborate on which expectations weren’t met by MosquitoLisp? It seemed like that you took it to a quite advanced stage. Why did you abandon it?
Wes Brown
June 5th, 2008 1:09 pm@brl: It’s been my observation that is hard to reduce an off-the-shelf VM to the size that is needed to be really usable.
@rikitan: I’ll be elaborating on this in a future blog post. But in a nutshell, it achived its goals admirably. It is just that at the end of the project, I realized that the goal it achieved did not serve the purpose that I had in mind as well as it could have.
In other words, it was small, it was targeted towards 64K-128K in size, and it had an amazing amount of versatility and functionality for that size, but it wasn’t small enough.
Wasp Lisp Developments
June 6th, 2008 9:09 amSurprised by Loud Noises…..
I was surprised to see how many people wanted to know why development of MOSVM had stopped; I had made some faulty assumptions about the community’s interest in MOSREF, MOSVM, and what would happen when Wes and I moved on to other things….
sdlkfjsldjkf
June 18th, 2008 3:32 amMr. Ptacek, I’m glad you posted that correction.
Matasano’s blog is informative and overall good, but the following statement:
“We settled on LibTomCrypt as our cryptography library, as we considered it the gold standard for such software.”
is one of the most ridiculous statements I have ever read on this blog. The author of LibTomCrypt is a complete idiot and I can authoritatively say that guy does not know jack shit about cryptography.
Wes Brown
June 24th, 2008 2:59 pmI have edited the post to reflect clearly on Matasano’s position. Developing Mosquito was a learning process, and LibTomCrypt looked appealing from the perspective of a programmer trying to minimize the impact of the cryptographic library on the VM in terms of size. LibTomCrypt did offer a nice catalog-esque functionality in which we were able to link in only the specific ciphersystem that we needed.
Needless to say, I have been educated as to LibTomCrypt’s inadequacies since then.
Leave a reply