Ruby’s Vulnerability Handling Debacle
Max Caceres | July 1st, 2008 | Filed Under: Development, Disclosure
A bit over a week has now gone by after critical vulnerabilities in ruby’s runtime were announced and a couple of interesting developments occurred:
The new official ruby releases which include patches for the vulnerabilities actually break critical functionality. In fact, some of the most popular ruby frameworks segfault with any of these releases.
It turns out that the breakage brought by the new releases is not due to the security fixes for Drew Yao’s vulnerabilities, but to other changes that were merged into each release’s branch.
No response or timeline from ruby’s maintainers to produce a good release has been made public.
In an attempt to provide a workable middle ground, third-parties create their own patches, and so do distributors. Some start from a patched release and then work back through the changes until tests pass, others take the opposite approach and start with a last “known-good” release against which they apply selected patches to plug the security holes. Needles to say, the results of each approach are not equivalent.
Developers relying on any of these frameworks are then faced with a difficult choice: wait an indeterminate amount of time for a good release from the official maintainers (and remain vulnerable in the meantime), or apply a patch from a third-party which may not plug all the security holes, or may include unforeseen bugs. This is clearly a problem. In the meantime maintainers have to scramble to get ruby back to a good state that actually plugs the holes while people complain. Everyone loses.
To some extent this is the other side of security response we sometimes advocate for and now we got a taste of: we want our patches and we want them now! And we got them, but our apps don’t work.
The beauty of open-source is not just that we get stuff for free but that we also get a privileged view into the processes around building and maintaining wildly popular software. This is a great opportunity for us as both security and software people to look into why these processes failed and hopefully learn something. Here are my top 3 takeaways:
Keep a “last-known-good” branch to apply critical patches to. The fact that patches had to be applied to unreleased versions of the code means that it is impossible to measure the impact of the change. IMHO this is the single most important mistake behind this state of affairs. This is almost akin to not using version control. Security problems will pop up unpredictably outside of any planned release schedule, and often require a rapid response. End-users may not be able to afford upgrading to a release with new features and even bug fixes every time this happens. It is then highly desirable that a new release which only includes the necessary security patches is made available, to minimize the risk of getting pwned by a new bug or some defiant kid. Are you ready to release critical patches to all the deployed versions of your software even as you are working on its next version?
Open line of communication with your community. It is hard to believe that passing Rails’ tests isn’t an acceptance requirement for every Ruby release, since Rails is the driving force behind Ruby’s adoption. The success of any platform depends on the applications it supports, without the apps the platform doesn’t matter. Microsoft learnt this a long time ago, and that’s why you’ll see them test third-party apps with new releases and security patches, even though technically is not their responsibility. Who are the relevant members of your community? Are you involving them in your release process?
Be more open about vulnerability details. When the vulnerabilities were originally announced very few details were made available (CVE entries for the vulns have been updated with more details since then, though there’s still no entry for the Bignum bug). Because ruby is an open-source project anybody can go and access their source control system to figure out what was actually patched. The fact that you can should be no news to readers of this blog, as people have been doing the same with binary patches for quite some time. Having direct access to not only source but the full source control system is a luxury! Given that the official release didn’t work it was important to isolate the security fixes, and to easily test if 3rd party patches were effective against the flaws. In fact, one of the test cases showed that the official release missed one of Drew’s patches. Lack of details just made it harder for the people trying to help. Do you have a policy about disclosing vulnerability details in your software? Would partners and customers have enough information to evaluate the impact of your patch to their infrastructure?
Just yesterday Apple released a security update which includes fixes for the ruby vulnerabilities. The patched build appears to deal correctly with all test cases, including the string concatenation case, and with Rails’ tests. The official ruby downloads still have issues with both.
mac
July 1st, 2008 2:57 pmGuess this is why metasploit has started segfaulting.
Drew Yao
July 1st, 2008 3:53 pm“In fact, one of the test cases showed that the official release missed one of Drew’s patches.”
To clarify, that(crash when concatenating a string to itself) was a regression introduced in the latest version. It’s fixed by http://svn.ruby-lang.org/cgi-bin/viewvc.cgi/branches/ruby_1_8/string.c?r1=17530&r2=17529&pathrev=17530
which is not yet in a released official version.
The version of Ruby shipped with Mac OS X is not affected because it never had that regression.
Eric Monti
July 1st, 2008 4:12 pm@Drew
May want to double check that. The string concatenation segfault I pasted into my post (http://www.matasano.com/log/1070/updates-on-drew-yaos-terrible-ruby-vulnerabilities/) was created using the stock ruby interpreter on OS X Leopard.
Drew Yao
July 1st, 2008 5:23 pmThat test case would have triggered a different bug, an integer overflow, prior to Mac OS X 10.5.4.
In the patched official version of ruby, it would trigger a separate bug (dangling pointer after realloc). This bug never existed in the Apple-shipped version of Ruby.
Yeah
July 2nd, 2008 4:50 amThis is why meta meta meta meta self hacked on hacked object building helper do it all syntax fletching projects fail.
Stephane Grenier
July 2nd, 2008 11:04 amThis seems to be something that needs to be learned, relearned, and relearned again. People complain that they want a fast and agile system, and that’s great. However as a system becomes more mainstream, some processes have to be built to verify that it still works for everyone.
When Ruby existed before Rails (RoR) it didn’t matter that much. However now that you have more and more people jumping in, the speed of changes must slow down.
It’s interesting to see the parallels between Ruby and Java. Early on Java was able to make bigger and quicker changes because it had a smaller community. Now, after over a decade, changes take more time, require more testing, and better adoption by the whole community.
Don’t think this won’t eventually happen to Ruby if it becomes mainstream. Eventually the speed of change will have to slow down to make sure nothing gets broken along the way. Breaks like this will become unacceptable.
Bleeding edge is great, but it has its costs. As long as the benefits outweigh the costs, it’s all great!
Adam
July 2nd, 2008 11:06 amSince 2 of your three are targeted at software creators, I’m surprised that you don’t list “test the hell out of your releases” or “be transparent about your test process.”
Why isn’t Rubyonrails part of the Ruby test suite?
Max Caceres
July 2nd, 2008 12:46 pm@Stephane: agreed, although I don’t think becoming more mainstream is a bad thing (not that you were implying that). The fact that there’s a substantial level of attention highlights the fact that the Ruby team has done a great job so far.
@Adam: I don’t know, and it is surprising in a number of ways. Testing seems to be ingrained within the Ruby ethos, most Ruby projects come with great test suites that can be run very easily.
To be clear though, I think Ruby releases do get tested before GA, the problem is that Rails is not part of these tests.
Stephane Grenier
July 2nd, 2008 2:16 pmHi Max,
You’re right, I’m not implying that going mainstream is a bad idea, it’s a good thing!
And they’ve done a good job so far.
All I’m saying is that getting to the next level means that you’ll lose a lot of your flexibility and speed. The more adoption a language has, the more luggage it needs to support. And don’t forget that if Ruby supports Rails, Rails supports other websites.
Something else I’d like to quickly note, I believe that a lot of the excitement about RoR also stems from the fact that most RoR implementations are new, and they don’t have to support legacy monsters. Give it another 5 years and it will be in the same boat as all the other more “mature” languages.
And this is not a bad thing, it’s just something we all need to be aware of. Getting to that stage is a very good thing, it means you have a great product/language that’s benefiting many people!
Chris
July 2nd, 2008 5:21 pmDid I read that correctly? Ruby maintainers didn’t have a release branch, so they patched the latest dev build and threw it out there, thinking there would be no problems?
Even without a branch, couldn’t they have just rolled back to whatever revision corresponded to the release date, created a branch after the fact, and patched from there?
While Ruby is no Java in terms of maturity, it’s still been around for long enough where there should be a more robust development/test process.
Carey
July 2nd, 2008 5:54 pmChris, it’s really easy to create a branch or tag from an arbitrary Subversion revision, without needing to roll back the checked out copy at all. See the example at the end of: http://svnbook.red-bean.com/en/1.4/svn.ref.svn.c.copy.html
I have to admit that this has made me a bit lax about tagging releases on internal projects.
Max Caceres
July 2nd, 2008 5:57 pm@Chris: ruby has release branches. My understanding is that by the time the security fixes were applied there were other unreleased changes in them. In trying to respond quickly to the vulnerabilities the releases were cut with these other changes in them.
Leave a reply