100,000 Joyent Accelerators

We just delivered the 100,000th Joyent Accelerator to a customer. That’s a big milestone. Congratulations to the Joyent team. And congratulations to our customers who are doing such interesting things with Joyent Accelerators, everyone from Prince (the artist known as), to all the Facebook developers, to the many enterprise shops removing the barriers of IT from the innovations of smart developers. Onwards to 1,000,000.

Billions Served: Joyent Accelerators a Real Platform for Growth

Mark Mayo of Joyent gave a presentation yesterday evening at a Facebook developer garage in Vancouver, Canada. One statistic from his presentation really stands out. Joyent provides on-demand infrastructure for one application serving nearly one billion page views per month. One billion. Moreover, the infrastructure cost for that application is just over $10K per month. And, if the fickle desires of Facebook users turn away from this customer, they aren’t tied down to a contract. We help people scale up, and scale down.

As Rod said in an earlier post, Joyent is powering 11% of Facebook application usage and growing rapidly. Joyent can do this because the Joyeurs have built a real, open protocol, open standards, professional cloud computer. We have hardware load balancers and high-end routers capable of driving billions of page views across our entire network every month.

I am really proud of what the Joyeurs have accomplished. Congratulations.

P.S. the application is a Rails app. I think these facts put to bed any issues regarding Rails and scaling.

(Photo and blog quote from: Miss 604)

DTrace for Ruby 1.8.6

Earlier this week we released a patch that adds DTrace capability to Ruby 1.8.5. We chose 1.8.5 because it’s what we’ve been using in production, though 1.8.6 is the latest stable release.

It now brings me tremendous pleasure to release a patch against Ruby 1.8.6. You’ll find it in the same repository as the other patch:

The probes are the same for the 1.8.5 patch. We will be building 1.8.6 binaries shortly and you’ll find them in the same place as the 1.8.5 binaries.

The patches for 1.8.5 and 1.8.6 are released under the same license as Ruby.


DTrace for Ruby is available

In previous posts we’ve mentioned we’re working on a full Ruby DTrace provider set for Ruby 1.8.5. We’ve finished a solid base set of probes and it is ready for general consumption.

The subversion repository is at http://svn.joyent.com/ruby-dtrace.

The repository contains:

There are 3 binaries for Solaris provided here:

  1. ruby-x86: Standard GCC build
  2. ruby-CSK: Coolstack compliant build intended to drop into Coolstack 1.1
  3. ruby-CSW: Blastwave drop-in binary

The warning regarding the ruby-CSW binary is that it breaks with Blastwave tradition of using Sun Studio. Instead, GCC (/usr sfw) was used, due to various build issues with Ruby 1.8.5 and Sun Studio that are not related to the DTrace patch. The build name has been changed to Solaris 2.8 to ensure that extensions continue to work. You’re better off using the csw binary for debugging only until we can get a fully compliant build (coming real real soon).


The probes that are currently in Ruby-DTrace

Probe name          Description
function-entry      Probe that fires when a Ruby method is entered.
function-return     Probe that fires when a Ruby method returns.
raise               Probe that fires when a Ruby exception is raised.
rescue              Probe that fires when a Ruby exception is rescued.
line                Probe that fires for every line of Ruby executed.
gc-begin            Probe that fires right before a GC cycle begins.
gc-end              Probe that fires right after a GC cycle finishes.
object-create-start Probe that fires directly before a Ruby object is allocated.
object-create-done  Probe that fires when Ruby is finished allocating an object
object-free         Probe that fires every time a Ruby object is freed.
ruby-probe          Probe that can be fired from Ruby code (see below).


Probe               args[0]            args[1]       args[2]       args[3]
function-entry      Ruby class         Method name   Source file   Line number
function-return     Ruby class         Method name   Source file   Line number
raise               Ruby class         Source file   Line number   -
rescue              Source file        Line number   -             -
line                Source file        Line number   -             -
gc-begin            -                  -             -             -
gc-end              -                  -             -             -
object-create-start Ruby type          Source file   Line number   -
object-create-done  Ruby type          Source file   Line number   -
object-free         Ruby type          -             -             -
ruby-probe          Arbitrary string   -             -             -

Ruby Tracer Module

Included with the patch is a built-in Ruby module called Tracer. Tracer provides one method, Tracer.fire(str), which will fire the ruby-probe, setting args[0] to the string you pass in.

We will continue to add and refine probes, will be blogging about some more examples, and we are accepting requests for additional probe points (just email jason at joyent dot com). We will be developing patches against 1.8.6 in the near future.

Solaris, DTrace and Rails

We committed ourselves to Solaris as our base operating system two years ago as Solaris was becoming OpenSolaris. We needed a solid operating system that was 32/64bit, can manage lots of CPUs and RAM, one that we could contribute to, and we realized that three features would be a competitive advantage if we became experts in using them in production: ZFS, Zones and DTrace (the pre-existing observability tools in Solaris are quite excellent).

With time, it’s been surprising to me how that’s not clear to a lot more people, and we at Joyent have found ourselves spending a considerable amount of time simply being Solaris advocates.

So this last Sunday we (Ben and I) had a DTrace jam session at the Obvious offices with Bryan Cantrill and Jeremy over at Twitter, and were just running through an expanded way of looking at what ruby processes are doing when in production (the luxury of the processes being an issue is what great load balancers gives you).

(Jeremy, Bryan, Ian; the MacBook Pro in front of Bryan happens to be Ben’s, Bryan only uses Solaris on an Acer Ferrari 😉 )

Bryan happens to be a great Solaris advocate, and his biggest hammer is DTrace. So we try and get him in front of as many customers as we can, so that he can make an experiential case for why everyone with applications in production should be using Solaris. James Governor seems to agree.

We use DTrace all the time in identifying performance issues in our customer’s and in our own applications (remember we have some of the largest and oldest rails apps around), but it helps when its creator makes an appearance and a case for it as well. It was a bonus that Ian Murdock was there as well. Ben covered his impressions (or as he said, his “verdict”) of Ian after being able to spend a Sunday afternoon with him talking about Solaris.

From just a little bit of time, and out of the box it was clear that a lot of CPU was spent in an odd place: raised exceptions would generate backtraces that were going through hundreds of frames.

The result was a ticket filed at dev.rubyonrails.org 16 hours ago, and David committed the changes to Rails itself 5 hours ago. Everyone benefits from something pointed to by DTrace on Solaris.

And I think this is much in line with David’s main points about communities around open source projects, and a nice example of an open source operating system and tool giving you specific insights into a open source language and web framework, and then submitting and applying fixes to the right places. Blaine, the guys at Twitter, and DHH should be proud of this example.

But this introspection into what the Ruby processes are doing is still not deep enough. DTrace tells us from the outside of an interpreted language what it’s doing in regard to tens of thousands of different “probes” throughout the operating system.

You can see the number of places DTrace is hooked in with

$ dtrace -l | wc -l
$ dtrace -n fbt:::entry'{@[execname] = count()}'
dtrace: description 'fbt:::entry' matched 24798 probes

That’s more probes then I often know what to do with, but it’s great that they’re there.

Nearly a year ago, we worked with Bryan and the DTrace team to get as complete a set as possible of is-enabled probes into Ruby and to maintain that going forward (there’s only been patches against 1.8.2 up to this point). We’ve done that internally, and now with the increased use of Solaris by our own customers, and with DTrace showing up in the what is likely the most common development platform for Rails, Mac OS X, we renewed our collaboration today.

So we’re polishing up the patches for Ruby 1.8.5 (the version we and our Accelerator customers still run in production, and we’ll do 1.8.6 as well once that’s done in QA), and we’ll be sending those off to Bryan and his crew by Thursday for some vesting, and we’re aiming to release those next week.

The result is going to be a tremendous amount of insight into production Rails (and other Ruby) processes, and we hope that lots of improvements (even application-specific) can come from that.

Scale With Rails Conferences

Big news for those intent on building the next dapper, gleaming and endlessly scalable web application on the Ruby on Rails platform. Joyent is presenting three conferences this year called Scale With Rails, featuring our own Jason Hoffman and Luke Kanies. Jason and Luke will be focusing on every aspect of building high-end Rails apps: “power, space, location, staffing, hardware, components, network, operating system(s), development, staging, servers and, most importantly, the cost.” These are intense, two-day sessions, invaluable for everyone from developers to CTOs.

Early registration and team discounts are available.