A brief update with some numbers for hardware load-balanced mongrels

Back in August, I posted about a good-sized evaluation I was going to start doing about the horizontal scaling of different proxy engines and load-balancers across lots of mongrels.

But in short, we’ve stayed with F5’s BIG-IPs for at least one additional reason beyond their ability to handled gigabits of traffic across many many backend servers, and that additional reason is the ability to see inside packets and write iRules. iRules that can scrub http-reponse outs for credit card numbers and replace them with a “NOT ALLOWED TEXT”; iRules that can direct API traffic, bot traffic and traffic from specific user-agents to different backend pools (so you can separate API traffic from what people are hitting in a web browser), and iRules that can send .gif, .jpeg, .png off to different backend pools. All without any code changes, and this is just touching on some of the capabilities.

And to give you a general performance example, we recently added another 48 mongrels on 6 Joyent Accelerators behind a single public IP address to a set of BIG-IPs that are already directing 100s of Mbps of traffic, and without even getting around to tuning yet or caring about the backend numbers and configuration, we’re able to do quick 1 second bursts (like below), and then 10 seconds, 100 seconds, 1000 seconds, and one hour running benchmarks and get numbers like these below:


$ httperf —hog —server adomainname.com —port 80 —num-conn 4000 —rate 4000 —timeout 5 —uri=/theuri
httperf —hog —timeout=5 —client=0/1 —server=adomainname.com —port=80 —uri=/theuri —rate=4000 —send-buffer=4096 —recv-buffer=16384 —num-conns=4000 —num-calls=1

Total: connections 4000 requests 4000 replies 4000 test-duration 1.023 s

Connection rate: 3908.6 conn/s (0.3 ms/conn, <=99 concurrent connections)
Connection time [ms]: min 2.0 avg 6.0 max 66.3 median 3.5 stddev 9.4
Connection time [ms]: connect 0.5
Connection length [replies/conn]: 1.000

Request rate: 3908.6 req/s (0.3 ms/req)
Request size [B]: 85.0

Reply rate [replies/s]: min 0.0 avg 0.0 max 0.0 stddev 0.0 (0 samples)
Reply time [ms]: response 4.8 transfer 0.7
Reply size [B]: header 216.0 content 7850.0 footer 0.0 (total 8066.0)
Reply status: 1xx=0 2xx=4000 3xx=0 4xx=0 5xx=0

CPU time [s]: user 0.39 system 0.59 (user 38.5% system 58.0% total 96.5%)
Net I/O: 31112.6 KB/s (254.9*10^6 bps)

Yes that’s the sub page of a rails application doing almost 4000 requests/second. But it’s also a small page with a blip text and a couple of small small avatarish images.

The 10% “rule” for infrastructure costs

I was reading GigaOM the other day, specifically Google’s 2006 Money Shot, $10 billion in revenues, and his quote from Google’s earning release caught my eye:

Other cost of revenues, which is comprised primarily of data center operational expenses, as well as credit card processing charges, increased to $307 million, or 10% of revenues, in the fourth quarter of 2006, compared to $223 million, or 8% of revenues, in the third quarter.

A common rule of thumb I tell people is to target their performance goals in application design and coding so that their infrastructure (not including people) is ≤10% of an application’s revenue.

Meaning if you’re making $1.2 million dollars a year off of an online application, then you should be in area of spending $120,000/year or $10,000/month on servers, storage and bandwidth.

And from the other way around, if you’re spending $10,000 a month on these same things, then you know where to push your revenue to.

This is of course a general rule of thumb, and really applies to applications (not say more fundamental hosting or infrastructure products), and applications that consume more bandwidth and storage then usual can push this up.

But it’s nice to see that Google is around there.

Jason Hoffman will be doing a tutorial at RailsConf 2007 in Portland

I’ll be doing a half-day tutorial at RailsConf 2007 Thursday morning titled Scaling a Rails Application from the Bottom Up.

The abstract is:

Ruby On Rails is an opinionated framework for developing web applications and has a considerable amount of flexibility in the back end.

While the framework is quite successful in removing the need for developers to worry about many back-end specific details, this separation tends to make issues with deployment, scaling and financing appear at inopportune times.

The tutorial will cover the full range of considerations in detail: power, space, location, staffing, hardware, components, network, operating system(s), development, staging, the choices in web, database and application servers and, most importantly, the cost. The range of costs will be from bootstrap to efficient enterprise. The perspective is one that will interest everyone involved in the development, deployment and financing of Rails-based applications.

The Joyent guys will also be getting a big suite in a hotel by the Oregon Convention Center for a nice series of parties while we’re in town. Long live the fat boys!

Hopefully we’ll see many of you there.