Thanks for the Techcrunchy party

The Techcrunch party was a good time and it’s always nice to see friends and even some hosting customers.

Thank you Arrington et al.

I had a beer for those of you who couldn’t make it.

Picture by Scott Beale of Laughing Squid

We’ll be in London in September for RailsConf Europe 2006

The talks schedule is out for the first Rails Conference in Europe and both Koz and I are giving talks.

I’ll be talking about Ruby On Rails Applications: A Systems View and Koz is called Playing Nice with Others .

My talk is related to the Scale with rails work and God knows what Koz’s about.

I’ll be putting a few days on both sides of the trip and want to make sure I’m able to buy each textdrive/joyent customer a pint or two or three.

I’m staying at the Radisson Edwardian Kenilworth Hotel and will in London from Wednesday the 13th through Sunday the 17th.

If anyone has some ideas let me know.

The Scale with Rails Workshops

On a daily basis we receive emails containing things like

“I am looking for a host for a commercial ‘Ruby on rails’ website development that is due to go live in 3 weeks time. It is an update to a relatively busy PHP-based site, which is also on old hardware. The site features media streaming, user uploads and E-commerce functionality.”

“We are concerned about the long-term viability of Ruby on Rails as a development language.”

“My company has some concerns on whether or not Ruby on Rails is the right platform to deploy on if we have a very large scale app.”

These statements are interesting for a few reasons:

  • The first requires a good deal of systems infrastructure and expertise that has nothing to do with the Rails framework: Rails is likely to be the least of their worries.
  • The second equates Rails to a “language”, which it isn’t, Ruby is the language, Rails is a framework for certain kinds of applications. Both are open source and if one is truly concerned (as in “worried”) about the long term viability then you can easily invest in it by simply hiring people to work on them. That’s how we address our concerns.
  • And the critical point about the last statement is that Ruby On Rails isn’t a platform. It is a development framework that allows people to code “modern” web-based applications, and is just one part of the platform required for releasing, sustaining and growing an application out in the wild. “Rails” is not a replacement for Java, Python or PHP, that would be Ruby.

But the word platform is in fact key, and exactly what’s missing for most people. What’s the best stack and how does that “scale” now (both up and down) and years into the future?

I thought why not take all of the questions and concerns that I’ve heard over the last two years and all of the issues we’ve studied and addressed ourselves, and present a platform?

That platform would not only be best practices for even the largest of companies (because we’ve all been in large Fortune 500 companies), but then we’ll frame the entire discussion in a set of questions that should be and likely are on anyone’s mind. A set of questions that in fact are unlikely to change. What you’ll get then is not blind Rails evangelism, but the perspective of an architect, engineer and executive who currently thinks that the answer to one of those questions was to use the Ruby and then the Rails framework (and sometimes it isn’t, sometimes C++ is).

The resulting workshop then is a two-day, focused, logical and complete bottom-to-top workshop where people can learn about how to staff, plan, manage, develop, stage, test, deploy, monitor, scale their Rails-based applications on an entire platform and actual do it in Solaris grid containers. And it won’t end there, you can keep those setups for another month after your workshop, and you’ll be part of an elite community that’ll have continued access to us and each other.

We’re a solid, revenue-funded company with a staff of about twenty people. Besides our Connector and Strongspace applications, we have a number of internal and external “enterprise” applications like customer.joyent itself, our e-commerce core, our store and our monitoring and management tools. We also founded and run the hosting service TextDrive. We’re fortunate in that each Joyent developer has been working with Rails since it’s public release in 2004, many have been with Ruby for longer, have solid backgrounds in other languages and a few of them are members of the Rails core team.

If you have any questions about the series, feel free to email me (jason at joyent) or Luke (lkanies at joyent).

There’s some additional information at the Scale with Rails site and you can go and sign up for the workshops

Some have emailed me asking for little buttons for their sites, here they are and thank you for asking. Looking forward to seeing many of you at the workshops.

<a href="" title="Scale With Rails"><img src="" alt="Scale With Rails Logo" /></a>

What sysadmins start doing when hanging around designers

Bryan and Justin always crank out great looking stuff, so the systems guys try to keep up.

Red and orange run out to storage switches.

Black is the dark, private interconnects between servers.

Blue is out to the public like water flowing.

White is for remote ALOM and goes to the console servers, like a the archetypal knight in white shining armor.

Green which don’t see is used for the interconnecting and meshing the switches themselves. What’s green mean to you?

DTrace through a rails app

Bryan has a couple of quick quick examples of using DTrace to go through some parts of a Rails application.

As one starts working more and more with web frameworks that depends on many pieces from other people, it’s pretty clear that being able to quickly and completely look at what your application is doing from the request all the way down to the kernel and networking stack is critical (look there’s even DTrace providers for PostgreSQL).

And this is done in production and in a non-interfering way.


If you’re coming to one of the Scale With Rails workshops, expect to see plenty of this.

Scale With Rails Conferences

Big news for those intent on building the next dapper, gleaming and endlessly scalable web application on the Ruby on Rails platform. Joyent is presenting three conferences this year called Scale With Rails, featuring our own Jason Hoffman and Luke Kanies. Jason and Luke will be focusing on every aspect of building high-end Rails apps: “power, space, location, staffing, hardware, components, network, operating system(s), development, staging, servers and, most importantly, the cost.” These are intense, two-day sessions, invaluable for everyone from developers to CTOs.

Early registration and team discounts are available.

Jason Hoffman speaking at the Silicon Valley Ruby Conference

Here’s more information about Jason Hoffman’s upcoming talk at the Silicon Valley Ruby Conference. We look forward to meeting you there.

The Sun Doesn’t Shine on Me

I snorted in my coffee when I first read Jonathan Schwartz’s The Death of the Cold Call where he had pulled Marc Andreesen up on stage at Sun’s annual Analyst Conference and said that he found it “interesting … that no one from Sun had ever paid them [Ning] a visit. No one had ever made a sales call, cold or otherwise.”

Why did I snort you ask?

Because I can’t seem to be able to buy any Sun servers.

I’ve tried.

I’ve been trying since July of 2005.

So let me tackle this in two parts.

Part one, we’ll roll back to July of 2005 when I, a person coming from a “you buy Dell desktops and laptops when you think you are saving money from buying Macs but all the servers are Suns, Compaqs, and SGIs” world, attempted to buy about 50ish servers and terabytes of storage from Sun or HP.

Part two, we’ll talk about my recent adventures that started immediately after reading “The Death of the Cold Call”.

Part One: The Mission was Servers

It was a multi-pronged approach, several of us contacted in-house sales reps for Sun, HP, and Dell who we knew and had (in other capacities) bought lots and lots of stuff from (lots as in millions, tens of millions). All of them took it up and forwarded us into the system. The pipeline. The whatever. The word was that we’d be contacted soon.

Then I was going to go to all of the web sites and fill out the have-a-local-sales-rep-contact-me forms. I got around to that at about 1 a.m. and filled out the forms at,, and I put in succinct sentences about who we were, what we needed, when we needed it and said that I’d be available via phone starting at 7 a.m. PST.

Phone rings at 7 a.m.

It’s Dell.

Kid named Michael from the Small Business Unit (where I filled out the form; we actually had no “employees” then, so we fit in their “fewer than 1,000 employees” section). I’m told that we’re in the system, that there were two other entries about us (from the in-house reps we knew, which was nice), that those were going to be merged with my inquiry, and then after discussing the company and what we did, he said we’d be handled by the Emerging Business Unit and, locally, a preferred corporate sales account executive would contact us (because we weren’t the printers-and-desktops buying type but servers-servers-servers).

I say “Great!”, and go back to making my coffee. I pour a cup and the phone rings, it’s Steve the local Dell account executive, and thus began the process where we all came under the impression that Steve actually worked for us. In less than 2 weeks we had some chances to “pitch” the company to get into a certain pricing tier, had all the servers in the datacenter, and had managed to get it all leased based purely on the company’s financials (no personal guarantees). And honestly, 95% of all of the work was done by Steve. I felt like a Big Company. I felt like Steve worked for me.

Throughout that entire 2 weeks, I called Sun and HP every day with the idea that I’d use the non-confidential Dell materials to wedge a similar deal.

With Sun I never got past their 800-number staff (“Hmmm … no one has contacted you yet?”), and HP kept pushing me to local VARs and integrators, none of whom either returned my phone calls or called me after (I assume) getting the referral from HP.

See there’s a difference between the sales models of these companies: Dell predominantly does direct sales, and Sun and HP tend to go through resellers (unless you’re big, then you get a rep that sits in your office staring at you, waiting for your printers to run out of toner).

More than once though, the Sun people did suggest I just buy the servers off the web site. “OK,” I said, “and what credit card should I put that $220,000 order on?”. “You can put in multiple orders over a few days”, he suggested. “OK … thanks! I’ll try that,” I said while hanging up the phone.

The result?

That’s just half of the room.

Part Two: Second Chances

OK. OK. So we’re running more and more Solaris on those Dell boxes (actually we’ll be entirely Solaris soon). Then a major client of ours wants entire racks specced out with HP servers and storage: I go with the HP ProLiant BL35p blades. They’re nice because among many things they default to DC power and you get a 1 unit or 3 unit AC-DC converter for them to use AC. That is quite nice. The client that wants HP also spends millions with HP a month: they have an in-house sales rep (he’s the one milling around by the printers).

Thus I begin the process of forcing my business onto Sun and HP.

I fill out the web forms again.

I call HP first. I get a very nice Minnesotan lady on the phone.

The conversation goes like this:

NICE LADY: How many employees does your company have?

ME: We have 24, but I know where this goes, please realize that we’re quite efficient and buy a number of servers every year that’s comparable to companies a 100 times our size.

NICE LADY: Hmm … let me ask where to send you. (I’m on hold now, nice elevator music)

NICE LADY: Hi. Sorry for the wait, I have some numbers for some partners in your area. Do you have a pen? The first is Golden …

ME: No, no, sorry to cut you off and I don’t mean to be rude, but we’re not going through local resellers. We could just be integrators ourselves if that’s what we wanted to do. I’d like someone in direct sales and I’d like to talk to them today.

NICE LADY: I see. Can you hold again? Let me ask. (The elevator music returns.)

NICE LADY: Now you said you were going to get ‘blades’?

ME: (how this mattered?) Yes the BL35p blades in particular.

NICE LADY: OK, I’m going to put you in contact with a blade specialist and he can help you.

ME: OK, thank you. (I get forwarded to Adam’s voicemail)

ME: Hi Adam, this is Jason Hoffman from a company called Joyent, I need to discuss and get a series of quotes for 2, maybe 3, enclosures of BL35p blades and the new 1510i storage system with 2 trays of SCSI and 4 trays of SATA drives. If you could call me back today I’d appreciate it.

Not too bad. I’m at least in contact (well, voicemail) with a real person. Adam actually calls back within about an hour, we have a great discussion about the server side of it and he arranges a conference call with the storage specialist the following day. Also goes well. I’m impressed with their effort up to this point, it’s Dell-ish.

The problem?

I know what the client pays for HP, I know what I used to pay for HP, and being generous (meaning I understand that we’re not huge) I know that the quote is at least 30 percent over what it could be (I can even get better prices on their own web site in some cases). Stating this doesn’t get me anywhere, I actually get the impression (and I think it’s correct) that he simply doesn’t have the power to do it or that the internal effort it would take just isn’t worth it.

At least give me some love: charge me list on a server but cost on the hard drives. Give me an extra 4GB of RAM for $0 extra. Something.

I take that quote and go to CDW. I push. I get a couple of percent lower.

HP is basically dead at this point, except what’s bought via that client with the in-house rep.

Now Sun. Who by the way I started calling 6 weeks ago: Monday February 6, two days after “The Death of the Cold Call”.

I call the 800 number, and I get a nice guy on the phone:

ME: Hi, my name is Jason Hoffman and I’m with a company that’s based out of San Diego and San Francisco, I’d like to talk to someone in sales about the Sun Galaxy line of servers.

NICE SUN GUY: I can take your name, number and information and have someone call you.

ME: That’d be great. Will it be someone in Southern California?

NICE SUN GUY: I don’t know.


I didn’t hear from anyone for two more days. So I called back again and stated two things: that I had called before and that I had to buy 4 racks of servers ($1 million; which is quite overstated by the way but I had to see …) “ASAP” and “in Q1”.

2 days later. Not a word.

So thus began the “experiment” where I called them every day for two weeks asking to be in touch with someone in sales in Southern California. Someone who could put together a quote for an order of this size, tell me where to wire money or send a check, make sure it’s shipping et cetera.


I walked down the road where there’s a four-story Sun building filled with people and asked if there was anyone in “sales” that worked out of that building. Nope.

Then one day the phone rings, I answer, and it’s Sue from SUN! Whoo-hooo!! But she’s not calling because of my attempts to reach someone so that I can buy a “million dollars” worth of servers, she’s calling because many months earlier I had attempted to buy a CoolThreads server to test, they said they’d actually send it for free for 60 days (it was the first version of that special) but then after trying to even just buy one for a week, I gave up. Seriously I would have just bought one.

Sue was here to fix all that, and supposedly we’re getting a CoolThreads server. It’s been a month but whatever. We’ll see.

But I used this opportunity to really dig into Sue about Sun’s sales. Interesting stuff.

Then, finally, about two weeks ago, I made parallel proposals to the Sun sales rep and CDW (a good-sized Sun reseller that we buy our console servers, power strips, and lots of miscellaneous stuff from). I say exactly how much we spend at Dell on a model exactly comparable to a Sun Fire X4100. I say how I was actually surprised how relatively low the list price of the X4100 is for what it comes with. I also make the case that the 20 percent discount special that they’re offering for switching from HP, IBM or Dell would put us in the right territory, and reminded them that the stated educational discount is 37 percent. What’s different from an “emerging company” and people emerging from academics out into the Real Server Buying World? Other than that we’ll actually buy more of them?

Either of those would put us in a place where we might pay less than with Dell or we’d be at a place where I wouldn’t mind paying for an intangible like the irrational comfort of running Solaris on Sun hardware. I also said that if the price was right, I’d buy about 15 of them now and I’d begin the process of swapping out full racks of Dells for Sun Fire X4100s.

Where are we now?

I bought a lot more Dell servers and storage on Friday, and they’re already here. Of course I told them I was shopping around, and they respectfully played ball.

The Sun rep responded by putting me in contact with a local VAR. Who did call and leave a very rambling voicemail along with sending me an email. I still haven’t gotten a quote and I just get voicemail when I call every day. I gave up after a week. Then being able to get pricing on X4100s from CDW that’s competitive with Dell apparently requires the involvement of several management layers at both Sun and CDW. That’s still “in progress”. I’m not hopeful.

Sun and HP are in tight spots, it’s actually expensive to have in-house people on the ground everywhere, and the amount of dedication and drive those people must have can be difficult for a company to infuse and keep going. Especially when cost-cutting strategies involve pushing more and more sales through VARs/partners/integrators. There seems to be no one with the power to make smaller deals: meaning that we don’t buy servers one at a time and we don’t spend $40 million a year on servers, but we could spend half a million on servers and storage easily.

I guess I just wish they’d take me seriously.

Thank you for reading all the way to end, and now I’m off to have lunch in Orange County with our Dell account executive and tour a facility there.

Database Decisions Made Easy

Every few months someone will post on the Rails mailing list asking for advice on whether to use MySQL or PostgreSQL when building their Rails applications. At Joyent we use … both.

Strongspace uses MySQL for the application database, and to drive the authentication for scp and rsync. The Joyent Connector uses PostgreSQL for all its ActiveRecord models.

They’re both fast enough for the majority of projects, and have a pretty similar feature set. Plus, if you’re building with a database independent framework like Ruby on Rails or Hibernate it’ll only take you a few hours to switch between them.

I’ve used both databases and I’ve come to the realisation that you they’re basically interchangeable. So don’t worry about the tech geeks arguing about which one is ‘better’ or ‘faster’, odds are they’re both fine for your application.

Having said all that, just like the FeedLounge guys I prefer PostgreSQL, and it’s the database we’ll be using for all future applications here at Joyent.