Jason A. Hoffman

What I would want a “cloud” to do for me: a functional view

Posted on June 25, 2009 4 Comments

I was on a panel at Enterprise 2.0 yesterday about “Cloud Computing” providers and wanted to take the entire definite of “what is cloud computing?” from a different perspective.

As a consumer, I fundamentally want the entire technology stack for an application to

Just work
Just scale
Just tell me everything

Just works means that I tell an infrastructure that I want my application to have user experience X 99.99% of the time to 99.99% of the my users, and what I want back from that query is a dollar amount. I’ll then make a decision about what I can afford.

Just scales means that I can go from zero to millions of users, one geographical site to twenty geographical sites, and one continent to four continents without a rewrite/rearchitecting.

Just tells me everything means that an API and data source is not only about provisioning machine images or the like, it can be queried for real time and time series data about everything. It let’s me make decisions around latencies and nothing ever just happened for no reason.

When you think about what it would take for these things to come to fruition, it starts to feel like a real step forward in the industry. It’s the collection of these things (in existence or development) that I don’t mind giving a new term to.

Cloud Computing is actually Transparent Computing: there’s shouldn’t be anything cloudy about it

Posted on June 15, 2009 1 Comment

A little over a year ago Joyent was identified in a Forrester report as one of ten “Cloud Computing” companies to watch. The report was really the beginning of “Cloud Computing” being applied to those of us doing infrastructure and I think marks the point in time when the term really began to appear in marketing materials.

Our inclusion on that list was notable because we were the only privately held company on the list that operates their own server farms and develops their own software stack to instantiate and manage it all.

Since then it’s been cloud cloud cloud cloud cloud cloud cloud cloud and I recall the first time I noticed it showing up as an adjective, I overheard someone actually ask “Is your application cloudy yet?”.

Does one really want a “cloudy” application?

Our premise here has always been that an application should “Just Work” and when it does not, then we should have the means to quickly figure out why and correct it. Importantly, one should be able to figure out the issue(s) within the context of the entire architecture.

The general “gut feeling” though is that something is changing in our industry, and I would say that like how “the network” was flattened, standardized and made into a utility, what people are talking about as cloud computing is the next wave of making data and compute available as utilities or even if entirely owned, one conceptualizes and runs it as a utility. Well that’s part of it. For now, we’re all just calling it Cloud Computing.

Like anything that is a utility, there are initial concerns that fundamentally boil down to trust.

Trust does not emerge from things that are cloudy (opaque). It emerges from complete transparency and the ability to determine what’s is happening or has happened. Trust comes from letting an infrastructure know what’s important to you. I want to query an infrastructure with “This application needs to be available within 1 second to 99.99% of my users 99.99% of the time. How much is that and can you do it?” or perhaps “This 246GB of data you’re about to handle is ‘level 3’ data in our privacy policies. How much is that and can you do it?”.

The hallmark of this “Cloud Computing” needs to be complete transparency, instrumentability and while making it certain that applications just work, the interesting aspects of future APIs aren’t provisioning and self-management of machine images, it’s about stating policies and being able to make decisions that matter to my business.

Part 3, On Joyent and Accelerators as Cloud Computing “Primitives”

Posted on April 6, 2009

In the last part of this series we ended by talking about 6 “simple” utilities that software uses on “servers”. They were

1) CPU space
2) Memory space
3) Disc space
4) Memory bus IO
5) Disc IO
6) Network IO

Along with their natural minimums (zero) and maximums.

Providing compute units that do these utilities

What we’ve always wanted to do at Joyent was provide “scalable network appliances”: online servers that just worked for given functions and were capable of both handling bursts and serving as logical lego-like building blocks for new and legacy architectures. Sometimes these appliances might contain our own software, sometimes not. They would be on a network where it would be difficult for a given piece of software to saturate the immediate parts of it.

For most workloads, a ratio of 1 CPU:4GB:2 spindles works pretty well. The faster the CPU (constrained by power) the better, memory is well … memory, and the size and speed of those spindles can be varied depending on what a node is going to be doing (one end of a workload possibility or the other). In other words, 1) CPU space, 2) Memory space and 3) Disc space aren’t terribly interesting or difficult to schedule and manage in most environments (in some ways, they’re a purchasing decision), and 4) Memory bus IO is set in silicon stone by their creators.

Which ones matter?

We’re left with disc and network IO as being the key utilities and the ability to move things on-and-off disc and in-and-out of the network comes from using more CPU. So when an application experiences a surge in activity, it typically results in more use of the CPU, disc IO and network IO. We decided the best approach was to put CPU and disc IO on a fair-share system, and standardize on an overbuilt network (10 Gbps interconnects, multi-gigabit out of the back of each physical node). Our physical servers exclusively use intel NICs in a PCIe format. This way we can have multiple gigabits per node, physical segregation and failover as options, and we can standardize on a network driver and get away from on-board NICs changing from server to server.

The fair share system overall is supposed to provide for short-term (could just be minutes) bursting needed to handle spikes in demand. Spikes that occur too fast for either hardware upgrades or even to spin up new VMs on most systems (and are often too fast to be noticed in 5-10 minute monitoring pings). This allows for you to stop thinking about the servers that you pay for as being these constrained maximums, and start thinking about a “1 CPU, 4GB of RAM” server as being a guaranteed minimum allotment.

This was at least why we stopped calling them “Grid Containers” and started calling them “Accelerators”.

Moving up the stack

The next installments are going to be talking about some kernel and userland experiences and choices, our desire to use ever more “dense” hardware, 32 and 64 bit environments, and rapid reboot and recovery times.

Part 2, On Joyent and Cloud Computing “Primitives”

Posted on March 31, 2009 2 Comments

In the first part of this series I made a key list of some of the underlying ideas at Joyent, that we believe that a company or even a small development team should be able to:

Participate in a multi-tenant service
Have your own instantiations of this service
Install (and “buy”) the software to run on your own infrastructure
Get software and APIs from Joyent that allows for the integration of all of these based on business desires and policies.

And said

The successful future “clouds” have to be more accessible, easier to use and to operate, and every single part of the infrastructure has to be addressable via software, has to be capable of being introspected into and instrumented by software and this addressability means that one can write policies around access, performance, privacy, security and integrity. For example, most of our customer really don’t care about the details, they care in knowing what is capable of providing 99.99% of their end users some great experience 99.99% of the time. These concepts have to be bake in.

I continue to think that from a developer’s perspective the future is closer to the SMART platform where Ramin’s comment on an older Joyeur article about EC2 versus Accelerators is relevant, let me quote him:

Whoever has the fewest number of steps and the fastest build/deploy time is likely to attract the most developers. Whoever can show that the operating cost scales linearly with use will have developers casting flower petals in their path 🙂

As an app developer, I don’t care that it runs on Solaris, FreeBSD, or Mac-OS. I want it to work. I want an optimized deployment workflow and a simple way to monitor and keep things running.

That all said.

In the second part to this series I wanted to start talking about “primitives”. I’m saying “start” because we’re going to be going to be covering primitives over the next couple of posts.

I’m going to loosely define “Primitives” (now with a capital P) as of all the stuff underneath your application, your language and the specific software you’re using to store your data. So yes, we’re talking about hardware and the software that runs that hardware. Even though most Primitives are supposed to eventually be hidden from a developer they’re generally important to the business people and those that have to evaluate a technology platform. They are important parts of the architecture when one is talking about “access, performance, privacy, security and integrity”.

Previously, I’ve talked about a bit about Accelerators ( On Accelerators) and that fundamentally we deal with 6 utilities in cloud computing.

The fermions are the utilities where things take up space

1) CPU space
2) Memory space
3) Disc space

The bosons are the utilities where things are moving through space and time

4) Memory bus IO
5) Disc IO
6) Network IO

All of these utilities have physical maximums dictated by the hardware, and they have a limit I’d like to call How-Likely-Are-You-To-Do-This-From-One-Machine-Or-Even-At-All.

I’ll admit at this point of a particular way of thinking. I think “what is the thing?”, “how it is going to behave?”, “what are the minimums and maximums of this behavior?” and finally “why?”.

The minimum for us is easy. It’s zero. Software using 0% of the CPUs, 0 GB of memory, doing 0 MB/sec of disc IO and 0 Gbps of network traffic.

The maximums:

Commercially available CPUs typically top out in the 3s of Ghz
“Normal” servers typically have <128 GB of memory in them and the ratio of 4GB of memory per CPU core is a common one from HPC (we use this and it would mean that a 128 GB system would have 32 cores)
Drives are available up to a terabyte in size but as they get larger you’re making performance trade-offs. And while you can get single namespaces into the petabyte range, even though ones >100 TB are still irritating to manage (for either the increased fragility of a larger and larger “space”, or the variation in latencies between a lot of independent “storage nodes”).
CPUs and memory talk at speeds set by the chip and hardware manufacturers. Numbers like 24 Gbps are common.
Disc IO can be in the Gbps without much of an issue
For a 125 kb page with 20 objects on it, 1 Gbps of traffic will give you 122,400,000 unique page views per day and that in a 30 day month this is 3,672,000,000 page views (Check my math). Depending on how much stuff you have going on, this basically puts you in as a top 100 web property. With the number of public website is ~200 million (source), being in the top 200 is what … 0.00001% of the sites?

As something to think about and as an anchor, I remember seeing a benchmark of a “Thumper JBOD” attached to a system capable of saturating the 4×10Gbps NIC cards in the back of it. Yes the software was special, yes it was in C, and yes it was written with the explicit purpose of pushing that much data off of discs; however, think about that for a minute.

Imagine having a web property doing 120 billion monthly page views coming off of a single “system” that you can buy for a reasonable price. Starting from there, expand that architecture and I wonder with the “right software” and “primitives” where you would end up. If we change it from a web property to a gaming or a trading application, where would you end up? What is the taxonomy of applications out there (common and uncommon) and do we come up with the best architectures for each branch and leaf on that tree?

Please think about that anchor and a taxonomy for a few days and then I’m going to get into some of the key differentiators of our Primitives and answer some of the “Why?”.

On Joyent and “Cloud Computing”, Part 1 of Many

Posted on March 17, 2009 6 Comments

With this post, I’m starting a series where we’re going to be much more explicit about what we’re thinking, what we’re doing, how we’re doing it and where we’re going. I’m not interested in any of it being thought of as impersonal “marketing material”, so I hope you’ll allow the occasional use of “I” and for the interweaving of my perspectives and perhaps a story or two. As the CTO and one of the founders whose lives has been this company, I’m going to go ahead, be bold and impose.

In May of this year, we’ll have been in business for five years, and looking back at it, it’s amazing that we’re still here. Looking back at it, it makes complete sense that we’re still here. Our idea was pretty simple: let’s take what it normally BIG infrastructure (i.e. 64GB of RAM, 16 CPU servers, 10 Gbps networking), virtualize the entire vertical stack and then go to people who normally buy some servers, some switches and some storage, and sell them complete architectures. All done and ready to go.

It is commonly said that businesses only exist out of necessity, that the best ones take away or care of “pain”. Makes sense, doctors only exist because there is disease.

It is a pain for most companies (“Enterprises”) to expertly own and operate infrastructure that can scale up or down depending on their business needs. Infrastructure is more a garden then a monument but rarely treated as one.

It is difficult to be innovative in the absence of observing tens of thousands of different applications doing tens of thousands of different things.

It is easy to make the same mistakes that others have made.

That final point is an important one, by the way. I’ll tell you why in a slightly roundabout way.

I’m a scientist by training and have always been a consumer of “large compute” and “big storage” not just a developer of them (I can write about why that is a good thing another time). While at UCLA as a younger man, I was lucky enough to have had a graduate physiology course where one of the lecturers was Jared Diamond and later on I went to an early reading of one of his books that was coming out, Guns, Germs, and Steel. What reached out and permanently imprinted itself in my mind from that book was the Anna Karenina principle. It comes from the Tolstoy quote, “Happy families are all alike; every unhappy family is unhappy in its own way.”, and simply put it means that success is purely the absence of failure.

Success is purely the absence of failure.

Education and “products” are meant to save people and enterprises from mistakes. Both the avoidance of “known” mistakes (education) and the surviving of “unknown” mistakes (luck, will power, hustle, et al).

Internally I commonly say,

It’s fine to make mistakes as long as 1) no one has ever made that mistake before, if they have, you need to question your education or get some, 2) we’re not going to be repeating this mistake, if we do, we need to question our processes, our metrics and how we communicate and educate and 3) the mistake does not kill us

This is all important because I believe the primary interest in “cloud computing” is not cost savings, it’s in the participation of a larger network of companies and people like you. It is the consumption of products that

Are more accessible and easy to use (usually from a developers perspective)
Are easier to operate

However, that’s not it. I can say with complete certainty that main barrier towards adoption is trust. Trust specifically around privacy, security and integrity.

The successful future “clouds” have to be more accessible, easier to use and to operate, and every single part of the infrastructure has to be addressable via software, has to be capable of being introspected into and instrumented by software and this addressability means that one can write policies around access, performance, privacy, security and integrity. For example, most of our customer really don’t care about the details, they care in knowing what is capable of providing 99.99% of their end users some great experience 99.99% of the time. These concepts have to be bake in.

These are some of the underlying ideas behind everything that we’re doing. At Joyent, we believe that a company or even a small development team should be able to

Participate in a multi-tenant service
Have your own instantiations of this service
Install (and “buy”) the software to run on your own infrastructure
Get software and APIs from Joyent that allows for the integration of all of these based on business desires and policies.

Sometimes people refer to this as the “public and private cloud”. Oooook.

I happen to believe we’re capable of doing this quite well compared to most of the large players out there. We own and operate, and we’re making those APIs and software available (often open source).

Amazon Web Services is a competent owner and operator and allows you to participate in a multi-tenant service like S3, but there are no API extensions for integrity, you cannot get your own S3 separate from other people, you cannot install “S3” in your own datacenter, and of course there is no software that allows an object to move amongst these choices based on your policies.

Question: “If you have a petabyte in S3, how do you know it’s all there without downloading it all?”, I can answer that question if I had a petabyte on Netapps in-house.

The Cisco, Sun, HPs and IBMs (I’ll toss in the AT&Ts too) of the world want to sell you more co-lo, perhaps something best called co-habitation, more hardware, more software, less innovation and no ease of operating.

Larry Ellison says that this is all a marketing term. I think he’s wrong. We think he’s wrong, and that Joyent seems to be unique in focusing on making the above a reality.

Next week, I’ll be discussing “primitives” and “architectures”.

A Loving Cloud

Posted on June 27, 2008 1 Comment

Yesterday several members of Joyent’s team attended Structure ’08. Jason Hoffman was on a panel that produced some interesting debate about whether clouds should aim to be open. The story was even picked up by the Wall Street Journal’s Don Clark in an article entitled Finding A Friendly Cloud

Jason Hoffman, founder and chief technology officer of a cloud-computing specialist called Joyent, was particularly pointed in warning that Google’s App Engine could represent a lock-in to developers. It is possible to build “a loving cloud,” he argued, that would make it easier to create applications that could be easily moved among different services. Other panelists kept calling Google’s App Engine “proprietary,” which to many techies is equivalent to labeling it both evil and outdated at the same time.

Here’s video of the exchange Jason had with Google’s Christophe Bisciglia

1 Billion Page Views a Month

Posted on June 24, 2008 2 Comments

Here’s a video detailing how LinkedIn built an application (Bumpersticker on the Facebook platform) using Rails (and C Ruby!) that serves up more than 1 billion page views a month.

In my opinion, this ends the debate about whether Rails scales. Rails is a component, it is how the components are architected and delivered that comprises the magic. LinkedIn did amazing work taking advantage of Joyent’s technology stack including innovative ways of leveraging Joyent Accelerators and our BigIP load balancers.

Congratulations to the LinkedIn team. Great accomplishment! You can read more about the how LinkedIn scaled bumpersticker on Joyent in a post on their blog entitled Web Scalability Practices: Bumper Sticker on Rails

View the video.

Update: ZDNet writes about the story.

Amazon Web Services or Joyent Accelerators: Reprise

Posted on May 15, 2008 9 Comments

In the Fall of 2006, I wrote a piece On Grids, the Ambitions of Amazon and Joyent, and followed up with Why EC2 isn’t yet a platform for ‘normal’ web applications and the recognition that When you’re really pushing traffic, Amazon S3 is more expensive than a CDN.

The point of these previous articles was to put what wasn’t yet called “cloud computing” into some perspective and to contrast what Amazon was doing with what we were doing. I ventured that EC2 is fine when you’re doing batch, parallel things on data that’s sitting in S3, and that S3 is economically fine as long as you’re not externally interacting with that data to a significant degree (then the request pricing kicks in). Basically it is incorrect that each are universally applicable to all problems and goals in computing, and that they’re cost-effective. An example of a good use case is a spidering application: one launches a number of EC2 instances, crawls a bunch of sites, puts that information into S3, and then launches a number of EC2 instances to build an index of that data and further store it on S3.

Beyond point-by-point features and cost differences, I believe there are inherent philosophical, technical and directional differences between Joyent and Amazon Web Services. This is and has been our core business, and it’s a business model, in my opinion, that competes directly with hardware vendors and customer taking direct possession of hardware and racking-and-stacking it in their own datacenters.

Cloud computing is meant to be inherently “better” than what most people can do themselves.

What’s changed with S3 and EC2 since these articles?

For S3? Nothing really. There are some additional data “silo” services now. SimpleDB is out and there has been some updates to SQS, but I would say that S3 is by far the more popular of the three. The reason is simple: it’s still possible for people to do silly things when storing files on a filesystem (like put a million directories in one directory), but it’s more difficult to do things as silly with a relational database (you still can, but they’re ultimately handled within the RDMS itself, for example, bad queries).

I’m consistently amazed by how many times I have to go over the idea of hashed directory storage.

For EC2 there’s been some improvements.

Annotating the list from “Why EC2 isn’t yet a platform for “normal” web applications we get:

1. No IP address persistence. EC2 now NATs and EC2 instances are on a private network. That helps. Are you able to get permanently assigned, VLAN’ed network address space? It’s not clear to me.

2. No block storage persistence. There is now an option to mount persistent storage in a “normal” way. Presumably it’s block storage over iSCSI (there’s not many options for doing this), hopefully it’s not a formalized FUSE to S3. We’ll see how this holds up performance-wise, now there’s a bit more predictability in data stored in EC2 but experience has shown me that it only takes one really busy database to tap out storage that’s supposed to be serving 10-100 customers. Scaling I/O is still non-trivial.

3. No opportunity for hardware-based load balancing. This is still the case.

4. No vertical scaling (you get a 1.7Ghz CPU and 1 GB of RAM, that’s it). There are now larger instances but the numbers are still odd. 7.5GB of RAM? I like powers of 2 and 10 (so does computer science).

5 & 6. Creation and handling of AMIs. Experience like this is still quite common, it seems.

Structure of modern applications

The three tiers of “web”, “application” and “database” are long dead.

Applications that have to serve data out (versus just pulling in like the spidering example earlier) are now typically structured like: Load Balancers/Application Switches (I prefer the second term) <-> Cache <-> Application <-> Cache <-> Data. Web and gaming applications are exhibiting similar structures. The caching tiers are optional and either can exist as a piece of middleware or as part of the one of the sandwiching tiers. For example, you might cache as part of the application, or in memcached, or you might just be using the query cache in the database itself. And while there are tiers, there are also silos that exist under their own namespaces. You don’t store static files in a relational database, your static assets are CDN’ed and served from e.g. assets[1-4].yourdomain.com, the dynamic sites from yourdomain.com and users logged-in at login.yourdomain.com. Those are different silos.

How to scale each part and why do people have problems in the first place?

Each tier either has state or not. Web applications are over HTTP, an inherently stateless protocol. So as long as one doesn’t introduce state into the application, the application layer is stateless and “easy” to horizontally scale. However, since one is limited in the number of IP addresses one can use to get to the application, and network latency will have an impact at a point, the “front” has state. Finally, the back-end data stores have state, by definition. We end up with: stateful front (Network) <-> stateless middle <-> stateful back. So our options for scaling would be: Load Balancers/Application Switches/Networking (Vertical) <-> Cache (Horizontal or Vertical) <-> Application (Horizontal) <-> Cache (Horizontal or Vertical) <-> Data (Vertical).

The limit to horizontal scale is the network and its latency. For example, you can horizontally scale out multi-master MySQL nodes (with a small and consistent dataset), but you’ll reach a point (somewhere in the 10-20 node range on a gigabit network) where latency now significantly impacts replication time around that ring.

Developing and scaling a “web” application means that you (or someone) has to deal with networking and data management (and different types of data for that matter) if you want to be cost-effective and scalable.

The approach one takes through this stack matters: platform directions

With the view above you can see the different approaches one can take to provide a platform. Amazon started with data stores, made them accessible via APIs, offered an accessible batch compute service on top of those data stores, introduced some predictability into the compute service (by offering some normal persistence), and has yet to deal with load-balancing and traffic-direction as a service. Basically they started with the back and should be working their way to the front.

At Joyent, we had different customers, customers making the choice between staying with their own hardware, or running on Joyent Accelerators. We started with the front (great networking, application switching), persistence, we let people keep their normal backends (and made them fast) and we are working for better solutions (horizontal) for data stores. Solving data storage needs weren’t as pressing because many were already wedded to a solution like MySQL or Oracle. An example of solving problems at the outermost edge of the network would be the article, The wonders of fbref and irules serving pages from Facebook’s cache. This is an example of programming in application switches to offload 5 pages responsible for 80% of an application’s traffic.

Joyent product progression is the opposite of AWS’s. We solved load-based scale with a platform that starts with great networking, well performing Accelerators, Accelerators that are more focused to do particular tasks (e.g. a MySQL cluster). We are working on data distribution for geographic scale, and making it all easier to use and more transparent (solve the final “scale”, administrative scale).

The technology stack of choice does matter: platform technology choices

Joyent Accelerators are uniquely built on the three pillars of Solaris: ZFS, DTrace and Zones. This trio is currently only present in OpenSolaris. What you put on metal is your core “operating system”. Period. Even if you call it a hypervisor, it’s basically an OS that’s running other operating systems. We put a solid kernel on our hardware.

Accelerators are meant to be inherently more performant then a XEN-based EC2 instance per unit of hardware, and to do so within normal ratios: 1 CPU/4GB RAM, utilities available in 1,2,4,8,16,32,64 gb sized chunks. The uniqueness of DTrace adds unparalleled observability, it makes it possible for us to figure out exactly what’s going on in kernel and userland and act upon it for customers in production.

ZFS lets us wrap each accelerator in a portable dataset, and as we’ve stated many times before, it makes any “server” a “storage appliance”.

Add to this Joyent’s use of f5 BigIP load-balancers, Force10 networking fabric, and dual-processor, quad-core, 32GB RAM servers.

Open and portable: platform philosophy

At Joyent, I don’t see us having an interest in running large, monolithic “services” for production applications and services. Things need to remain modular, and breakage in a given part needs to have zero to minimal impact on customers. Production applications shouldn’t use a service like S3 to serve files, they should have access to software with the same functionality and being able to run it on their own set of Accelerators.

We want software that powers services to be open, available, and enable you to run it yourself here on Accelerators, or actually anywhere you want. We develop applications ourselves exactly like you do, we tend to open source them and this is exactly what we would want from a “vendor”. This route also minimizes request (“tick”) pricing. We don’t want to entirely replace people choices in databases, instead Accelerators have to be made to be a powerful, functional base unit for them. Want to run MySQL, PostgreSQL, Oracle, J-EAI/ejabberd, … then by all means do that. No vendor lock-in.

For both platforms, we have our work cut out for us.

Rod Boothby Talks Cloud Computing at Web 2.0 Expo

Posted on April 24, 2008

Rod and I also held several interviews at the show – including Tim O’Reilly, Kevin Marks, Steve Gillmore, Dan Farber, and Rafe Needleman trying to get public opinion on ‘What is Cloud Computing’. We will be posting that video shortly as there is some good stuff in there.

Disclosure: That handsome fella interviewing him happens to be my husband, Chris Heuer, whose agency produced the Blogtropolus bloggers lounge at at the Web 2.0 Expo.

« 1 2 3 4 5 … 7 »