Updated April 7, 2011Doc Sheldon
This morning, someone tweeted about a live webcast where Facebook was going to make some big announcement. Envisioning a “Pinky and the Brain” moment, I tuned in and watched it. I’m really glad I did. They may well have revolutionized data center design and server design, simultaneously.
As a bit of background, when I worked as an electrical engineer, my specialty was the UPS (uninterruptible power supply) to keep the data center operating, at all costs. Downtime is costly, particularly for centers that lease time on their mainframes. I worked closely with other team members to design new facilities or expand existing centers (the former is MUCH to be preferred, from an engineer’s standpoint!)
Efficiency was usually the biggest selling point for any of the peripheral equipment. Just to give you an idea:
- A typical isolation transformer would lose at least 10% of its input power to inefficiency
- A typical UPS (in those days… we’re talking 30 years ago) lost 15-20% to inefficiency
- A PDU (power distribution unit) lost around 4-10 % to inefficiency.
What some people don’t realize is that when you look at the overall inefficiency of a system utilizing all three, which used to be very common, you had to multiply the effects. In other words, you multiply 90% x 85% x 95%, so you ended up with an overall efficiency of less than 73%. And you haven’t even gotten to the equipment that generates the most heat, yet.
And heat = efficiency loss, unless your goal was to heat something up. Trust me… in a data center, that is NOT the goal. More heat also caused severe problems with processors and disc drives, so environment control was critical.
Now, efficiencies in electrical distribution equipment have improved greatly over the last 30 years, but 15% losses in power distribution are still quite common. Jay Parks, of Facebook, used an overall inefficiency of a typical electrical distribution system of 11-17%, which is, IMO, probably not typical. I’d be inclined to say that most such systems today probably see closer to 18-20% loss.
Servers generate a LOT of heat! So much so that most such centers usually piped chilled water throughout their system, to compensate for it. Fifty to sixty gallons per minute of 50 degree F water in, that absorbed so much heat that it came back out at 75-80 degrees F, translates to a hell of a lot of heat being dissipated.
So developing a data center of Facebook’s scope that doesn’t need chilled water is an impressive feat, all by itself. Air cooled data centers exist, to be sure, but trust me again… not of the same scale. And they use TONS of air handlers to cool the room. Air handlers that have their own inefficiencies, by the way.
(click images to enlarge)
I used to pack up my Dranetz meter and go spend three or four days measuring the various losses throughout data centers, and it was very rare to find a large timeshare center that exceeded 55% efficiency. 52-53% was about average.
But through an end-to-end design effort, Amir Michael, Facebook’s Manager, Hardware Design, and his team managed to achieve some results that are certain to make anyone contemplating the construction or expansion of a sizable data center sit up and take notice.
Just a few of the notable achievements in Facebook’s new Prineville, OR data center:
- No air conditioning whatsoever in the entire center
- No air handling ductwork in the entire center
- No UPS system
- No PDUs
- 98% overall power distribution system efficiency, vs. the more common 83-89%
- A PUE (power usage effectiveness) factor of 1.07, vs. an industry average of around 1.5
Add to that their visionary design of a new server for their own use, which is 38% more efficient and 24% less costly to build than what is currently available on the market, and you have a recipe for success.
Amir just joined Facebook a couple of years ago, and reportedly, one of his first assignments was to pursue this project. At that time, I doubt anyone, including Amir, had any idea how much impact this could have on the company’s bottom line. The dollar savings on electrical power alone for the Prineville center could easily be in the 9-10 digits (Google’s Oregon facility was reported to be 5,300 megawatts. At about $1MM per year per megawatt, that’s 5.3 billion dollars. Reduce that by around 30%, and Facebook could quickly find itself in a very nice position, cashflow-wise.
As the title points out, I’m not a Facebook fan-boy. I’ve lashed out at Facebook in general and Mark Zuckerberg in particular in the past. But when credit is deserved, I’ll give it cheerfully. To Amir Michael and his team, to Frank Frankovsky, who spearheaded what must have been a monumentally complex supply chain challenge, Jonathan Heiliger, who surely walked the talk and even Mark, who was willing to support such an effort… kudos to ya, lads & lasses. I tip my hat to you all.
By the way, did I mention that Facebook also made available all their design notes, specifications and calculations on Open Compute? I’ll toss ‘em even more kudos for that!
Amir, you may call your server design “vanity-free”, but you can be awfully proud of yourself and your team!