"In the long run, it is cheaper to build and not buy," said Justin McWilliams, a software engineer in Google's corporate engineering department, which provisions and manages computers and other technology for Google employees. McWilliams shared some of the company's practices at the O'Reilly Open Source Conference (OSCON), being held this week in Portland, Oregon. "We typically don't default to buying a commercial offering. We think about building it from scratch first, or look to the open source world," he said.
Google uses a number of home-built or modified open source programs for IT management, including software for full disk encryption (FDE), remote computer management, compliance management, virtual private networks (VPN), video teleconferencing, and for single sign on (SSO).
Over the past few decades, IT departments at large organizations have learned to purchase commercial, off-the-shelf software to manage their infrastructure, typically because it is less expensive than writing and maintaining the software in-house. Due to a number of factors, however, this approach does not work well at Google, McWilliams explained.
"Even when we buy we still have to build on top of what we bought in order to be effective within Google. We want all of our systems to communicate with one another. Otherwise, we'd just have all these silos of data," McWilliams said. The cost of employing engineers to write and maintain code is still more cost-effective than maintaining costly support contracts with IT management software providers, McWilliams said.
One key reason behind this build-first philosophy is that Google is a rapidly growing company. The company currently has over 32,000 employees, almost twice as many as it did in 2008. Because of this rapid growth, the company's IT staff, which is not growing at the same pace, has to keep scalability in mind when setting up operations. "We have to find other ways to scale. We try to scale by building [in] automation and self-service, as opposed to just throwing more people at the problem," McWilliams said. Typically, use of commercial software can not scale at such a dramatic rates, at least not in an economically feasible way.
Like most organizations, machine management has been a challenge for Google. Google engineers get their choice of operating systems on their work machines, either Apple's OS X, Google's own Chrome, one of several distributions of Linux, or Microsoft Windows. The Apple machines in particular have been hard to manage, given the limited tools available from Apple and third parties for enterprises. By McWilliams estimate, Google has one of the largest corporate IT deployments of Macs in the world, with over 30,000 units now in use. "That creates a lot of challenges for us," McWilliams said.
To push patches and software updates to the Macs, Google initially used Puppet, . The organization quickly ran into scaling problems, however. It looked at commercial solutions, though most charge about $100 per machine per year. Additional Web servers, file servers and load balancers would also have to be deployed. "It would have cost us several million a year for the infrastructure and licensing," McWilliams said.
The company finally found an answer for its Mac support issues in open source software called , which was developed by an engineer at Walt Disney Animation Studios. McWilliams' team had deployed Munki on Google App Engine, which meant they did not have to manage any additional physical servers to run the software. "We have days where we are pushing out over six terabytes of traffic, or thousands and thousands of updates," he said.
Encrypting Macintosh disks was another task that Google tackled without the use of commercial software. OS X Lion 10.7 offered built-in FDE with a program called FileVault 2, but it had some issues for corporate users. For instance, the software doesn't force users to encrypt the disks, nor does it offer an escrow repository for storing keys, other than one provided by Apple itself. So Google developed its own software in-house, called (an anagram of the phrase "FileVault Escrow"), which provides a companywide escrow service.
"When a Googler forgets the password, an admin can fetch the recovery key, unlock the hard drive and reset the password," McWilliams said. As with Munki, Cauliflower Vest runs on the Google App Engine hosted service.
Another piece of open source software the corporate engineering uses is , which the company chose because "we could extend it and adopt it to our environment," McWilliams said. The company modified the software so that when employees log in, they can access internal Web sites without re-authenticating on those sites, thanks to a certificate VPN provided to the browser upon log-in. "With other proprietary software, we probably wouldn't have been able to do that," McWilliams said.
Google also has what is probably the world's largest civilian video conferencing network, which was built on top of . Even Google's telephony software is open source: The company uses for telephone routing for its call centers.
Interestingly, Google's approach to scalability is not limited to IT. According to McWilliams, the company also runs California's largest private fleet of buses, shuttles that covey employees to and from work. Each bus is equipped with WiFi, so "Googlers can remain productive while they are commuting," he said.
The IDG News Service