What will your keynote at be about?
I'm going to talk about the "scarcity mindset", how it holds us back, and how adopting an "abundancy mindset" would free us to think more creatively about IT. Certain things are scarce but when they become plentiful our thought processes and IT policies don't change fast enough. For example, disk space used to be very scarce and expensive but isn't any more (comparatively). However, many organisations have policies oriented around economics that haven't existed for years. There is very little reason to not use a disk mirror (RAID 1) for the boot disk of all servers, for example. Similar stories can be told about CPU, networking, and other technologies such as software. Free and open source software is another example (of a shifting landscape). The ability to have a new feature added to software was previously a scarce resource, something vendors doled out grudgingly, but with the free and open source software movement, we can adapt software in ways never thought previously. This will be my first time attending linux.conf.au and I'm really looking forward to it. While my talk will bend towards system administration, it will be broad enough that all developers should appreciate what I have to say.
What are some prime examples of ways that business can now adapt software in ways never thought previously?
People are adapting Linux to run on cell phones (Google's project Android gets the most press). That isn't feasible with a proprietary operating system purely for non-technical reasons. I suppose a room full of negotiators, lawyers, and stacks of cash would make it possible, but most developers don't have that. Taking that one step further, there is a research project (by Jennifer Wong, Dept of Computer Science, University of Victoria (in Canada): http://www.google.com/events/scalability_seattle/) where they are using Andoid as the operating system for a hand-held device for scuba divers. That's an adaptation of an adaptation! With proprietary systems these kinds of opportunities are missed because of the higher barrier to entry. When you have a "market of one" (like the scuba project), the old business justification routine doesn't always work. If a proprietary company had been approached about a one-off use of their operating system for scuba experiments, it wouldn't be considered without risk assessments, a marketing strategy and so on. And the answer would be, "No, this isn't worth our time." Having a frictionless option to make your own changes is very powerful. The ability to adapt software to make it the perfect fit for your organisation can be the differentiator that makes your operations stand out and impresses customers.
And what are the main things still holding business back from adopting free and open source software?
I think there are open source alternatives that are out there that haven't been broadly adopted because of a lack of PR. SugarCRM and Asterisk come to mind. That improves over time.
What are the favourite and least favourite parts of your job as a systems administrator for Google?
What I love about being a system administrator is automating things so that they take less time. Whether I have automated something that saves time for myself, my team, or my users I always feel like doing a "victory lap" every time I see my code go into production. The least favourite part of my job? I never enjoy writing budgets and budget proposals. I find it tiresome. It's important, but it puts me to sleep.
What is your favourite language to work with?
My favourite language is Perl, but I'm learning Python because it seems to encourage better, cleaner, coding practices.
Why Perl?
Perl is very expressive for system administration-type work. There are modules that let it connect to services such as LDAP and ActiveDirectory, and it can parse any kind of log file. In a very short amount of time I can throw together a quick Perl script that solves an ad hoc problem. It is also suited for writing larger systems, including Web-based applications. I've been using Perl since version 4.032 in 1991.
You took leave from a job at Lumeta to volunteer in the campaign of Howard Dean. Were you focusing on using technology and Internet methods in the campaigning?
I joined the Howard Dean campaign because I agreed with his political policies on health care and the war in Iraq. Earlier that year my father passed away after a long battle with cancer. During my last days with him we had many conversations about life the universe and everything. Though he was very weak, he would become very animated when talking about the importance of stopping the war. He was passionate about it. After he passed I learned about Howard Dean, his policies, and that he was doing interesting things with using internet technology to gather grassroots support for his campaign. I did join the campaign as a full-time volunteer, but I joined very late, so I wasn't there for some of their big innovations but I was glad to be there to help where I could.
What was one of your projects in the campaign?
One of my big projects was to help address the thousands of emails coming into the campaign every day. Volunteers were trying to sort through them all and barely keeping up. I was able to eliminate a large percentage of emails by removing spam and bounces. We set up better processes for managing the workflow of the volunteers and gave them some new tools and filters. Ironically I was introduced to the project because of my experience with help desk ticket software but the first thing I realised was that the amount of email they were receiving was larger than any FOSS (free and open source software) ticket system could handle. Therefore, I had to be creative.
Did you work on the campaign for Barack Obama as well?
Just the last three days of the campaign. I went to Pennsylvania where by day I walked door-to-door as part of the get out the vote effort, and at night I volunteered at the local Obama office. When I arrived the office manager was so overloaded she didn't have time to give me something to do. She was flooded with phone calls and people coming to her with questions. After listening to her answer questions for about 15 minutes I learned enough to answer about 50 percent of the questions that were coming in. The rest I tried to answer by doing Web searches from my laptop. I was deflecting enough people from interrupting her that she had enough spare cycles to give me something to do. Finally she asked me, 'Who are you and where did you come from?'
You are a self-proclaimed internet activist. Are you a fan or a critic of hacktivism, and have you used such strategies?
I don't support hacktivism that includes illegal activity. What I do support is geeks getting involved in social justice so as to make the world a better place. There are charities and service organisations everywhere that would benefit greatly from a weekly visit by a geek to do basic PC maintenance like making sure all Windows systems have working anti-virus software, backing up critical data, or helping replace an insanely manual process with a simple MS-Excel macro. These things seem minor to the average geek, but are incredibly valuable to such organisations. Helping set up Drupal or another content management system (CMS) can help such an organisation go from having a boring Web site with nearly no updates to an exciting Web site that can make them self-sufficient for routine updates.
Your published books include Time Management for System Administrators, The Complete April Fools RFCs and The Practice of System and Network Administration (second edition). Which book was most popular and what makes these books different from most other technical books?
The Practice of System and Network Administration and Time Management for System Administrators are both popular. I think they are both popular because they are self-help books, which help you both emotionally and technically. Usually technical books are tutorials (step by step instructions) or dictionaries (lists of commands and features). The Practice of System and Network Administration doesn't tell you what command to type to run backups, it explains why we do backups, that there are three basic scenarios that require backups, and how each scenario requires different planning. It gives readers the framework to decide what to do no matter what operating system they encounter.
Time Management for System Administrators gives a lot of help and encouragement on the emotional level. We sys admins are often beat up (metaphorically) by too much work to do, demanding users, and unappreciative management. This book isn't just about how to manage time. It gives the reader techniques that act as a defence against the stress and strain of the system administration lifestyle. My goal of the book is to help system administrators enjoy work more, get better yearly evaluations (and raises), all the while only working 40 hours each week. I'm most proud of the fact that I've gotten email from the spouses of readers telling me "thank you for giving my spouse back!"
Do you have another book in the working at the moment, or one that you want to write in the near future?
As part of my own time management I have sworn off writing any new books for a while. If I have a book idea, I post it to my blog and hope someone else writes it. I'd like to see the "design patterns" concept of software engineering applied to system administration.
You were the joint recipient, along with co-author Christina J. Hogan, of USENIX/SAGE's 2005 Outstanding Achievement Award for The Practice of System and Network Administration. Why do you think the book gained this level of attention?
You'll have to ask the committee! I like to think that in part SAGE was awarding our efforts in emphasising the "soft skills" of system administration, such as time management, interpersonal communication, and teamwork. When I was first involved in SAGE there was much more focus on technical matters (bits and bytes) and we were the oddballs for wanting to emphasise soft skills. I like to think that we were a small part of making that change in attitude.
SAGE AU says on its Website that "in a very short period of time, computers have become fundamental to the organisation of societies world-wide; entrenched at every level of human communication from government to the most personal. Computer systems are not simply constructions of hardware - rather, they are generated from an intricate interrelationship between administrators, users, employers, other network sites, and the providers of software, hardware, and national and international communication networks." If we go with this definition of a computer system, being a systems administrator must be one hell of a task. Is it?
It is! Y2K was a watershed event for IT because it woke up the world to how pervasive computers had become. CEOs suddenly realised that all aspects of their operations were being driven by IT systems. Previously IT was a "cost" to be "minimised" and/or ignored overhead. Now they realise that IT is an enabler and the source of innovation that needs investment, care and priority. People that thought e-commerce might augment their normal sales operations now understand that e-commerce is their sales operations (their human sales force is just a special user). Previously, when invoices were processed daily or weekly, the system could be down for a day and nobody would notice. Nowadays people submit an order and think something is wrong if they don't get confirmation email a few seconds later. Plus, they want to know the shipper's tracking number before the box has even shipped!
On the other hand, personal technology is getting smaller and more pervasive, and more connected. All the major video games now have some kind of networked component. That means you can't even blow up aliens or care for your Webkinz if there aren't system administrators designing and operating the networks and servers that make it all work.
Briefly, what are the main, and I imagine wide-ranging, challenges for contemporary system administrators?
There are two main emergent challenges. First, is operational efficiency. This means thinking about how can we measure what we do, so that we know what we are actually doing, and improve it over time. Monitoring and measurement becomes important when there are more devices than we can count on our fingers. I see many "green computing" articles today but the ones that have credibility include hard numbers, measured in the real world (not test labs or simulations or theories), that let me be the judge of just how efficient a product is as compared to what I'm doing today. We can make guesses about operational efficiency by looking at costs, but that is often an approximation: I worked at one place that considered electricity to be "free" because some other department paid for it. I think most companies don't have financial systems that give the proper granularity to measure power efficiency, nor efficiencies of labour, time or customer satisfaction. Direct measurements are the best. The trick is to create a service level agreement (SLA) for every service then directly collect metrics to determine how close to the SLA we comply. Then we have the numbers that let us determine which SLAs should be tightened or loosened. That is all a lot of work! The second is about personal efficiency. This means looking at how can we organise our work so that we get the most done, in a stress-free manner, so that we "play well with others". There are many impediments to this, sometimes artificially introduced by incompetent management, dysfunctional processes, or badly managed expectations. Worst of all, we receive little mentoring about time management in school or from our managers. Hey, that's why I wrote Time Management for System Administrators!
You were in Australia last year for a SAGE conference, and you have been several times in years past. You must have built up a few impressions of the place?
This will be my fourth trip to Australia. I've enjoyed every visit! On my first visit to Australia, I had some problems understanding the language. The language barrier is similar to a non-technical person calling an IT helpdesk: both parties are speaking English, but they can't understand each other. Eventually I've learned to use Australian terms much the same way that someone calling an IT helpdesk learns to call their mouse a mouse, not "the thingy that moves the cursor". When travelling I dine out a lot and I've had confusion because the parts of a meal (entree, main, etc.) are called different things than in the US. There are more embarrassing examples, but that's all I'll reveal for now.