So you'd think that by now, Harrah's has pretty much nailed which IT metrics it needs to track, right? Think again. "I'm literally holding in my hand a revised list of metrics and scorecards for this year, and I continue to nosh through this with my team, year after year," says Stanley, senior vice president and CIO at Harrah's.

Changes in the business and competitive landscapes force Harrah's IT department to regularly reassesses which key performance indicators to track. "We're constantly asking ourselves whether we're measuring the right things," says Stanley.

He isn't alone. IT leaders at companies such as DHL Express, Cendant Hotel Group and Kaiser Permanente say they continually wrestle with what they're measuring to gauge the effectiveness of their IT organizations. But by focusing on what business leaders are looking for, they're getting a lot more sophisticated at what to measure -- and what not to.

Beyond rote

Best-in-class IT organizations have gone beyond rote metrics such as system uptime and help desk problem resolution. Some IT shops on Wall Street, for example, are studying things like how many stock trades or customers they're able to support for each IT dollar they spend, says Howard Rubin, an analyst at Gartner Inc.