"You need someone in-between," said Bolton.

For his part, Pettibone believes the ideal CIO candidate is someone who rises up from the IT ranks and hasn't been transferred from another functional area. "I used to joke with the general counsel at New York Life that if they wanted to make him the CIO, they should make me the general counsel," said Pettibone.

Regardless of a CIO's roots, the individual "has to almost be a better business person than the business people because you have so much less time to figure out the challenges for each of those business silos," said Rose.

To engender hope in an IT organization, CIOs need several good qualities, including the ability to build relationships and listen carefully, said Bolton. "Communications is more about listening than it is about speaking," he said.

Rose can relate to the importance of listening. Ten years ago, when Rose was the chief technology officer at Standard & Poor's Retail Markets, an engineer who was experiencing an I/O problem with an operating system informed Rose about a discovery he had made with a then little-known operating system called Linux. "He said, 'You've got to check this thing out, it's amazing,' and I'm thinking, 'Like I have time for this,'" said Rose.