Rometty, who has worked at IBM for decades, most recently has been IBM senior vice president and group executive for sales, marketing and strategy.

"Ginni Rometty has successfully led several of IBM's most important businesses over the past decade -- from the formation of IBM Global Business Services to the build-out of our Growth Markets Unit," Palmisano said in a statement. "With every leadership role, she has strengthened our ability to integrate IBM's capabilities for our clients."

Rometty's work integrating PwC and building a global team of more than 100,000 consultants, as well as other roles she has held at the company, have made her a driving force for IBM's Smarter Planet strategy, according to IDC analyst Frank Gens. Smarter Planet is focused on using technology to transform businesses, governments and industries.

"I think that their strategy has proven to be a very strong one," Gens said. "She represents continuity of strategy."

Rometty has a good understanding of IBM's software, services and hardware business, and customers shouldn't expect any major changes with the new leadership, said analyst Chris Foster of Technology Business Research.