It's common for large companies to have policies that call for the termination of an executive who misrepresents his background, Enderle said. This kind of discrepancy also speaks to the person's ethics, which would call into question his action as leader of a major company, he said.

"A CEO can't be untrustworthy because they are the face of the company and they stand behind the company's financials," Enderle said. "It is common to ask an executive who has misrepresented their credentials to step down. But Yahoo had a great deal of trouble getting him in the first place... It will be very difficult to get someone to replace Thompson who is as qualified, and that may force a less draconian result."

Once a leader in the online world, Yahoo has been overshadowed by players like and . The company has been struggling over the past year or so to regain its top-tier status. With Thompson in Yahoo's top seat, industry analysts largely said he was working to rebuild the company. Now those plans may be in jeopardy.

Thompson launched a major restructuring of the company that included the , or 14% of its workforce, in April. The jobs cuts are expected to save the company $375 million.

That's not the only shakeup at the company.