Another mistake is collecting the most accessible data. "People tend to collect the information that is easiest to get," says Sherry Gordon, vice president of supplier performance intelligence at Emptoris Inc. "You have to collect the right stuff."

Managers who will be using the data need to decide what they're trying to accomplish with the information. Measures to manage the costs of products and services will be different from those used to improve quality, for example. "You don't want to just collect the data for the sake of data," Gordon says. "You need to be able to take action on the information."

The Allstate Corp. focuses its scorecards on performance criteria, including how well suppliers meet their deadlines, the quality of the service received, and whether products and services meet Allstate's budgetary goals. Scorecards help take the emotion out of the rankings. "We really like to focus on the performance of suppliers rather than on relationships," says Lori Yelvington, assistant vice president for procurement governance. "We have scorecards so that we have solid facts to base our relationship on."

2. Format. Scorecards range from simple spreadsheets to sophisticated software embedded in a company's ERP system. Anne Marie Biernacki, chief technology officer and co-founder of The Digiticians LLC, says it can be as simple as a table in a Word document.

In contrast, at Affymetrix, the ERP system tracks each vendor's transaction record, including purchase orders, invoices and shipments, for use in the scorecard. It also has a system that tracks the quality of products delivered and the time it takes to resolve problems related to quality. "We have different layers or systems [and] many users that contribute information on vendor performance," Healey says.