3. Standards. A common mistake with novice scorecard designers is to look for standard measures to use, Maurer says, but there are none. What is right for one company may not be right for another, so when building a scorecard, make sure you tailor it to deliver the specific information you need.

Many organizations, particularly those that outsource their IT infrastructures, use service-level agreements (SLA) as foundational criteria for building a scorecard, Maurer says.

"We start out with the SLAs in the contract" when developing some scorecards, says Yelvington. "Those are real. Everyone knows what those are, and they are pretty easy to grade."

4. Collection. There are various methods for gathering the data needed to create a vendor scorecard. Some companies, like Affymetrix, tap into ERP systems. At others, IT managers visit suppliers to get a detailed look at how they operate and to ensure quality. Many get information through internal customer surveys.

For key vendors, Allstate polls its employees, asking questions such as whether the vendors invoice properly and provide good reports and high-quality customer service. The survey may also include more subjective questions, such as whether an employee would recommend renewing the contract, Yelvington says.