The more you know about the ins and outs of beta testing, the better you'll be at choosing the programs that make the most sense for your company. Here's a primer on what it's all about.
Opportunity for Influence
Hudson's Bay participates in beta programs with the vendors of its point-of-sale application, systems software and enterprise data warehouse. The retailer's enterprise data warehouse vendor is Teradata , which is a division of Dayton, Ohio-based NCR Corp. Hudson's Bay didn't jump into beta testing. "[We] started using Teradata in 1998, but we've participated in four or five beta programs with NCR only in the last few years," says Jarvis-Haig. The impetus to participate comes from project managers who know the applications best and see the value in shaping new features. "They make the case to senior IT management," she says.
Project managers point to a number of benefits. "We get our requirements addressed and can influence the vendor," Jarvis-Haig explains. "A new feature may need some changes to work as we think it should. As a beta user, we get to say [that]. We also evaluate performance in a test environment before deciding to put code into production. We like to be on the cutting edge. It helps keep the technical staff motivated."
But there's more to beta testing than getting to play with the new toys first. "Early access to critical new functions shouldn't be the only reason to be a beta tester," says Michael Fine, author of Beta Testing for Better Software (Wiley, 2002). "Building the relationship with the vendor, especially for enterprise-level products, is another incentive."
Randy Lea agrees. "Success for the beta participant is to see realized features and growth in the [vendor] partnership," says Lea, vice president of products and services at Teradata.
And there may be a financial benefit as well. "Successful beta participation leads to a lower cost of implementation," says Kathy Lang, CIO at Marquette University in Milwaukee. That's because fast turnaround on issues uncovered, as well as support-team and end-user readiness, contribute to a smoother production installation when the beta is complete. "There may also be the incentive of a lower price," Lang says.
Vendors provide prerelease products to beta users at no charge, and some sweeten the deal. For example, Charleston, S.C.-based Blackbaud Inc., a developer of software for the nonprofit sector, offers its beta users the opportunity to earn "beta bucks," which can be used to offset the cost of other Blackbaud products and services.
Bucky Wall, release manager at Blackbaud, oversees the group that's responsible for product stabilization and release. "Beta is the main thing that occurs during stabilization," says Wall.
In an effort to double the number of beta testers this year, he is targeting user groups. Other vendors canvass at conferences, ask salespeople to recommend candidates and invite customers that have formally submitted requirements. Vendors may even mine technical support logs to identify potential beta participants.
Successful beta testing takes commitment and communication from both sides. "In a real beta, what the customer has to say is important," Fine says. "Every bit of feedback must be recognized by the vendor. If there's no response, beta testers get unmotivated. At the same time, a good beta tester shows sincere interest, participation and engagement."
Beta testing also demands resources from both parties. Blackbaud, for example, requires that beta customers have broadband Internet access. Wall explains that this is needed to allow the vendor to access the customer environment directly to install code, figure out problems and fix them.
"We seek a validation of resource commitment," says Lea. "We ask, 'How well can they commit to plan?'"
The Process
From a participant's point of view, the beta-testing process has five steps, says Jarvis-Haig: interest, negotiation, qualification, implementation and closure.
- Interest. The first step is identifying features or functions in an upcoming product release that may benefit your organization. That means that if your company has decided to regularly participate in betas, you need to be aware of the vendor's product release schedule -- to know what's coming and when. Renee Klish, IT supervisor and database administrator at the Detroit Zoological Society, signed on for a beta test when she identified a new analytic function in Blackbaud's Raiser's Edge that would allow her organization to eliminate another product, saving on license fees and simplifying its IT environment.
- Negotiation. This is an internal process. The user organization decides whether the potential benefit in the new release merits the time, energy and resources required for beta participation and then goes to senior IT management for approval. If your upper management is risk-averse, this step may be a challenge, says Jarvis-Haig.
- Qualification. Next, the user and vendor must agree that the beta test makes sense for both sides. If the user company pushed for a capability that the vendor just added, the vendor may be eager to have the user take part. On the other hand, the user company may be a stretch candidate, and the vendor may have to be convinced that it should participate. "We look for user breadth [of feature use] in the next release," says Teradata's Lea. Other qualifications may include having prerequisite products in the user environment, assurance that there are no schedule conflicts and a history of good beta participation.
"It's a two-way street," says Jarvis-Haig. "They're not going to qualify us if it isn't in their interest."
Once the user is qualified, there most likely will be a formal agreement. Blackbaud beta participants get a program guide that includes the beta agreement, nondisclosure forms, details on incentives and descriptions of the beta process. Teradata beta participants sign an agreement and develop a plan with NCR.
- Implementation. The duration of most beta programs is one to three months, although the period may be shorter. "Implementation is intense [and] of short duration," says Jarvis-Haig. This is when the real work happens. For a limited period, new software is installed on the user's development systems. Developers work with new or enhanced interfaces, or they may test the performance of benchmark transactions.
Users also perform regression testing to ensure that functions currently in use still work. Tasks here may include setup (to reproduce the production environment), running automated test cases and arranging for expert end users to operate in the beta environment. User feedback often leads to changes in the product. "We provide input to the vendor," says Jarvis-Haig. "We can influence the end result."
This ability to affect the final product is a key benefit for beta testers, says Hannah Roberts, communications coordinator at Opera North , an opera company in Leeds, England. For example, she says, "we had a say in shaping [a product's] database to better suit our needs."
Klish says that Blackbaud has added prompts, changed flows and simplified interfaces because of feedback from the Detroit Zoological Society.
Good communication between the vendor and beta testers during this time is essential. To facilitate that, each Teradata beta account has a site manager who is an experienced product engineer with links into the development organization. At Blackbaud, a "beta buddy" talks with each beta user weekly and as needed. Both vendors use the Web extensively for communication.
- Closure. This is the last step, and it typically includes a beta-test report written by the lead beta tester or project manager. "The beta report will discuss issues resolved, learnings -- did the product work like we thought it would?" Jarvis-Haig says. It also may include sections on expectations, the timeline, activities, problems found and recommendations. There may be a closure meeting for the vendor and testers, and some vendors ask beta participants to complete a final survey.
Despite the potential benefits, CIOs say you should be discriminating in your choice of beta programs to make the most of your time and resources. "Beta participation is not a standard activity," says Lang. "When a critical product has new features that are important to the school, we will be a beta." During her three and a half years at Marquette, the IT department has taken part in only two beta-testing programs, she says.
Beta testing can be an important tool for IT. By carefully choosing where to participate, being aware of and leveraging multiple benefits, and enabling full engagement, IT managers can motivate employees, increase user satisfaction and use computing resources more productively.
Shea is a freelance writer in Bayside, Wis. Contact him at garytshea@att.net.
SIDEBAR
Beta Without the Bother: A Different Approach to Beta Testing
Occasionally, beta testing requires minimal involvement by the user company.
As marketing manager for Teradata products and services, Randy Lea influences the product management group, which has ultimate responsibility for beta programs. To help ensure that a tight production schedule is met, product management sometimes hires NCR 's own professional services arm to perform beta testing at a customer's location. NCR consultants install beta code and manage testing in the customer's environment.
Lea says this relatively new approach serves several purposes. It brings NCR's professional services staffers up to speed on the latest release, it enables them to observe the customer experience directly, and it strengthens the user/vendor partnership through the vendor's physical presence at the user site.
SIDEBAR
Sidebar: Whats In It for You?
What users want:
- Requirements addressed
- New functions to try
- Partnerships
- Staff motivation
- Incentives
What vendors want:
- Validation
- Product readiness
- Customer perspective
- Partnerships
- References