While much of the work the group has completed thus far has revolved around the gathering of vulnerability formats and the various methods used to identify and remediate the coding problems, the project has recently involved a significant amount of testing of security scanning tools to get a better idea of the capabilities and limitations of those products.

By gaining an understanding of the vulnerabilities that popular code scanning engines can find --'and those they can't --'CWE can help developers understand the types of issues they will need to look for on their own, said Bob Martin, a CWE leader and the head of Mitre's related CVE (Common Vulnerability Exposures) Compatibility effort.

"We wanted to evaluate what the tools claim to cover and what they are most effective at finding," Martin said. "Right now, best test is to throw tools at a big pile of code and see what tools find the most vulnerabilities, but we're changing that paradigm into test cases where we now look at the answers so we can evaluate what the tools found and what kinds of complexities they can handle."

CWE's research will not list the names and performance results of the products it is testing -- provided by over 20 firms, including Cenzic, Fortify, SPI Dynamics, Veracode, and Watchfire -- but the work to compile a resource that offers developers an idea of the types of vulnerabilities missed by the tools should provide a great deal of value, Martin said.

Officials with the project said that they have been pleasantly surprised by the variety of methods employed by the commercial scanning tools and the different types of flaws found by the various products. Going into this phase of the research, the group expected to find that many scanning systems identified the same types of issues, said Sean Barnum, director of knowledge management at Cigital, a software quality assurance company.