Sponsored by the Department of Homeland Security (DHS) and maintained by a team of workers at nonprofit Mitre and other security professionals, the ongoing effort is roughly four months shy of publishing a final draft of its vulnerability encyclopedia, said leaders of the project.
Presenting at the ongoing Black Hat 2007 conference, CWE initiative leaders said they are busy aggregating and organizing the mountains of vulnerability data they have gathered and said they are working more closely than ever with applications security testing companies to help compare the abilities of various software scanning tools.
Launched in Dec. 2005, CWE seeks to establish a unified, measurable set of software flaws to help developers improve the quality of their products and drive out the types of vulnerabilities that have led to the ongoing malware explosion.
By gathering input on commonly seen mistakes from developers, researchers, and security vendors, the group believes it can create a common language and standard procedures for handling the many different types of loopholes that exist in programs' source code today.
Prior to the delivery of a final draft of its encyclopedia of flaws later this year, CWE officials said they are preparing a sixth iteration of the index to likely be published some time in April.
While much of the work the group has completed thus far has revolved around the gathering of vulnerability formats and the various methods used to identify and remediate the coding problems, the project has recently involved a significant amount of testing of security scanning tools to get a better idea of the capabilities and limitations of those products.
By gaining an understanding of the vulnerabilities that popular code scanning engines can find --'and those they can't --'CWE can help developers understand the types of issues they will need to look for on their own, said Bob Martin, a CWE leader and the head of Mitre's related CVE (Common Vulnerability Exposures) Compatibility effort.
"We wanted to evaluate what the tools claim to cover and what they are most effective at finding," Martin said. "Right now, best test is to throw tools at a big pile of code and see what tools find the most vulnerabilities, but we're changing that paradigm into test cases where we now look at the answers so we can evaluate what the tools found and what kinds of complexities they can handle."
CWE's research will not list the names and performance results of the products it is testing -- provided by over 20 firms, including Cenzic, Fortify, SPI Dynamics, Veracode, and Watchfire -- but the work to compile a resource that offers developers an idea of the types of vulnerabilities missed by the tools should provide a great deal of value, Martin said.
Officials with the project said that they have been pleasantly surprised by the variety of methods employed by the commercial scanning tools and the different types of flaws found by the various products. Going into this phase of the research, the group expected to find that many scanning systems identified the same types of issues, said Sean Barnum, director of knowledge management at Cigital, a software quality assurance company.
The tests also revealed that the products were looking for only 45 percent of the 600 common vulnerabilities that have already been entered into the CWE index.
"We found that less than half of what we already have in CWE is covered by these tools, so this helps prove that there are a lot of known issues out there that aren't being addressed," Barnum said. "We also thought that the tools would look for the same types of things, but they are actually very different, and there's not a lot of overlap; that's something that developers need to be aware of as they choose tools; you want to right set for aggregated coverage."
Before each release of CWE, workers with the project spend much of their time comparing all the vulnerability definitions and mitigation taxonomies in the index, attempting to refine the language used in the descriptions and add real-world examples of attacks that target the flaws.
That work is continuing and will remain the primary focus of CWE's efforts going forward, officials said, including work to de-emphasize nomenclature that describes common problems based on the attack methods used to exploit them.
Barnum said that at this point, the encyclopedia isn't as polished as it might be, but he observed that he and other project organizers believe that taking an all-inclusive approach is the best strategy for expanding the resource at this point.
"It's still a kitchen sink approach, but that's a smart approach," Barnum said.