StreamBase applications use an "inbound" query-processing model, in which records are processed before they're indexed and stored. The records flow through the query, which can also transform the data while it's moving.

Vision Systems & Technology Inc. (VSTI) in Ellicott City, Md., is helping several defense and intelligence agencies evaluate StreamBase prototypes. StreamBase can filter torrents of incoming data -- structured or unstructured -- and decide on the fly which should be presented to an analyst at once, which can be stored for later queries and which can be discarded, says Carol Lundquist, an IT consultant at VSTI.

The technology can generate alerts when a passing record contains, for example, a certain name or phone number. "You can put keywords in an Oracle table, and anytime a keyword is added, it gets dumped down to StreamBase immediately," Lundquist says.

"Some government systems are being flooded with data," she explains. "The Oracle systems are having trouble keeping up, and you get data falling on the floor." One government system Lundquist worked on loaded 1 billion records in a day, she says.

Filtering can be the salvation for some of those systems, says Bryan Harris, CTO of VSTI. "The idea is to load the needles, not the haystack," he says.