The DeepDyve search engine enables searches of the Dark Web to more easily find data life sciences, patent, and data. The new engine indexes 500 million pages, said DeepDyve, which was known as Infovell before changing its name on Tuesday.
The company said it will soon start indexing physical sciences content in the areas of information technology, clean technology and energy - which will help it meet its goal of growing its index to more than 1 billion pages by the end of the year.
Because much of the content on the Deep Web is made up of technical publications, databases, scholarly publications and unstructured data, it has been difficult for traditional search engines to access it. To tackle this issue, DeepDyve is partnering with those publishers to gain access to content overlooked by other engines, the company added.
earlier this month that it is ratcheting up its aim at the Dark Web by adding the ability to search PDF documents. In April, that it was trying to find a way for its search engine to index HTML forms such as drop-down boxes and select menus that are typically part of the Dark Web.
"According to IDC, more than 42 million consumers spend 25 hours per month online researching business and personal information, and they are frustrated with the results they get back and the tools they have to use," said William Park, CEO of DeepDyve, in a statement. "DeepDyve gives information-savvy consumers unparalleled access to quality information found only in the Deep Web, with features and functionality that make it easy to find, filter and organize their results."
The company's technology is aimed at allowing users to type in a few words or copy an entire article into a query box to find all related articles located in the Deep Web, DeepDyve noted.
Chris Sherman, a blogger at Search Engine Land, said that to scouring the Deep Web is innovative. He credited the company's chief scientists, who are veteran genomics researchers. To crack the genetic codes contained in DNA sequences, he noted, researchers must understand the hidden patterns in of massive amounts of data.
"DeepDyve takes a similar approach to understanding information on the Web," Sherman added. "Going far beyond basic keyword-based search, DeepDyve indexes every word in a document, but also computes the factorial combination of words and phrases in the document and uses some industrial strength statistical techniques to assess the 'informational impact' of these combinations. In essence, this approach looks at the meaning of an entire document and uses that to compute relevance, rather than factors like snippets of text or anchor text in links pointing to documents."
While DeepDyve isn't a threat to now nor is it likely to be in the future, the tool is good for searchers aiming to research in areas that DeepDyve indexes.
"DeepDyve also offers a genuinely different 'second opinion' of the Web if you're wanting to look beyond the top results returned by Google and the other major search engines," Sherman noted. "With its limited initial offering, DeepDyve has just scratched the surface of what's available on the invisible web, albeit in a very useful way. However, truly cracking the invisible web problem still seems like a distant dream."