Graphic image: Multiarmed woman getting information from multiple sources.

The Importance of Multiple Sources!

Google, the largest of the commercial search engines, currently claims about 3.3 billion indexed pages.   By some estimates this means that Google searches about half the available web pages on the Internet. This reality makes the argument for using multiple search engines even stronger. Search engines index different parts of the web. True, there is significant crossover; however each engine is finding new pages every second. To get the most available relevant information you should use several search engines. Just as you'd get three estimates for a car repair, searchers should use at least three unique search engines. You'll be searching far more of the potential 6 billion pages of available information if you use Google, HotBot, and Teoma. Clearly depending on a single source for information will limit your results.  

 

 

 

Image: startled man with a bank check with numbers so large they run off the page.

Can you estimate the size of the Invisible Web?

Indeed, most pages of information are beyond the reach of popular search engines. These pages are part of the invisible web. Bright Planet.com estimates the number of pages hidden from commercial search engines to be 400 to 550 times larger than what is available to commercial search engines.   These pages are part of the 'invisible web' estimated by Bright Planet to be as large as 3.5 trillion pages. Other experts feel the Bright Planet estimates are inflated, but still maintain that the 'invisible web' is from two to fifty times larger than the visible web. By the conservative estimate, we can guess at there being 50 to 100 billion pages of information on the invisible web.

Invisible web pages are hidden in password-protected systems, intentionally excluded from robotic search engines, or dynamically generated by online databases at a user's request. Invisible web information may be highly relevant to your search needs, and can be found if you know where to look. (See the IMSA Micro Module: Invisible Web, for more information on this issue.)  

What Does it all Mean?

Clearly the Internet will continue to change. The more web pages of relevant information a researcher can access the better. Competition between rival search engines, improvement in indexing and retrieval technology, and the ever-increasing number of pages available on the Internet means that searching the net will continue to take both specialized knowledge and persistence. The best approach is to use multiple search engines to search the public web, and to become more aware of invisible web resources.

Cartoon Image of a computer reading from paper text. Listen


Authored by Dennis O'Connor 2003-2004