The Invisible Web

What is the invisible web?

To understand the nature of the invisible web it is necessary to understand a little about how search engines work. These services are based on a database which holds information about all the web pages the search engine covers, known as an index. The index is automatically compiled by programmes known as spiders that 'crawl' the Internet, following links from page to page and adding each page to the index as it goes.

The invisible web is a collective term for information that spiders cannot reach. A huge amount of the information available on the Internet is contained in databases accessible via a search interface rather than a set of web pages. Humans can search databases for the information they need but spiders, however, have no means to extract information from these databases. If you rely on search engines you may miss out on content-rich databases from universities, libraries, associations, businesses, and government agencies around the world.

Guides to the invisible web

To find resources in the invisible web, you have to rely on the fruits of human labours. Finding these resources is a good reason for using Internet directories, but there are also websites that focus on making the invisible web accessible.

The Ultimate Guide to the Invisible Web, is a guide to invisible  web resources. You can search the directory of Humanities, Media, Science, State Databases and Biographies as well as other reference resources compiled by Gary Price such as the List of Lists and Fast Facts. Price is co-author with Chris Sherman of the book The Invisible Web: Finding Hidden Internet Resources Search Engines Can't See written by and Gary Price, published in 2001.

Futher explanation and information on How To Find and Search the Invisible Web by Wendy Boswell.