100 Useful Tips and Tools to Research the Deep Web

By Alisa Miller

Experts say that typical search engines like Yahoo! and Google only pick up about 1% of the information available on the Internet. The rest of that information is considered to be hidden in the deep web, also referred to as the invisible web. So how can you find all the rest of this information? This list offers 100 tips and tools to help you get the most out of your Internet searches.

Meta-Search Engines

Meta-search engines use the resources of many different search engines to gather the most results possible. Many of these will also eliminate duplicates and classify results to enhance your search experience.

  1. SurfWax. This search engine works very well for reaching deep into the web for information.
  2. Academic Index. Created by the former chair of Texas Association of School Librarians, this meta-search engine only pulls from databases and resources that are approved by librarians and educators.
  3. Clusty. Clusty searches through top search engines, then clusters the results so that information that may have been hidden deep in the search results is now readily available.
  4. Dogpile. Dogpile searches rely on several top search engines for the results then removes duplicates and strives to present only relevant results.
  5. Turbo 10. This meta-search engine is specifically designed to search the deep web for information.
  6. Multiple Search. Save yourself the work by using this search engine that looks among major search engines, social networks, flickr, Wikipedia, and many more sites.
  7. Mamma. Click on the Power Search option to customize your search experience with this meta-search engine.
  8. World Curry Guide. This meta-search tool with a strong European influence has been around since 1997 and is still growing strong.
  9. Fazzle.com. Give this meta-search engine a try. It accesses a large number of databases and claims to have more access to information than Google.
  10. Icerocket. Search blogs as well as the general Internet, MySpace, the news, and more to receive results by posting date.
  11. iZito. Get results from a variety of major search engines that come to you clustered in groups. You can also receive only US website results or receive results with a more international perspective.
  12. Ujiko. This unusual meta-search tool allows for you to customize your searches by eliminating results or tagging some as favorites.

Semantic Search Tools and Databases

Semantic search tools depend on replicating the way the human brain thinks and categorizes information to ensure more relevant searches. Give some of these semantic tools and databases a try.

  1. Hakia. This popular semantic search engine only accesses websites that are recommended by librarians.
  2. Zotero. Firefox users will like this add-on that helps you organize your research material by collecting, managing, and citing any references from Internet research.
  3. Freebase. This community-powered database includes information on millions of topics.
  4. Powerset. Enter a topic, phrase, or question to find information from Wikipedia with this semantic application.
  5. Kartoo. Enter any keyword to receive a visual map of the topics that pertain to your keyword. Hover your mouse over each to get a thumbnail of the website.
  6. DBpedia. Another Wikipedia resource, ask complex questions with this semantic program to get results from within Wikipedia.
  7. Quintura. Entering your search term will create a cloud of related terms as well as a list of links. Hover over one of the words or phrases in the cloud to get an entirely different list of links.
  8. [true knowledge]. Help with current beta testing at this search engine or try their Quiz Bot that finds answers to your questions.
  9. Stumpedia. This search engine relies on its users to index, organize, and review information coming from the Internet.
  10. Evri. This search engine provides you with highly relevant results from articles, papers, blogs, images, audio, and video on the Internet.
  11. Gnod. When you search for books, music, movies and people on this search engine, it remembers your interests and focuses the search results in that direction.
  12. Boxxet. Search for what interests you and you will get results from the "best of" news, blogs, videos, photos, and more. Type in your keyword and in addition to the latest news on the topic, you will also receive search results, online collections, and more.

General Search Engines and Databases

These databases and search engines for databases will provide information from places on the Internet most typical search engines cannot.

  1. DeepDyve. One of the newest search engines specifically targeted at exploring the deep web, this one is available after you sign up for a free membership.
  2. OAIster. Search for digital items with this tool that provides 12 million resources from over 800 repositories.
  3. direct search. Search through all the direct search databases or select a specific one with this tool.
  4. CloserLook Search. Search for information on health, drugs and medicine, city guides, company profiles, and Canadian airfares with this customized search engine that specializes in the deep web.
  5. Northern Light Search. Find information with the quick search or browse through other search tools here.
  6. Yahoo! Search Subscriptions. Use this tool to combine a search on Yahoo! with searches in journals where you have subscriptions such as Wall Street Journal and New England Journal of Medicine.
  7. CompletePlanet. With over 70,000 databases and search engines at its disposal, this is an excellent resource for searching the deep web.
  8. The Scout Archives. This database is the culmination of nine years’ worth of compiling the best of the Internet.
  9. Daylife. Find news with this site that offers some of the best global news stories along with photos, articles, quotes, and more.
  10. Silobreaker. This tool shows how news and people in the news impacts the global culture with current news stories, corresponding maps, graphs of trends, networks of related people or topics, fact sheets, and more.
  11. spock. Find anyone on the web who might not normally show up on the surface web through blogs, pictures, social networks, and websites here.
  12. The WWW Virtual Library. One of the oldest databases of information available on the web, this site allows you to search by keyword or category.
  13. pipl. Specifically designed for searching the deep web for people, this search engine claims to be the most powerful for finding someone.

Academic Search Engines and Databases

The world of academia has many databases not accessible by Google and Yahoo!, so give these databases and search engines a try if you need scholarly information.

  1. Google Scholar. Find information among academic journals with this tool.
  2. WorldCat. Use this tool to find items in libraries including books, CDs, DVDs, and articles.
  3. getCITED. This database of academic journal articles and book chapters also includes a discussion forum.
  4. Microsoft Libra. If you are searching for computer science academic research, then Libra will help you find what you need.
  5. BASE – Bielefeld Academic Search Engine. This multi-disciplinary search engine focuses on academic research and is available in German, Polish, and Spanish as well as English.
  6. yovisto. This search engine is an academic video search tool that provides lectures and more.
  7. AJOL – African Journals Online. Search academic research published in AJOL with this search engine.
  8. HighWire Press. From Stanford, use this tool to access thousands of peer-reviewed journals and full-text articles.
  9. MetaPress. This tool claims to be the "world’s largest scholarly content host" and provides results from journals, books, reference material, and more.
  10. OpenJ-Gate. Access over 4500 open journals with this tool that allows you to restrict your search to peer-reviewed journals or professional and industry journals.
  11. Directory of Open Access Journals. This journal search tool provides access to over 3700 top "quality controlled" journals.
  12. Intute. The resources here are all hand-selected and specifically for education and research purposes.
  13. Virtual Learning Resource Center. This tool provides links to thousands of academic research sites to help students at any level find the best information for their Internet research projects.
  14. Gateway to 21st Century Skills. This resource for educators is sponsored by the US Department of Education and provides information from a variety of places on the Internet.
  15. MagBot. This search engine provides journal and magazine articles on topics relevant to students and their teachers.
  16. Michigan eLibrary. Find full-text articles as well as specialized databases available for searching.

Scientific Search Engines and Databases

The scientific community keeps many databases that can provide a huge amount of information but may not show up in searches through an ordinary search engine. Check these out to see if you can find what you need to know.

  1. Science.gov. This search engine offers specific categories including agriculture and food, biology and nature, Earth and ocean sciences, health and medicine, and more.
  2. WorldWideScience.org. Search for science information with this connection to international science databases and portals.
  3. CiteSeer.IST. This search engine and digital library will help you find information within scientific literature.
  4. Scirus. This science search engine moves beyond journal articles and also includes searches among such resources as scientists’ webpages, courseware, patents, and more.
  5. Scopus. Find academic information among science, technology, medicine, and social science categories.
  6. GoPubMed. Search for biomedical texts with this search engine that accesses PubMed articles.
  7. the Gene Ontology. Search the Gene Ontology database for genes, proteins, or Gene Ontology terms.
  8. PubFocus. This search engine searches Medline and PubMed for information on articles, authors, and publishing trends.
  9. Scitopia. This "deep federated search" brings the best information from the fields of science and technology.
  10. Scitation. Find over one million scientific papers from journals, conferences, magazines, and other sources with this tool.

Custom Search Engines

Custom search engines narrow your focus and eliminate quite a bit of the extra information usually contained in search results. Use these resources to find custom search engines or use the specific custom search engines listed below.

  1. CustomSearchEngine.com. This listing includes many of the Google custom search engines created.
  2. CustomSearchGuide.com. Find custom search engines here or create your own.
  3. CSE Links. Use this site to find Google Coop custom search engines.
  4. PGIS PPGIS Custom Search. This search engine is customized for those interested in the "practice and science" of PGIS/PPGIS.
  5. Files Tube. Search for files in file sharing and uploading sites with this search engine.
  6. Trailmonkey’s Custom Search Engine. This outdoor adventure search engine will help find information such as trails, maps, and wildlife around the world.
  7. Rollyo. "Roll your own search engine" at this site where you determine which sites will be included in your searches.
  8. Webhoker.com. Use this custom search engine to find information about Northern Ireland.
  9. Figure Skating Custom Search Engine. Use this search engine to learn about figure skating. The more this search engine is used, the better the results become.
  10. Custom Search Engines. There are three custom search engines here, two of which may be relevant for anyone interested in Utah constitution or juvenile justice.
  11. Go Pets America Custom Search Engine. This search engine will help you find information on pets and animals, their health and wellness, jobs in the field, and more.

Collaborative Information and Databases

One of the oldest forms of information dissemination is word-of-mouth, and the Internet is no different. With the popularity of bookmarking and other collaborative sites, obscure blogs and websites can gain plenty of attention. Follow these sites to see what others are reading.

  1. Del.icio.us. As readers find interesting articles or blog posts, they can tag, save, and share them so that others can enjoy the content as well.
  2. Digg. As people read blogs or websites, they can "digg" the ones they like, thus creating a network of user-selected sites on the Internet.
  3. Technorati. Not only is this site a blog search engine, but it is also a place for members to vote and share, thus increasing the visibility for blogs.
  4. StumbleUpon. As you read information on the Internet, you can Stumble it and give it a thumbs up or down. The more you Stumble, the more closely aligned to your taste will the content become.
  5. Reddit. Working similarly to StumbleUpon, Reddit asks you to vote on articles, then customizes content based on your preferences.
  6. Twine. With Twine you can search for information as well as share with others and get recommendations from Twine.
  7. Kreeo.com. This collaborative site offers shared knowledge from its members through forums, blogs, and shared websites.
  8. Talk Digger. Find information on the Internet based on what others are saying about it. Members discuss web sites, blogs, and specific topics here.

Tips and Strategies

Searching the deep web should be done a bit differently, so use these strategies to help you get started on your deep web searching.

  1. Don’t rely on old ways of searching. Become aware that approximately 99% of content on the Internet doesn’t show up on typical search engines, so think about other ways of searching.
  2. Search for databases. Using any search engine, enter your keyword alongside "database" to find any searchable databases (for example, "running database" or "woodworking database").
  3. Get a library card. Many public libraries offer access to research databases for users with an active library card.
  4. Stay informed. Reading blogs or other updated guides about Internet searches on a regular basis will ensure you are staying updated with the latest information on Internet searches.
  5. Search government databases. There are many government databases available that have plenty of information you may be seeking.
  6. Bookmark your databases. Once you find helpful databases, don’t forget to bookmark them so you can always come back to them again.
  7. Practice. Just like with other types of research, the more you practice searching the deep web, the better you will become at it.
  8. Don’t give up. Researchers agree that most of the information hidden in the deep web is some of the best quality information available.

Helpful Articles and Resources for Deep Searching

Take advice from the experts and read these articles, blogs, and other resources that can help you understand the deep web.

  1. Deep Web – Wikipedia. Get the basics about the deep web as well as links to some helpful resources with this article.
  2. Deep Web – AI3:::Adaptive Information. This assortment of articles from the co-coiner of the phrase "deep web," Michael Bergman offers a look at the current state of deep web perspectives.
  3. The Invisible Web. This article from About.com provides a very simple explanation of the deep web and offers suggestions for tackling it.
  4. ResourceShelf. Librarians and researchers come together to share their findings on fun, helpful, and sometimes unusual ways to gather information from the web.
  5. Docuticker. This blog offers the latest publications from government agencies, NGOs, think tanks, and other similar organizations. Many of these posts are links to databases and research statistics that may not appear so easily on typical web searches.
  6. TechDeepWeb.com. This site offers tips and tools for IT professionals to find the best deep web resources.
  7. Digital Image Resources on the Deep Web. This article includes links to many digital image resources that probably won’t show up on typical search engine results.
  8. Federated Search 101. Learn about federated search tools in this article that will be helpful to businesses thinking about purchasing a federated search product.
  9. Timeline of events related to the Deep Web. This timeline puts the entire history of the deep web into perspective as well as offers up some helpful links.
  10. The Deep Web. Learn terminology, get tips, and think about the future of the deep web with this article.