Web Searching Strategies

Subject Directories -- Research Starting Points

What is a Subject Directory?

categorized lists of sites organized by subject
created, organized, and maintained by a person, not a computer program
many include a search engine that searches its own database or the Web

When is it best to use a Subject Directory?

When you have a broad topic and want to learn the aspects of a topic
When you want recommended, evaluated sites with substantive content
When you want to browse to see what is out there
Keep your search terms broad - search for concepts or the discipline of your topic

Commercial portals:

About.com - Over 1 million web sites. Uneven quality. Can click on the person's name to determine credentials.
Google Web Directory - includes sites from the Open Directory Project

LookSmart

Yahoo! - UNevaluated web pages mainly commercial and popular for the general public. Good regional coverage.

Academic, professional portals -- General

Academic Info - Collection of college & university level sites, databases, gateway pages. Good starting point for undergrads.
Best Information on the Net
BUBL Link - UK academic directory
INFOMINE - Contains 120,000+ annotated research university level sites. Divided into several databases by broad disciplines. Resi
I nternet Public Library - Good sites for thegeneral public, k-12 and academic levels of research.
Librarians Index to the Internet - 16,000+ annotated sites selected & evaluated by California academic & public librarians.
Resource Discovery Network - UK focus
Scout Report Archives - Academic sites maintained by the University of Wisconsin.
WWW Virtual Library - Academic scholarly directory created by Tim Berners-Lee. Individual sites maintained by international scholars.

Additional Links -- Topical Starting Points

(SOSIG) Social Science Information Gateway - subset of the Resource Discovery Network
Social Sciences Virtual Library
Subject-Specific Directories (Binghamton University) - academic level sites selected by BU bibliographers for the academic departments and programs on this campus.

TOP

Search Engines

What is a Search Engine?
Search Engine is comprised of 3 parts:

the computer program (spider, robot, worm) that roams Web looking at thousands of web pages gathering new documents. They detect changes in the pages, look at the links on those pages and download them to their database. The crawl schedule varies across all search engines. Even Google works off many servers and each Google crawler is on a different schedule.
the database of web pages created by this spider. This is what you are searching - the engine's database, NOT the entire web. Google's database of 4.2+ billion pages is claimed to be the larges database, but it is till much less than 1/2 of the entire web.
the interface is search engine software that you see and use to type keywords into the database. The engine automated indexer goes through the millions of pages in the database to find matches to your keywords and returns results based on relevancy using a complex mathematical algorithm. Each engine uses a different proprietary algorithm.

Search engines are not all the same

each uses different algorithm with different results
each differs in speed, search interface, frequency of updating (how often the spider travels the Web)

Over 2500 search services/search engines

When is it best to use a Search Engine?

When you have a very specific, narrowly defined topic
When your topic consists of more than one concept
When you are looking for a specific web page - organization, company, etc.

Selected Search Engines

alltheweb - Second behind Google in size. Similar search features of Google - searches 49 different languages, shows related searches. Searches some of the Deep Web.
Alta-Vista
Ask Jeeves
Google - largest database with specialty databases: Google Uncle Sam, Google Scholar, Google Books.
Teoma - Smaller that Google and Yahoo! Uses Subject-Specifi Popularity ranking. Can REFINE search with suggested terms and find RESEOURCES of directories and links to experts in the field.

No search engine indexes the entire Web

Additional Links:

Internet Tutorial (UC Berkeley) - Nice collection of web pages and handouts used in their workshops.
Internet Search Engines (SUNY Albany) - An up to date list of the newer engines and directories and their features.
Second Generation Searching on the Web (SUNY Albany)

TOP

Meta-Search Engines

What is a meta-search engine?

search engine that simultaneously searches multiple search engines
Good for obscure topics, but advanced search features are not available

When is it best to use a Meta-Search Engine?

When you have an obscure, difficult to find topic
When you are beginning a search on an unfamiliar topic
When you want a small number of sites
When you want to search several engines simultaneously
Dogpile
ProFusion
Query Server - breaks results into component parts, sites, domains
Vivisimo - Uses Clusty software to cluster the results into topics and subtopics

Additional Links: Meta-Search Engines (SUNY Albany)

TOP

Deep Web

content on the Web not visible to standard search engines
qualitatively different and superior than the surface Web pages
500x larger than the surface Web; estimated at 550 billion documents
password protected sites or those requiring a login
information stored in proprietary, commercial, legal, medical, and scientific databases - for example: phone books, patents, laws
pages that are displayed dynamically - airline flights, job postings, news, stock prices, auctions, etc.
government databases - .pdf files
non-html files - EXCEL, Word, PowerPoint, Flash, Shockwave, compressed files [.zip, .tar]
art and image databases, multi-media [audio, video, music] files [.gif, .tiff, etc.]
online catalogs
pages with frames
sites behind a firewall - Intranets
Good directories search the Deep Web - LII, INFOMINE, SOSIG (Social Science Information Gateway),
Some search engines already search the Deep Web - Google, alltheweb, ProFusion

Additional Links:

CompletePlanet - published a white paper on the scope of the deep Web.
Searchability - extensive annotated list of subject databases with background on what specialized databases are, who should use them, and how to search.
Subject-Specific Directories (Binghamton University)
Virtual Reference Collection (Binghamton University)

TOP

10 Steps to Effective Web Searching

Analyze your Topic to determine the kind of information you need - broad topic, specific document, statistics, primary sources, images
Choose the appropriate search tool
Identify key concepts (keywords) and synonyms, related terms, and spelling variations
- adolescent or adolescents or adolescence or teenager or teenage or teen
Use Boolean logic or Search Engine Math to construct search statements.
- + dogs
- +dogs -cats
- +"anorexia nervosa"
- adolescen* - retrieves adolescent, adolescents, adolescence
Use field searching - equivalent to Subject Searching in an online catalog
- URL:holocaust
- title:slavery
- site:nasa.gov
This varies with each search engine. Read the Help screens!
Enter 2-3 concepts in your search statement - "triangulation"
Use both a subject directory and a search engine or start with a meta search engine
Every search engine has a Basic and an Advanced search screen. Use both to refine your search. Often the Advanced screen is easier to use. Use the Help, search tips, and FAQs (Frequently Asked Questions) screens.
Limit your searches to domain names or type of document especially if you are interested in the scholarship on a topic. Limit to .edu or .gov.
Add or remove terms; truncate the URL from right to left to determine the source of the information and find other relevant sites.

HINT: Become proficient with 1 or 2 or 3 search engines. This is better than being a novice with many.
Additional Links:
- Finding Information on the Internet: A TUTORIAL (Teaching Library, California at Berkeley)
  Internet Tutorials (SUNY Albany)

TOP

Staying Current with Web Search Tools

The world of the Internet and the World Wide Web is an explosive one. Search tools and Web sites change frequently. Consult the following to stay on top of it all:

Researchbuzz - from Tara Calishan. Subscribe to a weekly newsletter
Resource Shelf - from Gary Price. Great compilation of news sites, search tips for online searchers. Subscribe to a weekly online newsletter.
Scout Report Subscribe to their weekly newsletter. There are specialized Scout reports for Business & Economy, Science and Engineering, and the Social Sciences & Humanities.
Search Engine Showdown - from Greg Notess
Search Engine Watch - from Danny Sullivan. Subscribe to a monthly newsletter.

TOP