Analyzing Search Engines

ethan

1. Formulate five criteria for the evaluation of search engines

To effectively evaluate three different search engines from the perspective of an advanced web user, the following criteria were established:

1. Relevance and accuracy of search results

2. Search speed

3. Advanced search options

4. Other services

5. Site design and layout

More information about these criteria and how they were used is available below.


2. Test three search engines against your criteria

For this evaluation, the following three search engines were tested:

1. AltaVista - http://www.altavista.com/

2. Google - http://www.google.com/

3. DirectHit - http://www.directhit.com/


3. Explain your methodology

The criteria outlined in Part 1 were reached by considering the things that would be important to an advanced user. Of course, factors like Relevance and Speed are critical to such a user (particularly in a work environment). The search options criteria is particularly important to advanced users who are after a specific page and do not want to waste time 'wading' through results from a 'standard' search. The other criteria are important to most users and mean that the search engines are more usable and useful.


The search results from the three test search engines (Part 2) were evaluate according the criteria, explained it more detail below.


Criteria 1 - Relevance and Accuracy of Results


This criterion was to evaluate the usefulness of a particular search engine according to the accuracy of its results. I defined the relevance, according to personal standards, after browsing the pages in the results, for three separate search terms. The Relevance Percentage is calculated by counting the number of relevant pages that appear in the first 20 search 'hits' and then multiplying by 20 (for a percentage).


Criteria 2 - Search Speed


This criterion was to evaluate which of the test search engines returns results quicker.

However, it is important to note that the speed test is both a test of the Internet connection to the remote machine, as well as the time taken by the search engine to perform the search. Furthermore, tests of Internet speeds are very variable based on time of day, other internet traffic etc and are therefore of only limited accuracy. For this reason, no 'hard numbers' were calculated (ie timed searches). Instead, I rated each search engine based on 5 searches per engine at the same time, same day. The possible Speed Ratings are as follows: Excellent, Good, Poor


Criteria 3 - Search Options


This criteria was to evaluate the search engines advanced search options. This is important for advanced web users because it allows them to track down specific pages, and weed out irrelevant pages from their searches, meaning they find the most relevant pages quickly.


In evaluating this criteria, the three test search engines were tested to see if they had the following search options:

- Variable number of results per page

- Search for "phrases"

- Boolean Searches (AND, OR, NOT)

- Wildcard (*) searches

- Search for text in only title or body

- Search for pages in different languages

- Search for pages located in specific countries

- Search for pages under a certain domain

- Search for pages containing links to a URL (eg link:http://www.yahoo.com/)

- Search for pages LIKE a certain page (eg like:http://www.yahoo.com/)


The number of search options each search engine had available was calculated and then recorded out of 10.


Criteria 4 - Additional Services


This criteria was to evaluate the other services offered by the search engine to make web searching and browsing easier for advanced users. These features make the search engines more useful and more of a 'one-stop destination' for finding information, including news, sport results, TV guides, weather reports and other information in a 'start page' design to allow the user to easily access the information he needs quickly. These services are particularly relevant to the advanced and regular web users because they use the web frequently, making such a page very convenient.


In evaluating this criteria, the three test search engines were tested to see if they offered the following additional services/features:

- Online shopping guide

- Suggested searches. Searches based on what you have searched for

- Other searches - whitepages, yellowpages, maps/directions etc

- Hand-picked categories or channels for web browsing

- Personalisation of search results, layout, information etc


The number of additional services offered by each of the test search engines was calculated and then recorded out of a score of 5.


Criteria 5 - Page Design and Layout


This criteria was to evaluate the usefulness and usability of the page design and layouts of the search engines. This criteria is important for