Diary

Internal Reports

Completed Papers

Papers in Progress

Conference Presentations

Site Visit and Quarterly Reports

Main DLI Page

Web Client- DeLIver

Summary of Usability Tests, Version 1 of DLI testbed (custom client) (Feb. 1996)

I. Method

In consultation with the Testbed team, we prepared 10 different tasks which tested the functions of the interface. Nine were simple tasks designed to test the prototype's search capabilities, and one was a story problem involving retrieval and manipulation of a full text article with Panorama. We told each participant to work on the simple tasks for 20 minutes and the story problem for 15 minutes. They were encouraged to talk aloud as they worked, so that the note takers would be able to record their thoughts. Their actions were recorded by a screen capture, and their comments were audio taped. All participants signed consent forms before the session began.

One note taker sat next to the participant to answer any questions that the participant asked. At least one other note taker sat behind the participant to record all of their actions. This set up increased the accuracy of our note taking. After each session, the participant was debriefed with general questions about the testbed and the usability session (see Appendix A for a copy of the usability instrument, including both the 10 usability tasks and the debriefing questions).

We recruited 10 participants from several different Physics research teams. Five participants are graduate students, two are professors, two are research programmers, and one is a post doc. Each session lasted one hour.

II. Task Summary

Of the ten participants, only one completed all of the tasks before the debriefing session. Two participants tried the first six of the simple tasks and the story problem. Four tried the first four tasks and the story problem. Two skipped around in the first set of tasks and completed the story problem. One answered the first four tasks but did not have time to try the story problem, and one answered three tasks but did not have time for the story problem.

Task 1:         10 participants 

Task 2:         10 

Task 3:         10 

Task 4:         10 

Task 5:         4 

Task 6:         3 

Task 7:         1 

Task 8:         2

Task 9:         1 

Part II Task Attempted:

8 of 10 participants (one did not have time to complete the task. The other could not try it because the system had crashed.)

Task 1:                 2 participants 

Task 2:                 2 

Task 3:                 4 

Task 4:                 4 (two gave up completely) 

Task 5:                 2 

Task 6:                 1 

Task 7:                 1 

Task 8:                 0 

Task 9:                 0 

Story problem           3

Task 1:                 1 participant 

Task 2:                 3 

Task 3:                 2 (both completely gave up on the task) 

Task 4:                 5 (three gave up completely) 

Task 5:                 1 

Task 6:                 0 

Task 7:                 0 

Task 8:                 0 

Task 9:                 0 

Story problem:          1 

Task 1:                 3 participants 

Task 2:                 3 

Task 3:                 2 

Task 4:                 1 

Task 5:                 1 

Task 6:                 2 

Task 7:                 0 

Task 8:                 1 

Task 9:                 1 

Story problem           1

Task 1:                         4 participants 

Task 2:                         2 

Task 3:                         2 

Task 4:                         0 

Task 5:                         3 

Task 6:                         0 

Task 7:                         0 

Task 8:                         1 

Task 9:                         0 

Story problem:          3 

Task 1: Those who were unsuccessful did not know how to deal with initials, punctuation, or capitalization

Task 2: did know how to change the fields searched. One asked us what search form he should use to complete this task. One hit "enter" key after typing in only one item. One typed both terms in one box.

Task 3: Five never realized they did an 'or' search. Two just gave up.

Task 4: Three had problems changing the search options. Two did not know how to combine a name and a subject. Five did not how to enter an author name in the search full text form.

Task 5: There was a bug that prevented two from changing the search options.

Task 6: One didn't know how to change the options.

Task 7: Had problems entering phrase in the search box. Put entire title on one box.

Task 8: no problems

Task 9: no problems

Story problem: Had problems entering terms (they did not notice their spelling errors). Four did not know how search for an author and a term. One wanted to do part one of the search (e.g., search for Kumar) and then modify his search. Two had trouble changing the search options. Four still did not know what the icons next to the figure captions meant. Two did not know how to get from the figure to the text.

Task 1: did not know how to deal with initials, punctuation, or capitalization

Task 2: One asked us what search form he should use to complete this task. One asked us how to exit the screen from the previous task (e.g., Author search form).

Task 3: Two did not know how to combine an author name and a term.

Task 4: One asked how to combine a name and a subject. Two asked how to enter an author name in the search full text form. One asked how to change the search options. One didn't understand the wording on the top level (the main menu); didn't understand what the buttons were for.

Task 5: How do you enter "MIT"?

Task 6: One asked: "How do you search by page number? Is it like a journal?"

Task 7: no one asked for help

Task 8: One asked how to find the affiliation of the person. One asked how to deal with punctuation ("Do you put Austin, Texas?")

Task 9: no one asked for help

Story problem: Had problems entering terms (they did not notice their spelling errors). Four did not know how search for an author and a term. One wanted to do part one of the search (e.g., search for Kumar) and then modify his search. Two had trouble changing the search options. Four still did not know what the icons next to the figure captions meant. Two did not know how to get from the figure to the text.

III. Major Findings

[Note: * indicates the % of those who were able to see the full text of a document]
 
Problem:  Difficult to find And/Or/Erase All Terms buttons
% of people: 100
Suggestions: (100% suggested this)  put those buttons above black, horizontal line so it's grouped together with search boxes.
  
Problem:  Scroll box next to "Add" button contains only "or"
% of people:  100
Suggestions: put "and" option in there too
  

Problem:  Icons next to figure captions in full text are not clear ("why a globe?")  (two participants thought the globe would take them to the world wide web)
% of people:  100
Suggestions:
  

Problem:  General confusion about seeming inconsistencies in windows and pup-up menus, e.g., pop-up menus didn't contain all options, windows didn't seem to function as windows (with scroll bars, resizing, etc.)
% of people:  100
Suggestions:  
  

Problem:  Not clear how to change search fields when in full text search mode;  thought they could use pop- up menu to choose/ change fields, but in fact had to double-click to bring up window where fields could be selected, revised.  Very tedious to unclick all unwanted fields.
% of people:  80
Suggestions:  Have one step for exhaustive list of fields;  indicate which are active.
  

Problem:  Font too small, difficult to read.
% of people:  80
Suggestions:  Make it bigger, change style
  

Problem:  "Exit" is too final; means you want to get out of the system.  Short entry window especially needs clear option to close it.
% people:  70
Suggestions:  Add or replace "Exit" with:  Done (10%); Clear (30%); Finished (10%);  Go Back (10%);  Start Over (10%); New Search (10%)
  

Problem:  Hints not helpful
% people:  70
Suggestions:  Put in example searches, more information.
  

Problem:  Color contrast needs to be changed;  difficult to read.  Particularly the white on black highlighting (40%);  red on gray on the perform search button (30%);  background aqua blue color too bright (20%).
% of people:  70
Suggestions:  coordinate color of screens with color of buttons to get into them; reduce number of colors overall.
  

Problem:  All forms look alike
% of people:  60
Suggestions:  coordinate color of screens with color of buttons to get into them.
  

Problem:  Buttons, icons not "ergonomically correct"
% of people:  60  
Suggestions:  rearrange for consistency, importance,  and function groupings, see as follows:
Problem:  And/ Or/ Erase buttons should all be above black, horizontal line in the search form.
% people:  100
Problem:  Perform Search button (in author search form) should be big and underneath the search fields.
% of people:  30
Problem:  Exit buttons are in different places in each search form.
% of people:  10
Problem:  Hourglass won't go away, even though you could still continue using the system;  made people wait
% of people:  40
  

Problem:  Window that asks to limit search is unnecessary, inconvenient at time it appears, contains no new information, changes must be made in search form anyway, covers other relevant information.
% of people:  50
Suggestion:  Should only appear at end of multiple term search in limited circumstances (not always)
  

Problem:  In author search form, check boxes on right side ("authors listed in...") do not indicate whether they are "or" or "and" searches.
% of people:  50 (50% of people thought they did an "and" search using those boxes)
Suggestions:  make both possible or indicate what is happening
  

Problem:  system said "Searching... click to entry mode".  This confused people.
% of people:  50
Suggestions:
  

Problem:  Titles in display window are in no apparent order and/ or do not contain enough citation information to decide whether to view short entry (very tedious to call up short entries one-by-one for items that may not be at all relevant.)
% of people:  50
Suggestions:  Titles should be displayed in chronological order (20%), have journal names next to titles (20%), have authors' names next to it (10%), be numbered (10%).
  

Problem:  Box that appears when you want to go from figure to text contains "gibberish".
% of people:  40
Suggestions:  Add phrase from text where that figure is mentioned.
  

Problem:  Colored and underlined items imply hyper links; but, many items (see top level screen) are not links.
% of people:  40
Suggestions:  Make clear distinction for links.
  

Problem:  In short entry form, captions are not useful without the figures themselves.  
% of people:  30  
Suggestions:  make hyper links from short form to figures.  
  

Problem:  Terminology in system unclear.  "Section title"- thought it meant section of a book (10%);  "Search key areas"- thought it meant subject search (10%);  "enter search term"- not sure if this included author or not (10%);  "F1" not clear (30%)  
% of people:  30  
Suggestions:  
  

Problem:  Captions should appear underneath displayed figure.  
% of people:  30  
Suggestions: captions should appear underneath displayed figure.  
  

Problem:  Title in display box highlighted twice, is redundant.
% of people:  20
Suggestions:  either highlight or make it appear in upper box, not both.
  

Problem:  Not clear why arrow next to author's name in full text goes to e-mail address ("I would never have guessed that in a million years.")
% of people:  20  (most people never noticed it.)
Suggestions:  change to a more intuitive icon.
  

Problem:  In window to change fields, did not find "author's last name" easily.
% of people:  10
Suggestions:  items in this window should be listed in order of importance, frequency of use.
  

Problem:  Shouldn't be able to scroll past abstract/ bibliography in short entry form;  i.e., shouldn't see so much white space at end of these fields.
% of people:  1
Suggestions:  make field as long as entry.
  

Problem:  Not clear what icon in front of search results is for. ("what is this, a cookie??")
% of people:  10
Suggestions:  make icon more intuitive, clear.
  

IV. Wish List

Wish:  Double clicking on title (in displayed results) should go directly to short entry.
% of people:  90
  

Wish:  Double clicking on caption (in short entry) should go directly to figure.
% of people:  50
  

Wish:  Hyper link from "F1" in full text to figure (bypass "globe" and figure list).
% of people:  50
  

Wish:  Box to check for case sensitivity (yes or no).
% of people:  40
  

Wish:  Figures embedded within text.
% of people:  40
  

Wish:  Modify search in Author Search form;  allow subject and author from this form.
% of people:  40
  

Wish:  Highlight term searched on in bibliography and/ or all fields where it appears.
% of people:  40
  

Wish:  Tab to next entry box in author search form (from "first name" box to "last name" box).
% of people:  30
  

Wish:  Type whole author name on one line in author search form.
% of people:  30
  

Wish:  Link from bibliography to other articles.
% of people:  30
  

Wish:  Perform Search button in author form should be big and underneath search entry boxes.
% of people:  20
  

Wish:  Subheadings (to demarcate different sections of the article) in full text display.
% of people:  20
  

Wish:  Make it possible to search within the bibliography.
% of people:  20
  

Wish:  Make it possible to customize windows;  wants to change font, and make windows bigger
% of people:  10
  

Wish:  Link to other databases.
% of people:  10
  

Wish:  See who cited article you are looking at.
% of people:  10
  

Wish:  Journal information in customizable format so can cut and paste into personal bibliography.
% of people:  10
  

Wish:  Zoom in function in full text so can see the layout of whole paper, pages.
% of people: 10
  

Wish:  Article should scroll when you scroll.
% of people:  10
  

Wish:  Ability to use this search engine/ interface when linked to other databases on web.
% of people:  10
  

V. Best Things About the System

Best item:  Ability to access full text
% of people:  60
  

Best item:  Choices on top level for different options
% of people:  30
  

Best item:  Responsiveness/ quickness of system.
% of people:  30
  

Best item:  Ability to access abstract.
% of people:  20
  

Best item:  Print out capability
% of people:  20
  

Best item:  Search fractions of article, (e.g. figure captions).
% of people:  20
  

Best item:  Figures included.
% of people:  10
  

Places where Instructions/ Examples would be Helpful

Many users had the same kind of uncertainties about their interaction with the testbed at particular stages. Based on these, we think the following are good junctures for further explanation or examples:

- Examples of how to formulate search terms, e.g., how author names should be entered

- Explanation of whether or not system is case-sensitive

- Clearer explanation of how to change search fields

- Clearer descriptions of what the top level search button options accomplish

APPENDIX A: INSTRUMENT FOR USABILITY TESTS

Date:

ID:

Intro.

(5 mins.)

We'll explain the usability lab set-up, the purpose of this study, and your role in identifying problems with the DLI prototype. We'll also give you a form establishing your informed consent to participate in this study. Try to perform as many of the searches listed below as you can comfortably get through in about 20 minutes (This is NOT a test of your speed, skills, etc., but a test of the capabilities and ease of use of the interface, so don't worry if you only make it few several of the tasks). Do not retrieve the full text of any of the articles. After trying each search, please remark on how important you think that type of search is.

Search Tasks

(20 mins.)

1. Find all articles written by N. Yoshida.

2. Find all articles with "GaAs" or "InP" in either the title or abstract.

3. Find all articles in which C. J. Summers appears as both an author and someone cited in the bibliography.

4. Find all articles written by Nick Holonyak that include the term "Bragg reflectors" anywhere in the article.

5. Find all articles written by someone at MIT (Massachusetts Institute of Technology).

6. Try to find all articles that are on a particular subject. Select a topic you're familiar with, or try one of the following:

-- molecular beam epitaxy -- quantum well lasers -- Hall Effect

7. Find a paper written by someone named Chuang that includes the book "Optoelectronics and Integrated Optics" in its bibliography.

8. You met an interesting guy at a conference and you want to see what he's written. Try to find his articles. All you remember is that his last name is Chen and he works somewhere in Austin, Texas.

9. Try to find all articles that cite someone at Cornell

Retrieving and Reading an Article

(15 mins.)

1. Yesterday, someone gave you a copy of a really neat figure. You asked for the reference, but the person didn't have it. All she knew was that the author's name was Kumar. The caption of the figure reads: "Absorption spectrum of 9740 in 1,2-dicloroethane" Find and display the full-text of the article.

-- Retrieve the figure in question

-- Then return to the place in the text where that figure is first mentioned

-- As you skim the text on-line, pause to read endnote 5

-- Then go back and skim the article's conclusions

-- Finally, return to search mode, as if you wanted to look for a different article in the DLI testbed.

THANK YOU FOR YOUR HELP

Wrap-Up (20 mins.)

Please offer any comments you have about the DLI testbed. We may ask you to comment on specific actions you took during this session. We'd also like your opinion on the following points:

1. What was the best thing about this prototype system?

2. What was the worst? What caused you the most problems?

3. What most needs changing or improvement?

4. Do you have any other suggestions for improving the system's functionality or interface?

5. Do you have any comments on how the testbed looks/ its appearance to you?

6. What did you think of this usability test session?