DLI social science team

DLI Social Science Team Home Page

send comments or questions to: l-neuma1@uiuc.edu

Usability technical report- DeLIver

29 May 98

Prepared by Laura J. Neumann and Madonnalisa Gonzales

I. Introduction

We are in our final year for the DLI project and as part of an independent study on how to conduct research and at the same time learn more about the use of the DeLIver system, Madonnalisa requested a task on the project. Laura worked with her to accomplish this. The purpose of this report is to examine the use of the system in relation to work practices. We also wanted to know if there were still some usability issues that users faced while using the system. This report is a summary of our findings based on 12 interviews of professors and graduate students who conducted searches for their research.

DeLIver had been generally available for 4- 5 months when interviews were conducted. At that time we had about 738 web client users, defined as not working for the project nor working at the library where the project is centered. The system had been primarily advertised through posters in libraries and a flier sent to all graduate students and faculty in engineering, computer science, physics and other related disciplines.

The report contains screen by screen usability comments and information from the interviews and some of the transactions logs. We hope that this report will aid the development team in refining the organization of the interface that answers many of the general comments made about the system.

II. Method

In this set of interviews, we wanted to explore both usability issues- what difficulties, if any, people had using our system, as well as the context of use- for what reasons and how people were using our system. With these dual goals in mind, we decided to use semi- structured interviews, and constructed an interview guide that would get at both sets of issues (see section V). The interview guide is organized according to the different pages and links in our system, and generally in the order that people might see them.

Our pool of people was a stratified non- random sample generated from a list of registered users. We selected categories of users we most wanted to talk to, based on the system's target audience and past data collection efforts. These were divided by field of research (Civil Engineering, Electronic and Computer Engineering, Physics, and Mechanical Engineering, with a few people in the "Other" category.) We wanted to talk to mainly professors and graduate students who had used the system anywhere from once or twice to fifty times, and selected people with an eye toward those who would interact with our system in manner that suggested that they wanted some results from our system. Our project has gained some campus visibility, and our interface has been used and demonstrated in classrooms which focus on usability issues. These people were deliberately excluded. A total of 45 people were contacted via e-mail and phone, and 2 people were removed from the sample because they did not fit our selection criteria.

Twelve interviews were usually conducted in each person's office, when at all possible. This gave us a glimpse at the resources that each person had gathered around them and gave us an opening to ask about how they organized and used other information resources. Two team members, Laura Neumann and Lisa Gonzales conducted the first interviews together, as Lisa learned the process. All the interviews were tape recorded, and detailed notes were generated by both Laura and Lisa through re- listening to the tapes. Lisa then did the last few interviews by herself.

After the detailed notes were all generated, memos were generated (see section VI) to pull out both usability and context of use comments. Laura and Lisa both worked on these. After another team member pulled the transaction logs for all the sessions to date for each of the people interviewed, Laura went through, line by line, six of these to see if she could see any of the usability and use issues reflected in these data. Actions on the system were summarized and added to the memos. Comparisons between interview comments and transaction logs were made.

Finally, the memos were used to generate frequencies of actions as well as this summary report.

These interviews are an interesting hybrid of both general interviews done previously by one team member and usability tests done previously by the team. As stated previously, the goal was to find out both about problems people were having with the system and what they were using the system for—to get at issues behind what was appearing in the transaction logs, and to get more detailed information than is possible through a large survey. We found that it is difficult to get at the details of both of these during one hour long interview. Most interviews either have more in- depth usability comments or more in- depth use comments, but not both. However, there were enough interviews that leaned toward either one so that there was a reasonable volume of information on each. Then, the analysis of a half of the transaction log records for these people further informed what people were telling us about what they did and did not do with the system. Usability issues, which were not mentioned in the interviews came to light-- such as the fact that people generally experimented with how to enter terms and names. We could also get a feel for how frequently people performed the actions they mentioned, such as using links to meta- data in other systems. Generally, we found the combination of usability and use questions, as well as the analysis of transaction logs to be a useful triangulation of available data.

III Results

A. Login Page:

No one mentioned that they had a problem. One guy commented that he didn't notice the system was free at first and thought he would be getting a bill for accessing it. Another mentioned that he noticed when the second window for password went away. When we saw him using the system, he typed his password instead of user name.

B. Downloading/ Software:

There were various problems that the users encountered in using the system, anywhere from platform conflicts of those who use Unix workstation to those who only use Microsoft Internet Explorer. Others had problems with the fact that the system was slow and the various pages to encounter just to download the SGML viewer.

1. Panorama-

There were a couple people who couldn't figure out which link on that page would download the software—one guy actually went to Panorama's HP and got a copy of the program that expired. He figured he couldn't use the system anymore because he didn't want to pay.

There were 7 volunteers who claimed that they used the download SGML page. Of the 7 users there were 6 comments and 1 no comment. There were two who were not asked this question and 3 who stated that they did not use the system.

Of the 6 who did comment on the system, 10 negative comments were made. There were no positive comments made on this system screen.

It was also found that the primary problem with the download was that the user lacked the knowledge of figuring out how and which software to download and how it would affect their computer.

For example one user was concerned about their computer and stated that "I don't want to have to [download the software]…my computer is working right now so I don't want to spoil anything." Also the user was faced with the problem of figuring out why they would have to use a particular browser to do the download. "Why Netscape? I always use that Explorer. Is there a difference? I don't see why it should matter, they are both for the web."

2. Adobe Acrobat-

There were not that many problems encountered with the downloading of Adobe Acrobat because many of the computer systems already had the software installed. Many of the users did not make any negative comments of the download page. The only problem that did occur was not necessarily related to the Deliver system, but rather a lack of understanding of how to download software. He was also concerned that it was unclear where to click to actually get the program.

Nine users claimed that they did use or saw the download page. 6 claimed that they did not use the system. 3 stated that they already had the software and there were 4 people who commented that they had some kind of problem.

C. Standard Link Bar: tips/help/other resources/browse

(freqs of people who SAID they used browse, related resources, tips and help) While people said they did not use browse, in the TL analyzed, we saw people using what WE would call browse, but that they would NOT call browse. Vocabulary mismatch— (people can access the "browse" via "about deliver")

Five people stated that they did use the browse section of the link bar. 3 stated that they never saw or used the link bar. There were various comments about browsing but no direct comments about the help pages. One user stated that "I am never patient enough to read the tips that tell you how to do a real search." Some just looked at the browse just to see which journals were available on the system.

One problem that a user encountered is that he had trouble returning to the DeLIver system after venturing into the web page of one of the publishers through the browse.

Transaction Logs:

From the transaction logs we did discover that there were some who did use read the help/tips screen as well as go into browsing although they would not have called what they did as browsing.

We found that those who did say in the interview that they browsed the journal was also reflected in the TL.

One user who stated that he did not use the help or tips portion of the standard link bar did in fact look at the tip 3 (with the and/or/but not gifs) and tip 8.

Another interviewee stated that he did not use the link bar, but from the TL we saw that he look at the "about DeLIver" information pages and the publisher's pages.

D. Search Screen:

Two people stated that they never even saw this page, and that they primarily used the browse feature of the system.

1. Input box –

Eight people users stated that they used the system. 6 made comments and 2 did not comment. Of the six there was only one negative comment while there were 5 positive comments. 4 stated that they did not encounter this page.

One user, a graduate student, talked about using the system in "unorthodox way" and wanted to persuade a friend to stay at this department by showing him that of the departments in his area and the field, University of Illinois publishes the most articles. From the transaction logs we did find that he did a dozen searches using the author's affiliation and a keyword search of their field to conduct this search.

The unfavorable comment made about the system was from a user who stated that "knowing where you can click and not click is a problem. Can't you make that more clear? When you look at it, this just looks like ordinary text. Maybe change the font? …Just knowing what is clickable and what is not."

Many of the users stated that they did searches primarily using keywords that were from suggested or heard from various sources including advisors, previous readings, or conference talks.

Transaction Logs:

One of the users did "mess around" with the system. It appeared that he was trying to experiment on how to enter the terms.:

He ran searches in both upper/lower case of author's name.
He used quotes around search term more than once.
Entered string in multiple ways e.g.: Urbana at Champaign, Urbana Champaign, Urbana-Champaign, Urbana- Champaign.

There was a similar instance in which the user did some experimenting with upper and lower case letters, and tried searching on author name several difference ways with and without initials.

Another user seemed to try different searches first by entering 2 words in first box, then one word in 2 boxes and using "AND."

2. Areas of article freq.:

There were seven comments made about the system. 6 were positive and one was unfavorable. This one user's negative comment was related to how "I don't think I've changed anything else…the equations are going too much into detail. It doesn't matter if it appears in the abstract or introduction. It's not useful for me." On the other hand, one user stated that "What I like about it is the many ways I can look for things." There were 3 users who did not comment.

There was only one user who was no asked and one who did not use it.

3. Boolean freq.:

There were 9 users who did state they used the system of which 6 made comments and 3 did not comment on anything specifically. 3 stated that they did not use the features. Of the 6 usability comments, 5 were positive and only one had unfavorable sentiments about "an uncertain about entering 2 words."

There was one user who stated that he did use the Boolean features, "but I am never patient enough to read the tips that tell you how to do a real search." This comment reveals that the user was aware that online help was available but that not being patient to read them could imply that he would not bother to read them because it's possibly cumbersome. Perhaps using the Boolean section of the page is not immediately apparent for its probably use.

One user did state that he overcame his problem of having too many results is "by refining the search using the Boolean boxes and using also must contain."

Transaction Logs:

User did use "and" , "or" and was the same person who had looked that tips explaining the Boolean terms.

Another user stated that he did use the Boolean feature, but from the TL we discovered it was not often and what he had done was create the boolean expression mostly by putting multiple terms in the first box.

One user stated that he had no problems using "must also include" but we did notice that he used "not" too. When he turns on an "and", he does not turn it off, even if he erases the search terms from the 2nd and 3rd boxes.

There was one case in which the user did 6 searches using "and" and on one occasion, he entered a term in the second box but chose no Boolean operator.

3. Publishers

There were 8 users who did use this feature. Five people stated comments while 3 did use it but did not comment. Two users were not asked and 2 did not use the feature.

Of the comments, 1 was positive and 3 were negative. Some comments were related to how "sometimes you don't know where they [the publisher] are going to be publishing." The user in this case did not use the feature, but echoes similar sentiments by other users.

Another comment was that a user "couldn't imagine wanting to select by publisher."

One user stated that "this is probably the least thing I wold be interested in." Transaction Logs:

From the TL we found that our sample never changed the default setting for the publishers.

4. Search button

There was one comment on this about how one user hated to scroll down every time. He stated that

"I always forget the 'perform search' button is down there at the bottom."

"Why can't you put the perform search at the top of the page? Or a second button up there? That way I won't have to scroll down to the bottom all of the time. I know it and the other stuff is down there, but I don't want to always have to do to the bottom of the page."

This was a common problem that we have faced in other usability studies and this user's experience is not different.

Mostly there were no comments and only one user did not know this screen existed so he did not have anything to say about this button.

EG: they guy who uses Deliver from home because it has FT—he wanted an article by some people he was collaborating with and about to talk to. A student who needed a couple of sources for a paper because he needed FT late at night for something due the next day.

E. Results screen

There were 9 users who stated that in some way they had seen the results screen and used it. There were 9 comments, 2 people did not use it, and one user was not asked because they had never seen the page. Of the 9 comments, 7 were positive and 2 were negative.

The types of problems that the users encountered include the slow download, the user wasn't sure whether or not to use these or the back arrows on the browser, one user stated he had a workstation and was unable to look at full text link from this page ("That is a problem because I have a workstation here on my desk and it does not look as if I can get a look at them."), another comment was that he tells his students to figure out the reader and to take care of it, this same user had difficult looking at the compendex citation when he clicked on it and got an error message. One user stated that it was all confusing, meaning that with all the link choices, he opted to only click on the Inspec citation because of its familiarity.

Many stated that they look at the title or go straight to the full text of the citation, some mentioned scanning the list for their authors.

F. Extended citation

1. Citation:

There were 9 users of this screen. 6 made comments and 3 did not. All of the 6 comments were positive and 3 stated that they did not use this screen, in these cases, the user never reached this screen because they did not reach the search screen in the first place or they were not interested in this feature.

Many of the users stated that they used the extended citation to access the abstract. Two users stated that they liked to look over the figures and one other stated that he "found the references citing him. Saw the citation and read the abstracts." A couple mentioned that from this page they would go to the full article to skim over the article.

2. Authors:

There were 8 users of this screen but only 4 made comments. They ranged from looking at the authors because they wanted to see what the range of work the author is most familiar to be work with as well as "looking for author's names of people I know."

Four users did not comment on the system and 3 stated that they did not see the screen or did not use the screen.

3. Abstract:

There were 8 users of the system, of which 5 made comments and 3 made no comments. 3 stated that they did not use the screen and all of the comments made on the system were positive. Those who did not use this screen were those who did not even reach the search screen.

Some comments were made about how it helps how to understand new concepts in their field for example: "That is the first thing. And I would be looking for those keywords…the themes or the subjects that I would hope be there. You can see what the author put in, and sometimes when I'm not so familiar with a subject I can see what other keywords I should have used." This is obviously a combination of the use of parts of the extended citation for getting a glimpse of new research. Another comment similar to this stated that "…sometimes in the abstract, it requires you to use your imagination to figure out what they did…I can just skim through and look at the figures and find out what they did."

There were no negative comments made about looking at the abstract through the extended citation.

4. Figures:

There were 3 comments and 4 users who did not comment and 4 users who did not use the system. Of the comments made, 3 were negative and 1 was positive. Of the problems, we found out that one of the people who stated he would not use this feature was related to reading out of context.: "I am trying to support something and the figures are what I need, and I will get those from an article, but first I need the article."

One of the users did state that "In this case, it would be hard for me to interpret one of these figures without seeing the text because I am looking at a paper about this new thing that I don't know much about. If I am looking at a paper by one of my close colleagues, that is in exactly the same field that I work in, then I would have a pretty good idea what the figure is about. Then it would be really useful for me to look at these figures. I would understand them."

Another stated that he wanted the figures in order to support his own work and that the figures helped his own research because it validated his work.

One problem is that one of the users encountered error messages when trying to access the graphics. Apparently, he claimed that he didn't think he had the correct graphics package to see the figures. He feels "that it should be made clear that you must have what ever program." One other problem about figures was that a user had print out the whole article, "and then it was a little complicated when I had to look at the figures, because you had to print them separately."

5. References:

Many of the comments made on the references echoed those of viewing the figures which was primarily the difficulty of taking the references out of context. One of the users stated that he looks "at mostly the references section only after I look at the article, it's difficult to take out of context and that goes for tables ad figures."

There were 8 users of this section of this screen, 6 made comments and 2 made no comments, while 4 users stated that they did not use it. Many of the comments were positive in regards to the active links in the references for the full text article that was also available on the system.

G. SGML Full Text:

Of the 5 who did access the SGML FT, 4 made comments and one did not. 7 total did not access the FT due to various reasons ranging from having problems downloading the viewer or because they had to print the figures separately and "it was a little complicated."

There was one user who stated that he looked through the article and then printed it, and another stated that "There is a level of interactivity that is not even available with the paper journals- being able to move back and forth between these parts."

H. PDF FT:

There were 8 users of this feature. Of the 8, 5 made comments while only 3 made comments. 4 stated they did not use the system.

Two of the 5 comments were positive comments: One person stated that he "likes that it comes out 'beautifully'- especially the equations and figures. From our research of this comment, we came to the conclusion that it was not the software itself, but that his computer may or monitor had been the problem. Some positive comments are that a user did not have any problems, "it's pretty easy. PDF any day, looks lot nicer, and I can use it on my workstation." Having a workstation was a common issue for some of the users who could not access the SGML full text. There is an SGML panorama available, but apparently it is not for the specific workstation that the user had as one had done some of his own research.

Another stated that "overall I like it because I've used the PDF format. I like it because I've used the pdf format." This same person also stated that he had one complaint, "there's something strange with scaling of the document, that makes it difficult to use. There's something that messages up that article." One negative comment had to do with having to print them because "the main thing is I don't like to read articles off the computer screen."

I. Compendex/ Inspec

One user said that when he first got the results screen, he would first check the inspec citation. From the transaction log of this user we found that he went into the compendex link 6 times of the 4 sessions we logged.

Another user had difficulty looking at the compendex citation, he would get the error message "no compendex citation" or something like that. "there is no information on inspec for this…it happens to me a lot."

An interesting use of the inspec link was that "I only clicked on the inspec citation. It's too confusing…that is why I stick to the abstract." From this comment, we could conclude that from all the links of the results, he felt that he would stick to what was familiar and that would be the inspec link. From the transaction log the user did use the extended citation page but did not click on the inspec citation. It's difficult to tell between the interview data and the transaction logs, that this user may had some difficulty remembering what he did and did not do on the system.

Transaction Logs:

We found that one user did use the compendex link 6 times. But from a broader picture, this user did click on the extended citation more frequently than the inspec/compendex link which is contrary to what he stated as his order of browsing: "First I would check the inspec citation. Then I would go for the extended citation." Obviously with a 14 time frequency of use for the extended citation as compared to the 6 times for the compendex link, he definitely used the extended cite more.

J. Word in context:

In the interviews, we found that 2 people state that they used this feature. 3 were not asked because they never did encounter the search screen. 7 stated that they did not use the system. Of the positive comments, one stated that "I used word in context when I need an idea of how the term is used…when I am not sure of the subject and I need to know for future searches." Another person plainly stated, "That looks good. That's pretty neat."

There was one user who stated that he did not use the word in context, but that he "usually I'm not looking for specific work, I'm looking for related."

Transation Logs:

There was only one instance in which a user did use word in context 6 times while other users used it once or twice. Other than that, nothing significant to users in general from the TL.

IV: general comments on the system

There were various types of comments that were all related to how the volunteers conducted their own search. Trying to implement all these "wish list" requests would be unreasonable, but in some way these reasons also hint to requesting an interface or a system that is customizable.

A common issue that echoed from seven of the users of the system was that they wanted bigger collections, more journals in their field including ACM publications, and one stated that he wanted some international journals such from Japan and Europe.

Another factor in the lack of enthusiasm to use this system, was the currency of material on the system was very important for many of the users. Some of the professor expressed concern over how long it takes for the articles to become available on DeLIver while they can access such sources as Los Alamos pre-print server, while others say they go straight to the author's web page or the home page of the journal's publishers for subscription to table of contents before the journals are published.