| Index
Site Visit and Quarterly Reports send comments or questions to: l-neuma1@uiuc.edu |
User EvaluationSocial Science Research- Social Science TeamQuarterly Report (work from February through April, 1995) User-centered research related to testbed development continues to be conducted by faculty and students in the Graduate School of Library and Information Science, Sociology, and Economics. The primary focus in this quarter has been on testbed usability studies, observation of users of a variety of electronic media, and the development of system instrumentation mechanisms. As a warmup to the usability of the testbed prototype itself, a series of usability tests of the thesaurus component of the testbed (see below under Technology) were undertaken and summary results were presented to the thesaurus designer. The tests were conducted in the Grainger usability lab. Subjects (including a graduate assistant who works at the reference desk in the Grainger library, as well as with a graduate student, undergraduate, and faculty member in computer science and engineering) were given a brief demonstration of the system and were then asked to perform two thesaurus search tasks. The search sessions were captured online as well as videotaped. The researchers also wrote down the users questions and problems during the sessions and concluded each session with a series of questions about the users impressions of, and recommendations regarding, the prototype thesaurus. The usability tests revealed a general appreciation of the utility of the prototype thesaurus, but also identified problems that users had with the design of the thesaurus interface, including general difficulties with manipulating windows, insufficient labeling and instructions, and lack of clarity in the purpose of certain thesaurus functions. During this quarter the evaluation team also helped to draw up a preliminary plan for an experiment designed to compare the effectiveness of both the INSPEC and automatic thesaurae being developed for potential inclusion in the DLI testbed. The evaluation team is currently planning for the first stage of deployment of the prototype DLI testbed. Several public sites for deployment at the University of Illinois have been identified. The sites have been selected to allow the study of system use within both library and workteam environments, and across several engineering disciplines, by various segments of the academic engineering community, such as faculty members, graduate and undergraduate students, and librarians. The next quarter will see both continued usability testing of the prototype along with the collection of data about system usage in these actual work and learning situations through observations, interviews, and the solicitation of user feedback. Members of the evaluation team also began conducting a series of observations of the use of various electronic information resources and tools by students at nearby University High School. Some interviews with students and teachers are also being conducted. The purpose of these observations is to study the manner in which students obtain information through the use of a variety of computer applications. Both the computer labs and the library are serving as observation sites. Learning about the needs and habits of current high school students will give us insights into their future needs as college students; it will also suggest the potential utility and usability of DLI testbed for younger segments of the population. In addition, these observations provide an opportunity to investigate the use of a range of electronic resources (e.g., multimedia fulltext sources) relevant to development of functions and features of the DLI testbed. Another important activity of the evaluation team during this quarter was the development of a preliminary version of a DLI testbed user registration form and procedures for implementing it. User registration procedures are necessary to collect basic demographic data on testbed users and protect publishers against unauthorized use of their testbed materials. Some of the basic issues that need to be dealt with are the technical feasibility of automating the registration process, determining whether all testbed usage will require registration or whether unregistered usage will be permitted at certain public sites, and determining the manner in which registration forms will be processed and the data they contain stored and analyzed. The next steps will be to obtain consensus from system designers and publishers on the registration process, implement the registration form in the appropriate online application, and test the processes by which users are authenticated and data about them are collected. Instrumentation related to capturing data on testbed use also progressed during this quarter. The Grainger team has instrumented the current testbed prototype to record actions taken by users as they perform searches. The Mosaic team has produced a pilot instrumented version of that application as well, in consultation with the evaluation team. Work on designing an instrumented version of Panorama is just beginning; this instrumentation would be used to record the manner in which people display and read documents in the testbed collection. The next steps in the instrumentation effort will be to test the robustness and utility of the instrumented software. Careful consideration needs to be given to the nature and handling of what will be a huge store of usage data. In addition, the manner in which data from the different applications that make up the complete testbed will be integrated, over time, must be investigated. During this quarter, Bishop conducted an empirical investigation of the phenomenon of the virtual art gallery by conducting interviews with artists and museum administrators, implementing an online survey of visitors to several Internet art exhibits sponsored by the University of Illinois, and analyzing server statistics on the use of those exhibits (Bishop and Squier, 1995, in press). The study was not undertaken as a primary activity of the DLI evaluation team, but it is relevant in that it provided some exposure to online data collection methods. It also illuminates general user behavior of networked information services and stimulates thinking about new models for the presentation of information in networked multimedia environments. Bishop also conducted a comparative evaluation of seven different electronic journals to gain insights into current mechanisms for (and problems in) structuring, navigating, reading, and archiving scholarly material in electronic formats (Bishop, 1995, in press). The activities of the evaluation team during this quarter have led to a growing realization of the need for a theoretical framework to guide research on digital libraries and the nature of individual and group processes for creating and using knowledge. The realization stems from their own internal studies as well as from their interaction with other DLI evaluation researchers and their review of the current landscape for networked knowledge production. Thus, the evaluation team will concentrate on developing its own theoretical framework in the next quarter and will continue its efforts to share ideas and results with the broader research community. One important avenue for sharing results continues to be the dissemination of informal working papers via the project homepage. Co-PI Bishop continues as the evaluation team leader and Star continues as the technical lead for ethnography and research. In addition to their responsibility for evaluation research related to the University of Illinois project, they have played an important role in promoting evaluation efforts across the other DLI projects. Bishop serves as chairperson of the DLI- project-wide working group on user-centered evaluation and they have developed and received funding for a research symposium on user-centered digital library design and evaluation that will be held in October 1995. Star has participated in several national meetings related to furthering humanities and social science efforts for digital libraries, and Bishop will participate in a panel on the DLI initiative at the ACM's Computer Human Interaction (CHI- 95) conference in May as well as host a user research session at the Digital Library 95 conference in Austin this June. Greg Newby (Assistant Professor) and Bob Sandusky (doctoral student) in the Graduate School of Library and Information Science joined the evaluation team toward the end of this quarter. They will take the lead in the technical aspects of developing and implementing evaluation procedures which necessitate extensive online data collection and analysis, such as system instrumentation and the processing of user registration data. Publications Bishop, A. P. (1995, in press). Scholarly journals on the net: A reader’s assessment. Library Trends, 43 (4), 544- Bishop, A. P. and Squier, J. (1995, in press). Artists on the Internet. In INET ‘95 Proceedings. Public Presentations Bishop, A. and Mischo, W. Project briefing by Mischo and Bishop at the Coalition for Networked Information’s spring meeting on April 10, 1995. |