The role of user evaluation in design and ongoing development of a WWW browser-based interface to library and networked information resources at the University of Western Australia

Over a period of two years, the Library at the University of Western Australia transformed the interface between users and library information resources from a simple on-line public access catalogue to a flexible, complex, and dynamic networked information resource (CygNET Online, .. Throughout this period, the development team monitored user response to the developing resource. This paper focuses on the methods used for formal evaluation of user response at two key stages: during a pilot implementation, and after one year’s implementation of the new resource across the full library system. The paper is illustrated with extracts from the evaluation instruments. The formal evaluations were based on an established instrument for measurement of interface usability, the QUIS (developed by the Human-Computer Interaction Laboratory at the University of Maryland). The QUIS was combined with instruments to measure users' confidence and attitudes to use of CygNET Online. Additional items were included to confirm, from a larger sample of users, observations made by individuals in e-mailed comments and informal discussions. Users were also asked how they intended to use CygNET in the future. This enlarged set of items helped evaluators to identify the system and social characteristics most closely associated with future use and to recommend that developers concentrate their efforts on improving those characteristics. The CygNET Online interface was first implemented in parallel with the then existing Innopac online public access system during a six month pilot period in one of the university’s science libraries. Three months into the pilot period, CygNET users were asked to complete a survey form which included a QUIS for Innopac and a QUIS for CygNET. From this comparative evaluation, it was possible both to identify that users found the CygNET Online interface to electronic information resources an improvement over the Innopac interface, and to measure the extent to which they found it an improvement. Specific aspects of the interface and the context of use were identified as areas for improvement. The success of the pilot implementation encouraged the Library’s management and staff to implement CygNET Online as the standard system-wide interface to information resources. One year after the pilot implementation (and 7 months after withdrawal of the text-based interface), the survey was repeated. In addition to questions which enabled comparison with earlier responses, questions which reflected learning and users’ comments during the year were added to the survey form. Improvements that had been implemented as the result of responses to the pilot study were well received, and new areas for improvement were identified. Formal, systematic, evaluation in the field enabled the sponsors and developers of CygNET Online to gain a good understanding of users’ response to the networked information resource. By measuring users’ perceptions of the interface itself, their attitudes to its use, and characteristics of the context of use, and by identifying those characteristics of both resource and context that are most closely associated with use, they have been able to identify and to respond positively to users’ recommendations for improvement