A software metric suite for evaluating the usability of Learning Management Systems

In the last decade the World Wide Web has grown rapidly in both acceptance and application across a wide range of business, scientific, educational and societal contexts. From the mid 1990's there has been a move from static Internet sites to dynamic database driven Internet applications, Learning Management Systems (LMS) being an example of the latter in the education domain. Although intrinsically different from previous information systems because of the ubiquitous character of the Internet, many issues associated with more traditional software engineering projects are observable in the dynamics of newer Internet-based systems development activities. Usability is one of the most significant of those issues carried over into the contemporary Internet systems development environment that are crucial to their acceptance. Deploying a system that is not usable can greatly increase post-implementation maintenance and can incur training costs that could have been avoided if more rigorous usability testing had been conducted during development. Numerous usability testing methods exist. While often very effective, most require input from test subjects and can therefore be costly in time and effort. A further drawback to these methods is that they are generally applied close to the time of final deployment, meaning that the developers do not receive sufficiently detailed feedback to enable them to focus on fixing the offending elements that caused the usability problem in the first place. This suggests that developers need a low cost usability assessment method that can be used iteratively throughout the development process, giving them greater insight into usability problems than current methods allow. The research described in this thesis is primarily concerned with establishing and evaluating a cost-effective metric suite for determining the usability of Learning Management Systems. These LMS are complex interactive Internet applications that are increasingly integral to IT assisted teaching and learning environments. The research issues raised in this thesis lead to the development of the LMS Metric Suite. This metric suite is based upon a number of design and contextual criteria (a combination of intuitive characteristics and criteria established in the research field of human-computer interaction), enabling developers to evaluate the usability of LMS without requiring opinionative, subjective and expensive input from users. By reviewing the outcome of the calculated metric suite the developer should be able to clearly identify weaknesses within the usability of the design. A study is conducted that compares two common usability methods and the LMS Metric Suite against two versions of a deployed LMS in order to evaluate the effectiveness of the metric suite in establishing (approximating) usability acceptance potential. The results indicate that the LMS Metric Suite is a useful tool for developers to gauge the usability of their LMS and provides them with sufficiently detailed feedback to allow them to fix the offending elements. As such it is an effective complementary method that should be used in conjunction with later-phase user-based testing.UnpublishedAdrion, W. R., Barnstad, M. and Cherniaysky, J. (1982). Validation, verification, and testing computer software, ACM Computing Surveys 14(2): 159-192. AL-Qaimari, G. and McRostie, D. (1999). KALDI: A computer-aided usability engineering tool for supporting testing and analysis of humancomputer interaction., Proceedings of the Third International Conference on Computer-Aided Design of User Interfaces, Kluwer Academic Publishers. Albrecht, A. J. (1985). Function points help manager assess applications, Computerworld (SR/20): 20-21. Anderson, C., Domingos, P. and Weld, D. (2001). Personalizing Web Sites for Mobile Users, Proceedings of the 10th International WWW Conference, ACM Press, pp. 565-575. Anderson, R., Homer, A., Sussman, D. and Blexrud, C. (1999). Professional Active Server Pages 3.0., Birmingham, Wrox Press Inc. Arroyo, I., Beck, J., Beal, C. R. and Woolf, B. (2003). Learning within the Zone of Proximal Development with the AnimalWatch intelligent tutoring system, Proceedings of the American Educational Research Association annual meeting. Accessed: 5/Mar/2004. ASU (2000). Arizona State University - 10 Critical Success Factors for Cyber Campuses, http://clte.asu.edu/active/Critical°420Succesa20Factors.doc. Accessed: 16/Feb/2002. Atkins, M. (1993). Evaluating interactive technologies for learning, Journal of Curriculum Studies 25(4): 333-342. Babiker, E. M., Fujihara, H. and Boyle, C. D. B. (1991). A metric for hypertext usability, Proceedings of the 9th annual international conference on Systems documentation, ACM Press, pp. 95-104. Bannon, L. (1991). Design at Work: Cooperative Design of Computer Systems. J. Greenbaum E M.Kyng(Eds.), Hillsdale, NJ: Lawrence Erlbaum Associates, chapter From human factors to human actors: The role of psychology and human-computer interaction studies in system design, pp. 25-44. Basili, V. and Perricone, B. T. (1984). Software Errors and Complexity: An Empirical Investigation, Communications of the ACM 27(1): 42-52. Beal, C. R., Arroyo, I., Royer, J. and Woolf, B. (2003). Wayang Outpost: An intelligent multimedia tutor for high stakes math achievement tests, Proceedings of the American Educational Research Association annual meeting. Accessed: 5/Mar/2004. Becker, R., McLaughlin, K. and Rebelsky, S. (1999). Project CLIO: Tools for tracking student use of course webs, World Conference on Educational Multimedia, Hypermedia and Telecommunications (ED-MEDIA, Seattle, USA. Benbunan-Fich, R. (2002). Improving education and training with IT, Communications of the ACM 45(6): 94-99. Berkman, J. (1997). Web Design and Maintenance, 25th SIGUCCS Conference on User Services, Monterey, CA. Berners-Lee, T. (1990). WorldWideWeb: Proposal for a HyperText Project, http //www . w3 . org/Proposal . html. Accessed: 11/Jun/2003. Bevan, N. (1999). Quality in use: meeting user needs for quality, Journal of Systems and Software 49(1): 89-96. Bevan, N., Barnum, C., Cockton, G., Nielsen, J., Spool, J. and Wixon, D. (2003). The "magic number 5": is it enough for web testing?, CHI '03 extended abstracts on Human factors in computing systems, ACM Press, pp. 698-699. Bias, R. G. (1994). The pluralistic usability walkthrough: coordinated empathies, John Wiley and Sons, Inc., pp. 63-76. Biggs, J. B. and Telfer, R. (1987). The Process of Learning, 2 edn, Sydney, Prentice- Hall. Billsus, D., Brunk, C. A., Evans, C., Gladish, B. and Pazzani, M. (2002). Adaptive interfaces for ubiquitous web access, Communications of the ACM 45(5): 34-38. Blackboard (2004). Blackboard, http://www.blackboard. com/. Accessed: 5/Jan/2004. Blackmon, M. H., Poison, P. G., Kitajima, M. and Lewis, C. (2002). Cognitive walkthrough for the web, Proceedings of the SIGCHI conference on Human factors in computing systems, ACM Press, pp. 463-470. Bodart, F., Hennebert, A.-M., Leheureux, J.-M. and Vanderdonckt, J. (1994). Towards a Dynamic Strategy for Computer-Aided Visual Placement, 2nd Workshop on Advanced Visual Interfaces, Bari, Italy. Bodker, S., Ehn, P., Knudsen, J., Kyng, M. and Madsen, K. (1988). Computer support for cooperative design (invited paper), Proceedings of the conference on Computersupported cooperative work, ACM Press, pp. 377-394, Borges, J. A., Morales, I. and Rodrguez, N. J. (1996). Guidelines for Designing Usable World Wide Web Pages, Proceedings of the CHI '96 conference companion on Human factors in computing systems: common ground, ACM Press, pp. 277-278. http://doi.acm.org/10.1145/257089.257320. Botafogo, R. A., Rivlin, E. and Shneiderman, B. (1992). Structural analysis of hypertexts: identifying hierarchies and useful metrics, ACM Transactions on Information Systems (TOIS) 10(2): 142-180. Broder, A. and Kumar, R. (2000). Graph structure in the web, Proceedings of the 9th Intl. WWW Conference, ACM Press, pp. 309-320. Brown, A. and Palinscsar, A. (1989). In Lauren B. Resnick (Ed.) Knowledge, Learning and Instruction, Lawrence Erlbaum, Hillsdale, NJ, chapter Guided, Cooperative Learning and Individual Knowledge Acquisition. Brown, J. and Duguid, P. (1994). Borderline issues: Social and material aspects of design, Human-Computer Interaction 9(1): 3-36. Brusilovsky, P. and Maybury, M. T. (2002). From adaptive hypermedia to the adaptive web, Commun. ACM 45(5): 30-33. Brusilovsky, P., Ritter, S. and Schwarz, E. (1997). Distributed intelligent tutoring on the Web, 8th World Conference on Artificial Intelligence in Education, Amsterdam, IOS. Brusilovsky, P. and Rizzo, R. (2002). Map-based horizontal navigation in educational hypertext, Proceedings of the thirteenth ACM conference on Hypertext and hypermedia, ACM Press, pp. 1-10. Buie, E. (1999). HCI standards: a mixed blessing, Interactions 6(2): 36-42. Bygrave, J., Sallis, A., Matheny, J. and Carran, G. (1999). Collaborative Learning Online in International Business, Southeast Asian Institutions of Higher Learning (Asa,ihl), Auckland, New Zealand. Byrne, E. (2000). Evaluate Web Resources, http: //www. clubi o ie/webserch/ resources/content . htm. Accessed: 21/Dec/2003. C2T2 (2001). Online educational delivery applications: a web tool for comparative analysis, http://www. c2t2. ca/. Accessed: 12/Jan/2004. Canale, R. (1998). Review of Web based, Course Delivery Software Systems, http: //www.dls.au.com/papers/review-cds.html. Accessed: 1/May/2004. Center, 0. S. (2002). Tools, Reviews and Comparisons, http: //www.osc edu/ education/webed/Tools/review. shtml. Accessed: 8/Feb/2003. CERN (1998). The world wide web, CERN, http : //public . web . cern ch/public/ about/achievements/www/www.html. Accessed: 12/Jan/2004. CERN (2002). History and growth, CERN, http : //public . web . cern . ch/public/ about/achievements/www/history/history.html. Accessed: 15/Jan/2004. Chakrabarti, S., Gibson, D. and McCurley, K. (1999). Surfing the Web Backwards, Computer Networks 31: 1679-1693. Chi, E. H., Rosien, A., Supattanasiri, G., Williams, A., Royer, C., Chow, C., Robles, E., Dalal, B., Chen, J. and Cousins, S. (2003). The bloodhound project: automating discovery of web usability issues using the InfoScentπ simulator, Proceedings of the conference on Human factors in computing systems, ACM Press, pp. 505-512. Chi, M. T. H., Bassok, M., Lewis, M. W., Reinmann, P. and Galser, R. (1989). Self- Explanations: How Students Study and Use Examples in Learning to Solve Problems, Cognitive Science 13: 145-182. Choudhary, P. (1998). A Comparative Analysis Of Online Education Tools, http //www.visc . vt . edu/succeed/wwwframework/survey . html. Accessed: 14/Dec/2002. Cohen, E. (1994). Restructuring the classroom: Conditions for productive small groups, Review of Educational Research 64(1): 1-35. Coll, R. and Wingertsman, J. C. (1990). The effect of screen complexity on user preference and performance, International Journal of Human-Computer Interaction 2(3): 255-265. Collins, A. (1990). Educational Values and Cognitive Instruction, Lawrence Erlbaum, Hillsdale. NJ, chapter Cognitive Apprenticeship and Instructional Technology, In Idol, L. and Jones, B. F. (Eds.). Comber, T. and Maltby, J. (1996). Computer-aided design of user interfaces, J. Vanderdonckt. Namur, Belgium, Press Universitaries de Namur, chapter Investigating layout complexity, pp. 209-227. Conklin, J. (1987). Hypertext: An introduction and survey, IEEE Computer 20(9): 17- 41. Constantine, L. L. and Lockwood, L. A. D. (2002). Usage-centered engineering for web applications, IEEE Software 19(2): 42-50. http : //www. computer. org/software/ so2002/s2toc.htm. Conte, S. D., Dunsmore, H. E. and Shen, V. Y. (1986). Software Engineering Metrics and Models, Benjamin/Cummings Publishing Company, Inc., Menlo Park, California. Corry, M., Frick, T. W. and Hansen, L. (1997). User-centered design and usability testing of a web site: An illustrative case study, Educational Technology, Research and Development 45(4): 65-76. http : //education. indiana. edu/ -edfolks/persons/frick.html. Coulston, C. and Vitolo, T. M. (2001). A hypertext metric based on huffman coding, Proceedings of the twelfth ACM conference on Hypertext and Hypermedia, ACM Press, pp. 243-244. Crampes, M. and Ranwez, S. (2000). Ontology-supported and ontology-driven conceptual navigation on the world wide web, Proceedings of the eleventh ACM on Hypertext and hypermedia, ACM Press, pp. 191-199 Darken, R. P. and Sibert, J. (1996). Navigating in Large Virtual Worlds, International Journal of Human-Computer Interaction 8(1): 49-72. De Angeli, A., Matera, M., Costabile, M. F., Garzotto, F. and Paolini, P. (2000). Validating the sue inspection technique, Proceedings of the working conference on Advanced visual interfaces, ACM Press, pp. 143-150. DeBra, P., Houben, G., Kornatzky, Y. and Post, R. (1994). Information retrieval in distributed hypertexts, 4th RIAO-94 Conference, New York. DeMarco, T. (1982). Controlling Software Projects: Management, Measurement and Estimation, Yourdon Press, New York. Denaro, G. and Pezz, M. (2002). An empirical evaluation of fault-proneness models, Proceedings of the 24th international conference on Software engineering, ACM Press, pp. 241-251. der Aalst, W. V., Bra, P. D. Houben, G. J. and Kornatzky, Y. (1993). Computing Science Notes 93, chapter Browsing semantics in the "tower" model, p. 47. Desurvire, H. W. (1994). Faster, cheaper!! Are usability inspection methods as effective as empirical testing?, John Wiley and Sons, Inc., pp. 173-202. Desurvire, H. W., Kondziela, J. M. and Atwood, M. E. (1993). What is gained and lost when using evaluation methods other than empirical testing, Proceedings of the conference on People and computers VII, Cambridge University Press, pp. 89-102. Dicks, R. S. (2002). Mis-usability: on the uses and misuses of usability testing, Proceedings of the 20th annual international conference on Computer documentation, ACM Press, pp. 26-30. DiNucci, D., Giudice, M. and Stiles, L. (1998). Elements of web design, Technical report, Berkley CA, PeachPit Press. Dix, A., Finlay, J., Abowd, G. and Beale, R. (1993). Human-Computer Interaction, 2nd edn, London, Prentice Hall. Doubleday, A., Ryan, M., Springett, M. and Sutcliffe, A. (1997). A comparison of usability techniques for evaluating design, Proceedings of the conference on Designing interactive systems, ACM Press, pp. 101-110. Draper, S. (1998). Niche-based success in CAL, Computers and Education 30(1/2): 5– 8. Draper, S., Brown, M., Henderson, F. and McAteer, E. (1997). Integrative Evaluation: An Emerging Role for Classroom Studies of CAL, Computers in Education 26(1- 3): 17-32. Dugger, W. and Naik, N. (2001). Clarifying misconceptions between technology education and educational technology, The Technology Teacher 61(1): 31-35. Dumas, J. R. and Redish, J. C. (1993). A practical guide to usability testing, Technical report, Norwood, NJ, Ablex Publishing Corporation. Dunham, T., Wells, J. and White, K. (2002). Biotechnology education: A multiple instructional strategies approach, Journal of Technology Education JTE 14(1). Accessed: 14/Jan/2004. Ebling, M. R. and John, B. E. (2000). On the contributions of different empirical data in usability testing, Conference proceedings on Designing interactive systems : processes, practices, methods, and techniques, ACM Press, pp. 289-296. eduTools (2002). Compare All Products by All Features, http : //www. edutools inf o/ course/compare/all . j sp. Accessed: 1/May/2004. Entwistle, N. and Ramsden, R. (1983). Understanding Student Learning, Croom Helm, London. Ericsson, K. A. and Simon, H. (1980). Verbal reports as data), Psychological Review 87(3): 215-251. Etgen, M. and Cantor, J. (1999). What does getting WET (web event-logging tool) mean for web usability, Proceedings of the Fifth Conference on Human Factors and the Web, Gaithersburg, MD. Ewing, J. and Miller, D. (2002). A framework for evaluating computer supported collaborative learning, Educational Technology and Society 5(1): 112-118. Feizabadi, S. (1996). History of the World Wide Web, Technical report, Virginia Polytechnic Institute and State University. Accessed: 11/Jan/2004. Ferguson, J. D., McGettrick, A. D. and Smeed, D. (1999). Quality Issues in a Virtual University, Computers and Advanced Technology in Education, Proceedings CATE99, Philadelphia, pp. 8-12. Ferguson, J., Miller, J. and Weir, G. (1999). Estimating the Number of Defects at a WWW Site, World Conference on the WWW and Internet (WebNet), Honolulu, Hawaii. Finck, N. (1999). Form vs. Function: Finding the Balance, Digital Web Magazine. http : //www . digital-web . com/f eatures/f eature_1999-10 shtml. FirstClass (2004). FirstClass, http : //www. centrinity. . com/AboutFC/. Accessed: 6/Mar/2004. Flanders, V. and Willis, M. (1998). Web pages that suck : Learn good design by looking at bad design, Sybex, Alameda, CA. Fleming, J. (1998). Web Navigation: Designing the User Experience, Technical report, Sebastopol, CA, O'Reilly and Associates. Florins, M. and Vanderdonckt, J. (2004). Graceful degradation of user interfaces as a design method for multiplatform systems, Proceedings of the 9th international conference on Intelligent user interface, ACM Press, pp. 140-147. Francis, B., Kauffman, J., Llibre, J., Sussman, D. and Ullman, C. (1998). Beginning Active Server Pages 2.0., Wrox Press, Birmingham. Frederiksen, J. R. and White, B. Y. (1997). Cognitive Facilitation: A Method for Promoting Reflective Collaboration, Proceedings of the Second International Conference on Computer Support for Collaborative Learning, Toronto, pp. 53-62. Fritsch, M. (1997). Platforms for Virtual Seminars: Comparison of the features of the three internet-communication-tools, http : //www f ernuni-hagen . de/ZIFF/ vergleichE . html. Accessed: 7/Feb/2004. Gagne, R., Briggs, L. and Wager, W. (1992). Principles of Instructional Design, 4th edition edn, HBJ College Publishers, Fort Worth. TX. Garzotto, F. and Matera, M. (1997). A Systematic Method for Hypermedia Usability Inspection, The New Review of Hypermedia and Multimedia 3: 39-65. George, P. (2000). Breaking ranks, Principal Leadership 1(4): 56-61. Goddard, T. and Sunderam, V. S. (1999). Toolspace: Web based 3d collaboration, Proceedings of the fourth symposium on Virtual reality modeling language, ACM Press, pp. 161-166. Goodman, B., Soller, A., Linton, F. and Gaimari, R. (1998). Encouraging Student Reflection and Articulation using a Learning Companion, International Journal of Artificial Intelligence in Education 9: 237-255. Goodwin, N. C. (1987). Functionality and usability, Communications of the ACM 30(3): 229-233. Gromov, G. R. (1995). History of Internet and WWW: The Roads and Crossroads of Internet History, http: //www .netvalley. com/intvalold html. Accessed: 12/Mar/2004. Grudin, J. (1994). Groupware and social dynamics: eight challenges for developers, Communications of the ACM 37(1): 92-105. Grudin, J. (2002). Group dynamics and ubiquitous computing, Commun. ACM 45(12): 74-78. Guillemette, R. A. (1995). Human factors in information systems: Emerging theoretical bases, C. J. M. (Ed.). Norwood, NJ, Ablex, chapter The evaluation of usability in interactive information systems, pp. 207-221. Gunn, C. (1997). CAL Evaluation: future directions, ALT-J 5(1): 40-47. Halstead, M. H. (1977). Elements of software science, Elsevier, New York. Hammond, M. (1994). Measuring the impact of IT on learning, Journal of Computer Assisted Learning 10: 251-260. Hammond, N. and Allinson, L. (1998). Travels around a learning support environment: Rambling, orienteering or touring, CHI'88 - Conference on Human Factors in Computing Systems, New York. Harasim, L., Hiltz, S. R., Teles, L. and Turoff, M. (1995). Learning Networks: A Field Guide to Teaching and Learning Online, MIT Press, Cambridge MA. Heer, J. and Chi, E. H. (2002). Separating the swarm: categorization methods for user sessions on the web, Proceedings of the SIGCHI conference on Human factors in computing systems, ACM Press, pp. 243-250. Herder, E. (2002). Metrics for the adaptation of site structure, Proc. of the German Workshop on Adaptivity and User Modeling in Interactive Systems ABIS02, pp. 22- 26. Hiltz, S. R. (1994). The Virtual Classroom: Learning without limits via computer networks, Ablex Publishing Corporation, New Jersey. Holtzblatt, K. and Beyer, H. (1993). Making customer-centered design work for teams, Communications of the ACM 36(10): 92-103. Huang, Y.-W., Huang, S.-K., Lin, T.-P. and Tsai, C.-H. (2003). Web application security assessment by fault injection and behavior monitoring, Proceedings of the twelfth international conference on World Wide Web, ACM Press, pp. 148-159. IEEE1061 (1992). IEEE 1061 Standard for a Software Quality Metrics Methodology, Report. IS013407 (1999). International Organization for Standardization, ISO/IEC. 13407 Human-Centred Design Processes for Interactive Systems, Report. IS014598-5 (1998). International Organization for Standardization, ISO 14598-5 Information technology – Software product evaluation -- Part 5: Process for evaluators, Report. IS09126 (1991). International Organization for Standardization, ISO/IEC 9126. Information Technology - Software Product Evaluation - Quality characteristics and guidelines for their use, Report. IS09241 (1998). International Organization for Standardization, ISO 9241-11 Ergonomic requirements for office work with visual display terminals