Evaluating an agent-based software maintenance knowledge management system (KMS) tool

Background: Software maintenance (SM) environment is highly complex,knowledge-driven and collaborative. Therefore, a knowledge Management System (KMS) is critical to provide an environment for sharing and sustaining knowledge among members of SM community of practice (CoP). An agent-based KMS prototype system called Software Maintenance Collaborative Agent Team (SCAT) was developed based on a new KMS framework, to assisst maintainers to share information and perform their SM activities. Aims: The aim of this study is to evaluate SCAT'ssuccess as a KMS tool against another existing similar tool called Rodriguez KMS SCAT was benchmark against another agent-based KMS prototype. We hypothesize that the extent of KMS success of SCAT is better than Rodriguez KMS Sample: A total of 15 users and maintainers took part in the evaluation of both tools and survey, consisting of 3 users, 6 helpdesk personnel, 8 developers, 2 systems analyst, a maintenance manager and a quality assurance analyst. Some SA and developers also doubled as helpdesk personnel, hence the extra numbers. Method: A survey was conducted to gauge the responses of users and maintainers on the success of both KMS tools, covering system quality,knowledge quality, user satisfaction, perceived KMS benefits and system use. The result of the survey are analyzed using Rasch rating-scale model to determine if SCAT fares better than Rodriguez model.Two Winsteps sessions were used, with person means set to 0. T-Test was used to compare both tools measure means and Common Person Equating plot and Measure Difference chart are used to further explain the results. Results:Based on t-test, we cannot conclude that SCAT KMS is better than Rodriguez KMS. However, in several important aspects such as Effectively Managing Knowledge, Improves Quality, and Efficient, SCAT fares significantly better. Rodriguez strong point is in decision making. Conclusion: Rasch could be utilized to effectively evaluate two KMS tools vis-a-vis a questionnaire survey. In our case, even though SCAT could not be concluded better than the benchmarked KMS tool, the analysis managed to identify several strong and weak key areas. As such, in future, SCAT could be enhanced to improve the meak areas