Towards a Technology for Computational Experimentation

The evaluation of cognitive models is sometimes compromised by inadequate comparison of model behaviour with human be-haviour. Inadequacies may arise through unconstrained set-ting of model parameters to yield the target behaviours and/or through incomplete replication of the human participant’s en-vironment when running the model. This paper reviews meth-odologies for the evaluation of cognitive models that address (to varying extents) these difficulties. It is proposed that standard empirical psychology provides the basis for an ad-equate methodology, in which within-model variations are treated as between-subjects variables. The control of large-scale between-subjects designs requires substantial computa-tional infrastructure, and the second half of the paper presents details of a general web-based client-server system that em-bodies this infrastructure. The system may be used to sup-port the execution of computational experiments, human ex-periments, and computational parameter studies. Three case studies illustrate these different uses