Standardized assessment of knowledge of basic programming concepts for primary school level

Preverjanje znanja je bistvenega pomena pri vsakem izobraževalnem procesu. Preverjanje je sestavni del vrednotenja znanja, pri katerem zbiramo odzive učečih, ki povedo, v kolikšni meri učeči dosega učne cilje oz. standarde znanja. Preverjanje znanja ima formativno vlogo oz. temelji na spremljanju znanja učencev in ima izobraževalni namen s ciljem izboljšati učenje in znanje. Področje računalništva v osnovni šoli nima veljavnih in zanesljivih inštrumentov za preverjanje znanja v pedagoške in raziskovalne namene. Zveza učiteljev računalniške znanosti (angl. Computer Science Teachers Association - CSTA) predlaga razvoj inštrumentov, ki upoštevajo učne cilje iz učnih načrtov in so neodvisni od programskih jezikov. V disertaciji obravnavamo možnosti prilagoditve standardnih metod za izdelavo inštrumentov za preverjanje znanja iz uvodnega programiranja v osnovni šoli. Raziskujemo možnost izdelave novega inštrumenta za preverjanje znanja temeljnih konceptov programiranja neodvisno od programskih jezikov in izobraževalnih okolij. V postopku izdelave inštrumenta upoštevamo predpostavke in omejitve poučevanja uvodnega programiranja v osnovni šoli in analiziramo artefakte za presojo veljavnosti in zanesljivosti inštrumenta. Cilj te disertacije je oblikovanje nove metodologije, ki bo omogočala izdelavo veljavnih in zanesljivih inštrumentov za preverjanje znanja temeljnih konceptov uvodnega programiranja v osnovni šoli neodvisno od programskih jezikov in izobraževalnih okolij. V ta namen smo predlagali štiri prilagoditve standardnih metod: dodaten korak za preverjanje neodvisnosti nalog od programskih jezikov in izobraževalnih okolij, poenostavitev postopka za identifikacijo zmotnih prepričanj in drugih težav učečih pri uvodnem programiranju, kombinacijo raziskovalnih in statističnih metod za presojanje nekaterih merskih lastnosti inštrumenta, dodaten korak za napovedovanje primernosti nalog z metodami strojnega učenja. Prilagoditve analiziramo v sklopu petih študij in odgovorimo na sedem raziskovalnih vprašanj. Najprej opredelimo konceptualno vsebino, ki jo želimo preverjati. Nato predlagamo enovit konstrukt nalog, s pomočjo katerega so oblikovane vse naloge v raziskavi. V postopku oblikovanja nalog odprtega tipa preverjamo neodvisnost nalog od programskih jezikov in izobraževalnih okolij. V postopku izdelave nalog izbirnega tipa analiziramo možnosti oblikovanja nepravilnih odgovorov (motilcev) na podlagi izsledkov analize izzivanja zmotnih prepričanj in drugih težav učečih. V zaključku postopka obravnavamo vse zbrane artifakte za presojo veljavnosti in zanesljivosti inštrumenta in testiramo metode strojnega učenja za napovedovanje primernosti nalog. Na koncu predstavimo novo metodologijo v obliki urejenega seznama vseh korakov in diagrama poteka korakov. Nova metodologija omogoča standardizirano izdelavo veljavnih in zanesljivih inštrumentov za preverjanje temeljnih konceptov uvodnega programiranja na nivoju osnovne šole. Nastali inštrumenti bodo primerno težavni in občutljivi za zmožnosti učencev in bodo preverjali različne taksonomske stopnje znanja.Assessment of knowledge is essential in any educational process. Assessment is an integral part of knowledge evaluation where we collect responses from learners that provide information about the extent to which the learner is achieving the learning objectives or knowledge standards. It has a formative function where the results are generally not used for grading but for improving learning and teaching processes. In the field of computer science in primary schools, there are no valid and reliable knowledge assessment instruments for pedagogical and research purposes. The Computer Science Teachers Association (CSTA) proposes the development of valid and reliable instruments that take into account the curricular learning objectives and are independent of programming languages. In this dissertation, we explore the possibilities of adapting standardized methods for the development of valid and reliable instruments to assess knowledge of basic programming concepts independent of programming languages and educational environments. In developing the instrument, we consider the assumptions and limitations of teaching introductory programming in primary school and analyze the artifacts to determine the validity and reliability of the instrument. The goal of this dissertation is to develop a new methodology for developing valid and reliable instruments for assessing knowledge of the basic concepts of introductory programming in primary school, regardless of programming languages and educational environments. For this reason, we proposed four adaptations of standardized methods: an additional step to verify the independence of tasks from programming languages and educational environments, a simplification of the procedure for identifying programming misconceptions and other problems of learners in introductory programming, a combination of scientific and statistical methods to determine certain measurement properties of the instrument, and an additional step to predict the suitability of tasks in the instrument using machine learning methods. We analyze the adaptation in five studies and answer seven research questions. First, we define the conceptual content we want to assess. Then, we propose a unified task construct that was used to design all tasks in the study. When creating open-ended tasks, we analyzed the independence of the tasks from programming languages and learning environments. When creating closed-ended tasks, we analyzed the potential for creating distractors based on identified programming misconceptions. At the end of the process, we examine all collected artifacts to determine the validity and reliability of the instrument and test machine learning methods for predicting task suitability. Finally, we present a new methodology in the form of an ordered list of all steps and a flowchart. The new methodology allows for the standardized development of valid and reliable instruments for assessing the basic concepts of introductory programming at the primary school level. The resulting instruments will be appropriately difficult and sensitive to student ability and will assess knowledge at multiple taxonomic levels