Methodological advances for deepening the interrogation of data from education evaluations

Thesis (MCom)--Stellenbosch University, 2021.ENGLISH SUMMARY : The ability to read is arguably the most important skill in most academic settings. All present and future learning is dependent fundamentally on the ability to appropriately interpret text and internalise its meaning. Students that do not possess the level of literacy skill necessary to decode text will be incapable of keeping pace with an education curriculum that is based primarily on the sequential acquisition of certain skills. It is a serious concern that a large number of students within the South African education system do not possess adequate literacy ability; this illiteracy is omnipresent in poorer schools. Therefore, the link between poverty, inadequate education quality, and low reading ability becomes a binding constraint on the ability of poor students to perform academically. This link is a means through which limited labour market prospects - and in many cases, poverty - become entrenched. While the improvement of education system is the central focus for policymakers, the inability to adequately model and explain the determinants of reading ability is a fundamental constraint on research. And as well-designed policy is dependent on accurate research, it is imperative that research in this area be performed as well as is possible. Contemporary work in the field of the economics of education centres on the use of well-established methods that have remained relatively unchanged. This thesis takes the stance that commonly used methods may be inappropriate in certain contexts. The broad aim of this thesis is twofold. First, to demonstrate that value exists in the use of novel techniques that differ from what is currently common. Second, in doing so, derive findings that can be useful in guiding both current and future policy and research. The approaches taken in this thesis borrow extensively from methods used in the educational psychological and statistical learning literature. Two broad empirical themes are emphasised - latent constructs (factor modelling) and non-linear modelling. The thesis includes three empirical chapters (2, 3, and 4), each of which considers a unique aspect of the application of non-linear and factor modelling to large education-based datasets. Chapter 2 investigates the potentially harmful effects of negative item responses using a latent construct framework. The intention of Chapter 2 is to use procedures of factor modelling to investigate whether the nature and design of survey questions can influence the information contained in the derived measured variable. It finds evidence that negatively worded survey questions capture information that is inconsistent with that derived from equivalent positively worded survey questions. Chapter 3 analyses how the choice of factor estimation technique affects the magnitude of measurement error transmitted to the factor from the underlying features. That is, Chapter 3 compares different methods of dimension reduction with regard to their relative ability to extract meaningful variation from N features while minimising the extraction of noisy variation. It finds significant performance differences between alternative methods of dimension reduction in the presence of erroneously measured variation, with evidence that Exploratory Factor Analysis is a superior approach. Chapter 4 fits a non-linear model using measured features and estimated factors to determine their relationships with one another and with reading performance. While Chapters 2 and 3 are methodological studies, Chapter 4 is an empirical analysis that models and interprets functional relationships that determine reading ability among South African grade 4 students. It finds several policy-relevant outcomes that are discussed in the chapter.AFRIKAANSE OPSOMMING : Die vermoë om te kan lees is waarskynlik die belangrikste vaardigheid om te hê in die meeste akademiese omgewings. Alle huidige en toekomstige geleerdheid is fundamenteel afhanklik van die vermoë om teks op die gepaste manier te analiseer en die betekenis daarvan te internaliseer. Leerders wat nie die vlakke van geletterdheid het om teks te ontleed nie sal nie in staat wees om by te bly met ‘n opvoedkundige kurrikulum wat hoofsaaklik gefokus is op die sekwensiële verkryging van sekere vaardighede nie. Dit wek ernstige kommer dat ‘n groot aantal studente in die Suid Afrikaanse onderwys stelsel nie genoegsame geletterdheid het nie. Hierdie ongeletterdheid is alomteenwoordig in armer skole. Daarom word die verbintenis tussen armoede, onvoldoende opvoedkundige kwaliteit en lae leesvermoë ‘n bindende beperking op die vermoë van armer studente om akademiese sukses te behaal. Hierdie verbintenis is hoe beperkte arbeidsmark vooruitsigte en in baie gevalle, armoede, verskans word. Terwyl die verbetering van die onderwysstelsel die sentrale fokus is van beleidmakers, is die onvermoë om die faktore wat leesvermoë bepaal te modelleer en verduidelik op ‘n voldoende manier ‘n fundamentele beperking op navorsing. En omdat beleide wat goed ontwerp is afhanklik is van akkurate navorsing is dit noodsaaklik dat navorsing in hierdie gebied so goed as moontlik gedoen word. Huidige werk in die veld van ekonomie van opvoeding is gesentreer rondom die gebruik van goed gevestigde metodes wat relatief onveranderd gebly het. Hierdie tesis neem die standpunt dat algemeen gebruikte metodes onvanpas kan wees onder sekere omstandighede. Die breë doelwit van hierdie tesis is tweedelig. Eerstens, om te demonstreer dat daar waarde is in die gebruik van nuwe tegnieke wat verskil van die huidige algemene metodes. Tweedens, deur dit te demonstreer, bevindinge te kry wat bruikbaar kan wees om beide huidige en toekomstige beleide te beïnvloed. Die benaderinge wat in hierdie tesis gebruik word maak baie staat op metodes wat in opvoedkundige sielkunde gebruik word sowel as die statistiese onderrig literatuur. Twee breë empiriese temas word beklemtoon – latente konstrukte (faktor modellering) en nie-liniêre modellering. 1 Die tesis sluit drie empiriese hofstukke in (2,3 en 4) waar elkeen ‘n unieke aspek van die toepassing van nie-liniêre en faktor modellering tot groot opvoedkundige gebaseerde datastelle oorweeg. Hoofstuk 2 ondersoek die moontlike skadelike effek van negatiewe item reaksies wat ‘n latente konstrukte raamwerk gebruik. Die doel van Hoofstuk 2 is om prosedures van faktor modellering te gebruik om te ondersoek of die natuur en ontwerp van vrae in ‘n vraelys die inligting omvat in die afgeleide ondersoekte veranderlike kan beïnvloed. Dit vind bewyse dat negatief bewoorde vrae in ‘n vraelys inligting opneem wat strydend is met inligting wat verkry is van ‘n ekwivalente vraag wat positief bewoord is. Hoofstuk 3 analiseer hoe die keuse van ‘n faktor estimasie tegniek die grootte van die metings fout oorgedra na die faktor vanaf die onderliggende eienskappe affekteer. Dit wil se, Hoofstuk 3 vergelyk verskillende metodes van dimensie vermindering met betrekking tot hul relatiewe vermoë om betekenisvolle variasie vanaf N eienskappe te kry terwyl die ekstraksie van raserige variasie geminimaliseer word. Dit vind beduidende verskille in die optrede tussen alternatiewe metodes van dimensie vermindering in die aanwesigheid van verkeerdelik gemete variasie met bewyse dat Eksploratiewe Faktor Analise ‘n beter benadering is. Hoofstuk 4 pas ‘n nie-liniêre model toe met die gebruik van gemete eienskappe en geskatte faktore om te bepaal wat hul verhoudings is met mekaar en met lees prestasie. Terwyl Hoofstuk 2 en 3 metodologiese studies is, is Hoofstuk 4 ‘n empiriese analise wat die funksionele verhoudinge wat leesvermoë onder Suid Afrikaanse Graad 4 leerders beïnvloed modelleer en interpreteer. Dit vind verskeie beleid relevante uitkomstes wat bespreek word in die hoofstuk.Master