Describing language assessments for school-aged children: A Delphi study

Keywords: developmental language disorder, language impairment, assessment, terminology, Delphi study Background Internationally, the problem of inconsistent use of professional terminology by speech-language pathologists (SLPs) is widely recognised (Walsh & IGOTF-CSD, 2006). Often one term may be used to refer to a range of different concepts or, conversely, different concepts are described by the same term (Walsh, 2005). Inconsistently applied terminology leads to breakdowns in professional communication and thus limits scientific debate needed to advance the profession. Lack of detailed terminology also hinders research translation as practices may not be described well enough to be replicated (Roulstone, 2015; Walsh & IGOTF-CSD, 2006). In the field of child language assessment, many models and terms exist for describing the different types of language assessments that SLPs may use. As a result, the way in which one SLP conceptualises and describes their language assessment may well be different to another SLP’s description of the same assessment. This creates significant barriers for the collection of accurate data on current practice both within and across service agencies (Cowie et al., 2001). Without an accurate understanding of current SLP assessment practice, it is difficult to compare current practice with evidence-based practice and thus identify clinical recommendations that align contemporary practice with policy and research evidence (Eadie, 2003). A framework frequently used to describe SLP assessment practice is the International Classification of Functioning and Disability (ICF) (McLeod & Threats, 2008; World Health Organisation, 2015). This framework was designed to provide a structure by which concepts related to health and wellbeing may be viewed but, as such, lacks detail for describing language assessment (Barnes & Bloch, 2018; Hughes & Orange, 2007). Since communication spans all aspects of health and wellbeing, it is acknowledged that SLPs often experience difficulty mapping assessment practices onto ICF categories (Barnes & Bloch, 2018; Hughes & Orange, 2007). Considerable disparity exists across literature with regards to how language assessments are classified within the ICF. For example, in some studies, norm-referenced language assessments, such as editions of the Clinical Evaluation of Language Fundamentals (Wiig, Semel, & Secord, 2004) are identified as assessing the body structure and functioning category of the ICF, while other studies identify these measures as assessing the activity category of the ICF (Paul & Norbury, 2012; Westby & Washington, 2017). The development of frameworks that are specifically targeted at describing SLP practices may facilitate greater consistency with regards to how assessments are described and thus enhance professional communication (Barnes & Bloch, 2018). Specifically within the field of paediatric SLP, there are a number of terms used to describe language assessments. One common feature is to describe the skill domain targeted in the assessment. This may be through the use of Bloom and Lahey’s taxonomy, which describes language domains across three aspects including form, content and use (Lahey, 1988); or through terms such as semantics, syntax, morphology, narrative or executive functioning (Larson & McKinley, 2007; Paul & Norbury, 2012). A second way in which assessments may be described is according to the purpose of the assessment. Categories include analytical or prognostic; summative or formative; or distinctions related to diagnostic purposes, screening, selecting intervention or determining service eligibility (Dockrell & Marshall, 2015; Newton, 2007; Paul & Norbury, 2012; Wade, 2004). Assessments may also be described by the method in which the assessment is conducted or the environmental context targeted in the assessment. Examples of methods include parent questionnaires, tests administered either face-to-face or through telehealth, or assessments conducted by automated computer software. Examples of terms related to environmental context include curriculum-based, naturalistic or authentic (Parsons, Law, & Gascoigne, 2005; Paul & Norbury, 2012). Finally, assessments may be identified by the type of data collected or the type of tasks embedded in the assessment. This includes terms such as norm-referenced, criterion referenced and dynamic for describing the type of data collected; or terms such as discrete-skill, functional, contextualised or language sampling for describing the types of tasks being assessed (American Speech and Hearing Association, 2018; Kaderavek, 2015). In addition to the presence of numerous sets of terms for conceptualising language assessments, the definitions of these terms are often not precisely defined or are defined differently across the literature. For example, the term standardised has been used to describe any assessment that has structured guidelines for administration (regardless of the type of data collected) but has also been used interchangeably with the term norm-referenced to refer specifically to assessments that provide normative data from a sample of age-matched peers (Kaderavek, 2015). The terminology used to describe assessments that are non-standardised in nature is even more loosely defined, with terms such as authentic, alternative, informal, naturalistic, behavioural and observational all used with unrestrained boundaries for the types of assessments covered by these terms (Caesar & Kohler, 2009; Hegde & Pomaville, 2017). Furthermore, the detail in assessment practice is not captured through the use of one framework or one set of terms. Two assessments described by the same term could be vastly different. For instance, a morphology assessment could refer to a series of clinician-directed sentence completion tasks organised developmentally or an analysis of the morphological forms produced in a language sample (Paul & Norbury, 2012). Similarly, a language sample might be a highly structured, norm-referenced narrative retell task or observations by an SLP during unstructured free play (Westerveld & Claessen, 2014). To describe assessments in detail, SLPs need access to a framework that facilitates the conceptualisation of language assessments across multiple distinguishing features. Given the current problems associated with terminology, there is a pressing need for actions that facilitate rigour and consistency with regards to the terms SLPs use for describing child language assessment (Walsh & IGOTF-CSD, 2006). It is evident that a single action is unlikely to solve all problems related to such a complex problem (Walsh, 2005). Nonetheless, solutions are needed for situations where terminology must be consistently applied in order to be useful, such as when collecting survey data on the types of practices SLPs use (Cowie et al., 2001). With this in mind, the establishment of an agreed-upon taxonomy for conceptualising various child language assessment practices is a logical step towards addressing some of the challenges associated with inconsistent terminology. In addition to facilitating data collection, such a taxonomy has the potential to stimulate much-needed professional discussion and reflection on assessment practice, which is vital for continued advancement in the field (Eadie, 2003; Roulstone, 2001). There is also a significant need for further research examining the application of professional terminology. This will assist in better understanding the issues and complexities associated with developing consistent use of terminology in the SLP field (Walsh & IGOTF-CSD, 2006). The current study This study employed a Delphi study technique to address two aims: (1) to develop a taxonomy (i.e. categorisation framework) that is agreed upon by experts and provides distinct, well-defined categories for describing language assessment practises for children aged 4–18 years; and (2) to examine SLP application of a taxonomy for describing language assessments in clinical contexts. For the purposes of this study, language assessment may be any data-gathering action including case histories, test performance, language sampling, behavioural observations, reports from significant others, and reports on educational achievement (Paul & Norbury, 2012). Given this is the first study to examine terminology for describing SLP assessment practice and acknowledging the potential complexities associated with developing consensus, the participant group in this study focussed on SLPs in a single country (Australia). It is envisaged that outcomes from this initial study will then act as the basis for further future research internationally. Method This study used a Delphi study technique with mixed-methods data collection and analysis (Tapio, Paloniemi, Varho, & Vinnari, 2011). The Delphi technique is a structured process which aims to develop group consensus on a defined topic through a series of survey rounds (Boulkedid, Abdoul, Loustau, Sibony, & Alberti, 2011). The same participants complete each round (though not all may continue with each round) and rounds are held until consensus is reached (or it becomes apparent that consensus cannot be reached). Results from previous rounds are used to inform changes that may facilitate consensus in subsequent rounds. Participants Criteria for participation in the Delphi study were: (1) eligibility for certified practicing membership with the Australian professional association for SLPs (Speech Pathology Australia); and (2) having spent at least 5 years (full-time equivalent) in the last 10 years engaged in professional activities, where 50% or more of professional time is related to children aged 4–18 years with a language support needs. These activities may include research, academic teaching, consultancy, resource development, provision of SLP professional development, provision of clinical services or a combination of these activities. Potential participants were contacted by email after being identified from the Speech Pathology Australia Find a Speech Pathologist website, the 2016 Speech Pathology Australia National Conference attendance contact list, and from the professional networks of the authors. In some states, recruited participants were also asked to identify other potential participants. A total of 202 invitations were emailed and all SLPs who responded to invitations were sent a link to the first survey. As each survey was developed based upon the results of the preceding Delphi round, participants who did not complete a survey round were excluded from subsequent rounds. This helped to ensure that all participants had the same knowledge of taxonomy. The number of participants who completed each Delphi round was 55 in round one (71.4% response rate), 43 in round two (78.2% response rate) and 32 in round three (74.4% response rate). Participant demographics for each round are presented in Table I. Table I. Participant demographics. CSVDisplay Table Procedure Following a deductive (top-down) approach (DeJong, Horn, Gassaway, Slavin, & Dijkers, 2004), theoretical literature and research publications related to language assessment for school-aged children were reviewed by the first author to identify key concepts and terms that provide both a description of and differentiation between assessments. These concepts and terms were further developed through discussions within the research team and organised into an initial taxonomy. This taxonomy consisted of four aspects for describing features of assessments including language modalities and domains, purposes, delivery methods and contexts, and the assessment form in terms of the type of data collected and type of tasks used. The initial taxonomy was then presented to study participants in a three-round Delphi study for feedback. Each Delphi round was conducted as an online survey using Qualtrics software. The round one survey was piloted initially with two SLPs to check the clarity of questions and completion time before being opened to Delphi study participants. Each survey was estimated as taking 90 minutes to complete. Delphi rounds were conducted between April 2017 and October 2017 with each survey being accessible for three to seven weeks. The study details were outlined at the beginning of each survey; participants were required to indicate consent to participate before accessing the remainder of the survey content. The study was ethically approved by the Curtin University Human Research Ethics Committee (Approval number: HRE2017-0126). Each Delphi study round consisted of two parts. Part A addressed the first research aim of developing consensus regarding the structure and definitions of the taxonomy. These questions were Likert scale responses (Strongly Disagree, Disagree, Neither Agree or Disagree, Agree and Strongly Agree). Participants who did not indicate agreement with an aspect were asked an open-ended question about what they would change with regards to the structure or definitions within the aspect. Part B explored the second research aim of examining the application of taxonomy by SLPs when describing assessments. Participants were asked to select taxonomy categories that they thought applied to assessments presented in the Delphi study, with open-ended questions also provided for participants to make comments regarding the use of the taxonomy. In the first survey round, participants were asked to describe four assessments that were identified to them by name. These included: Clinical Evaluation of Language Fundamentals-4th Edition (CELF-4) (Wiig et al., 2004), Children’s Communication Checklist-2nd Edition (CCC-2) (Bishop, 2003), Language Sampling Protocol (Westerveld & Gillon, 2002), and the Pragmatics Profile of Everyday Communication Skills in Children (Dewart & Summers, 1995). Participants who identified themselves as being unfamiliar with any of the assessments were not required to provide categorisations for those assessments. At least 24 of the study participants categorised each assessment for each aspect of the taxonomy. Analysis of data after round one led to the use of case studies in survey rounds two and three. The use of case studies made it possible for all participants to complete all questions as background information was provided about each assessment as well as a link to the published test website. The case studies were constructed to examine components of the taxonomy that may be more difficult to apply, while still being assessments that were characteristic of paediatric SLP practice. Two assessments were embedded within the case studies and the same case studies were used in both rounds two and three. Case study one described a parent interview using the Pragmatic Profile of Everyday Communication Skills for Children (Dewart & Summers, 1995) for a 4–10-year-old child with Autism Spectrum Disorder. Case study two described a language sampling procedure using the Language Sampling Protocol (Westerveld & Gillon, 2002) with a 7–10-year-old child experiencing language difficulties at school. These assessments were selected as they had the greatest inconsistency in agreement noted in round one compared to the agreement for the CELF-4 (Wiig et al., 2004) and CCC-2 (Bishop, 2003). In each of three survey rounds, the proposed taxonomy structure and definitions were presented in a reference document along with a summary of background information and references to literature. After each round, changes to the taxonomy in response to quantitative data (level of agreement with taxonomy structure and definitions or level of agreement with assessment categorisations) and qualitative data (themes from participant comments and feedback)were made by updating the taxonomy reference document. Changes were made with the aim of either increasing agreement with the taxonomy itself, or improving application of the taxonomy by addressing identified sources of confusion with definitions. In rounds two and three, participants were also provided with a document summarising the quantitative and qualitative group results from the previous round. Analysis Survey responses were analysed using the Statistical Package for the Social Sciences (SPSS version 20 software (IBM Corp, 2011). The number of rounds and criteria for agreed consensus were determined before the study commenced. In Part A, agreement with taxonomy structure and definitions were defined as 75% or more participants selecting ‘‘Strongly Agree” or ‘‘“Agree” (i.e. median score of 4 or more on the five-point scale and inter-quartile range (IQR) of 1). In Part B, consensus on the categorisation of assessments was considered achieved when 75% or more participants selected (or opted not to select) a taxonomy category for an assessment. While the agreement between the authors’ categorisations and Delphi participant’s categorisations was not a requirement for consensus, examining concordance between the two provided an additional means of examining the application of taxonomy. Participants’ survey responses to open-ended questions were analysed using conventional content analysis (Hsiu-Fang & Shannon, 2005) to identify themes for each aspect of the taxonomy. These themes were considered alongside quantitative data to identify potential reasons for the lack of participant consensus (Tapio et al., 2011). Data analysis was conducted by the first author, who was blinded to the identity of participants during analysis, and results were reviewed by the other authors. Result Part A: Agreement with taxonomy structure and definitions Delphi participants reached consensus on the structure and definitions of the taxonomy in both rounds one and two, with 100% of participants strongly agreeing or agreeing with the overall structure of the taxonomy and at least 88.4% of participants strongly agreeing or agreeing with the sub-components and definitions within every aspect. No participants strongly disagreed with any aspect of the taxonomy. These results are provided in Table II. As consensus was established across both rounds one and two, participants were not asked to rate their level of agreement regarding the structure and definitions in round three. Therefore, the round-three survey contained only the content related to Part B. Table II. Agreement with the structure of taxonomy and definitions (Part A). CSVDisplay Table Part B: Categorisation of assessments using the taxonomy At the end of round three, consensus was established regarding seven out of the 10 components for case study one (parent interview for a child with Autism Spectrum Disorder) and four out of the 10 components for case study two (language sampling for a child experiencing language difficulties at school). The level of agreement with regards to the categorisation of case studies across each taxonomy component is provided in Table III. Table III. Agreement with the categorisation of assessments in case studies (Part B). CSVDisplay Table Final taxonomy The agreed-upon taxonomy has four aspects, labelled in Roman numerals I–IV, each containing a number of components that describe assessments. The finalised structure of the taxonomy after round three is represented in Figure 1 and the finalised definitions of each taxonomy component after round three are provided in Supplementary Table I. Each taxonomy aspect is described below followed by a summary of the components that were not consistently categorised in case studies at the end of round three. The themes identified from participant comments and associated changes to the taxonomy through the Delphi study rounds are summarised in Supplementary Material II. Figure 1. Assessment taxonomy. Display full size Aspect I (Modalities and domains) Aspect I provides terminology for describing the skills that are specifically measured by an assessment and reported on in assessment findings. There are three components: modality, domain and comprehension/production. Modality includes categories spoken and written (including AAC). Domains include semantics, morphosyntax, social abilities & discourse, meta-abilities and executive functions. Assessments are also described as targeting comprehension (reception) or production (expression) of language. The categories in Aspect I are not mutually exclusive. An assessment may target either or both spoken and written modalities, either or both comprehension and production, and target one or more domains. At the end of round three, the categories semantics and executive functioning remained inconsistently selected. Themes from participant comments suggested the following possible reasons for lack of consensus: (1) participants considering other ways an assessment could be conducted or selecting domains that may be involved in completing assessment tasks, but are not the key domains being measured by the assessment; (2) perceived overlap between the categories of semantics and executive functioning; and (3) the high level of information processing required from Delphi participants when reading and applying definitions. Aspect II (Assessment purpose) Aspect II describes the purposes for which assessments are used. There are seven categories that include predict outcome, select intervention, plan dosage (prognostic purposes relating to the identification of possible future needs or supports) and screening, diagnostic, detect change and describe status (analytical purposes related to describing current functioning). These categories are not mutually exclusive as an assessment may have more than one purpose. After round three, consensus on all Aspect II categories was not reached for either case study. Themes from participants comments identified the following reasons for lack of consensus: (1) th