<p>The estimated top 20 most strict and most lenient scoring performances by a rater in a year for all five years. Each point is annotated with the rater’s ID and several raters appear more than once (for performances in different years). The bars indicate the standard deviation of the estimates.</p
‘Risk of bias’ graph: Review authors’ assessments for each risk of bias item presented as percentage...
Bias-corrected efficiency scores per country (the closer to ‘1’, the more efficient).</p
<p>All inter-rater reliabilities were measured using Krippendorff's alpha for ordinal data. *Include...
<p>Mean estimated absolute bias and reliability with standard deviations, broken down by year of sco...
Alternative methods to correct for rater leniency/stringency effects (i.e., rater bias) in performa...
<p>The limit of agreement was calculated from differences between composite reference standard (CRS)...
How common patterns of rater errors may be detected in a large-scale performance assessment setting ...
The present study examined the long-term usefulness of estimated parameters used to adjust the score...
<p>The bars of the rater bar charts were ranked according to rating activity. <i>r</i>–number of rat...
This paper first analyzed two studies on rater factors and rating criteria to raise the problem of r...
Effect of number of raters on reliability, standard error and predictive validity of scoring.</p
Of the potential sources of construct irrelevant variance or unwanted variability in performance ass...
Of the potential sources of construct irrelevant variance or unwanted variability in performance ass...
<p>Scores ranged from 1 to 3, with higher scores indicating likelihood of committing the bias. Stars...
<p>Note in particular the large variability in the consistency of the label ‘Other’ across datasets,...
‘Risk of bias’ graph: Review authors’ assessments for each risk of bias item presented as percentage...
Bias-corrected efficiency scores per country (the closer to ‘1’, the more efficient).</p
<p>All inter-rater reliabilities were measured using Krippendorff's alpha for ordinal data. *Include...
<p>Mean estimated absolute bias and reliability with standard deviations, broken down by year of sco...
Alternative methods to correct for rater leniency/stringency effects (i.e., rater bias) in performa...
<p>The limit of agreement was calculated from differences between composite reference standard (CRS)...
How common patterns of rater errors may be detected in a large-scale performance assessment setting ...
The present study examined the long-term usefulness of estimated parameters used to adjust the score...
<p>The bars of the rater bar charts were ranked according to rating activity. <i>r</i>–number of rat...
This paper first analyzed two studies on rater factors and rating criteria to raise the problem of r...
Effect of number of raters on reliability, standard error and predictive validity of scoring.</p
Of the potential sources of construct irrelevant variance or unwanted variability in performance ass...
Of the potential sources of construct irrelevant variance or unwanted variability in performance ass...
<p>Scores ranged from 1 to 3, with higher scores indicating likelihood of committing the bias. Stars...
<p>Note in particular the large variability in the consistency of the label ‘Other’ across datasets,...
‘Risk of bias’ graph: Review authors’ assessments for each risk of bias item presented as percentage...
Bias-corrected efficiency scores per country (the closer to ‘1’, the more efficient).</p
<p>All inter-rater reliabilities were measured using Krippendorff's alpha for ordinal data. *Include...
DOI
Add to ORKG