Close menu

SURE

Sunderland Repository records the research produced by the University of Sunderland including practice-based research and theses.

An inexpensive Retrospective Standard Setting Method Based on Item Facilities

McLachlan, JC, Robertson, K. Alex, Weller, Bridget and Sawdon, Marina (2021) An inexpensive Retrospective Standard Setting Method Based on Item Facilities. BMC Medical Education. ISSN 1472-6920

Item Type: Article

Abstract

Standard setting is one of the most challenging aspects of assessment in high-stakes healthcare settings. The Angoff methodology is widely used, but poses a number of challenges, including conceptualisation of the just-passing candidate, and the time-cost of implementing the method. Cohen methodologies are inexpensive and rapid but rely on the performance of an individual candidate. A new method of standard setting, based on the entire cohort and every item, would be valuable. Methods We identified Borderline candidates by reviewing their performance across all assessments in an academic year. We plotted the item scores of the Borderline candidates in comparison with Facility for the whole cohort and fitted curves to the resulting distribution. Results We propose that for any given Item, an equation of the form y ≈ C.eFx where y is the Facility of Borderline candidates on that Item, x is the observed Item Facility of the whole cohort, and C and F are constants, predicts the probable Facility for Borderline candidates over the test, in other words, the cut score for Borderline candidates. We describe ways of estimating C and F in any given circumstance, and suggest typical values arising from this particular study: that C = 12.3 and F = 0.021. Conclusions We propose that C and F are relatively stable, and that the equation y = 12.3.e0.021x can rapidly be applied to the item Facility for every item. The average value represents the cut score for the assessment as a whole. This represents a novel retrospective method based on test takers. Compared to the Cohen method which draws on one score and one candidate, this method draws on all items and candidates in a test. It can be used to standard set a whole test, or a particular item where the predicted Angoff score is very different from the observed Facility.

[img] Microsoft Word
McLachlan_Sawdon et al 2020 accepted version.docx - Accepted Version
Available under License Creative Commons Attribution.

Download (58kB)

More Information

Depositing User: Marina Sawdon

Identifiers

Item ID: 12741
Identification Number: https://doi.org/10.1186/s12909-020-02418-5
ISSN: 1472-6920
URI: http://sure.sunderland.ac.uk/id/eprint/12741
Official URL: https://bmcmededuc.biomedcentral.com/articles/10.1...

Users with ORCIDS

ORCID for JC McLachlan: ORCID iD orcid.org/0000-0001-5493-2645
ORCID for Marina Sawdon: ORCID iD orcid.org/0000-0002-8668-257X

Catalogue record

Date Deposited: 03 Dec 2020 08:58
Last Modified: 21 Jan 2021 13:23

Contributors

Author: JC McLachlan ORCID iD
Author: Marina Sawdon ORCID iD
Author: K. Alex Robertson
Author: Bridget Weller
Author: JC McLachlan
Author: K. Alex Robertson
Author: Bridget Weller
Author: Marina Sawdon

University Divisions

Faculty of Health Sciences and Wellbeing > School of Medicine

Subjects

Education > Higher Education

Actions (login required)

View Item (Repository Staff Only) View Item (Repository Staff Only)