AIRR - ANZCA Institutional Research Repository
Skip navigation
Please use this identifier to cite or link to this item: https://hdl.handle.net/11055/867
Full metadata record
DC FieldValueLanguage
dc.contributor.authorCastanelli DJen_US
dc.contributor.authorMoonen-van Loon JMWen_US
dc.contributor.authorJolly Ben_US
dc.contributor.authorWeller JMen_US
dc.date2018-11-14-
dc.date.accessioned2019-08-05T05:32:04Z-
dc.date.available2019-08-05T05:32:04Z-
dc.date.issued2019-02-
dc.identifier.citation66(2):193-200en_US
dc.identifier.issn1496-8975en_US
dc.identifier.urihttp://hdl.handle.net/11055/867-
dc.description.abstractAbstract PURPOSE: Competency-based anesthesia training programs require robust assessment of trainee performance and commonly combine different types of workplace-based assessment (WBA) covering multiple facets of practice. This study measured the reliability of WBAs in a large existing database and explored how they could be combined to optimize reliability for assessment decisions. METHODS: We used generalizability theory to measure the composite reliability of four different types of WBAs used by the Australian and New Zealand College of Anaesthetists: mini-Clinical Evaluation Exercise (mini-CEX), direct observation of procedural skills (DOPS), case-based discussion (CbD), and multi-source feedback (MSF). We then modified the number and weighting of WBA combinations to optimize reliability with fewer assessments. RESULTS: We analyzed 67,405 assessments from 1,837 trainees and 4,145 assessors. We assumed acceptable reliability for interim (intermediate stakes) and final (high stakes) decisions of 0.7 and 0.8, respectively. Depending on the combination of WBA types, 12 assessments allowed the 0.7 threshold to be reached where one assessment of any type has the same weighting, while 20 were required for reliability to reach 0.8. If the weighting of the assessments is optimized, acceptable reliability for interim and final decisions is possible with nine (e.g., two DOPS, three CbD, two mini-CEX, two MSF) and 15 (e.g., two DOPS, eight CbD, three mini-CEX, two MSF) assessments respectively. CONCLUSIONS: Reliability is an important factor to consider when designing assessments, and measuring composite reliability can allow the selection of a WBA portfolio with adequate reliability to provide evidence for defensible decisions on trainee progression.en_US
dc.titleThe reliability of a portfolio of workplace-based assessments in anesthesia training.en_US
dc.typeJournal Articleen_US
dc.type.contentTexten_US
dc.identifier.journaltitleCanadian Journal of Anesthesia/Journal canadien d'anesthésieen_US
dc.identifier.doi10.1007/s12630-018-1251-7en_US
dc.description.pubmedurihttps://www.ncbi.nlm.nih.gov/pubmed/30430441en_US
dc.type.studyortrialStudyen_US
dc.identifier.studynameUsing workplace-based assessments to make decisions on trainee progression through the ANZCA training programen_US
dc.ispartof.anzcaresearchfoundationYesen_US
item.fulltextNo Fulltext-
item.grantfulltextnone-
item.cerifentitytypePublications-
item.openairetypeJournal Article-
item.openairecristypehttp://purl.org/coar/resource_type/c_18cf-
Appears in Collections:Scholarly and Clinical
Show simple item record

Page view(s)

36
checked on Mar 28, 2024

Google ScholarTM

Check

Altmetric


Items in DSpace are protected by copyright, with all rights reserved, unless otherwise indicated.