Interviewer bias in selection of anaesthesia Fellows: A single-institution quality assessment study

Abstract

Fellowships are competitive training posts, often in a subspecialty area. We performed a quality assessment of potential interviewer bias on anaesthesia Fellow selection. After research locality approval, we analysed interview scores for all Fellowship applications to our department over six years. Panel interviewers participated in a structured interview process, asking a series of standardised questions to rate applicants. A mixed model analysis of total applicant rating with crossed effects of applicants and interviewers was used. A total of 94 applicants were interviewed by 27 panel members, with between two and four panel members per interview, giving a total of 329 applicant ratings. The random effect of applicants accounted for 45.8% of total variance in ratings (95% confidence intervals (CI) for intraclass correlation (ICC) 35.8%-57.2%) while interviewer effects accounted for 13.4% of total variance (95% CI for ICC 5.3%-30.0%). We found no evidence of bias for most potential sources after analysing multiple applicant and interviewer factors. After adjusting for interviewer training programme, applicants from other training programmes were rated a mean of 1.87 points lower than Australian and New Zealand College of Anaesthetists (ANZCA) applicants (95% CI 0.62-3.12, P = 0.003) and 1.84 points lower than Royal College of Anaesthetists (RCoA) applicants (95% CI 0.37-3.32, P = 0.014). After adjusting for applicant gender, female clinicians rated applicants 1.12 points higher (95% CI 0.19-2.06, P = 0.019) on average than male clinicians. The observed differences in interview scores amongst male and female clinicians and lower scores in applicants from programmes other than ANZCA/RCoA were small, and require confirmation in independent studies.