[Other] The impact of faculty development interventions on supervisor narrative feedback quality in competency-based medical education Copy

Objectives: Improving the feedback that supervisors provide trainees in medical education is challenging. The development of automated methods of rating the quality of narrative feedback has created an opportunity to quantify the impact of interventions focused on this goal.

Methods: We used a previously developed natural language processing model based on the Quality Assessment of Learning (QuAL) score to rate the quality of each comment in our Royal College Emergency Medicine specialty training program from July 2018 to June 2024. We quantified the effectiveness of three faculty development interventions (e-mail feedback, audit and feedback workshops, and assessment form modifications) by comparing pre- and post-intervention QuAL scores.

Results: During the study period 9709 narrative comments were scored. The average QuAL score rose from 3.2/5 in 2019 to 4.5/5 in 2024. All interventions resulted in an overall improvement in the mean QuAL score as measured by assessor weighted mean: individual e-mail with suggestions for improvement (35.1%), assessment form template change (24.3%), workshop (19.1%) and individual e-mail thanking them for their high-quality comments (11.0%).

Conclusions: By combining a multi-faceted faculty development approach with a scalable measurement of our outcome, we have demonstrated an improvement in supervisor narrative comment quality and quantified the impact of each intervention. While it is likely that the interventions were complementary, individual e-mails to faculty with suggestions for improvement and modifications to the assessment form were associated with largest improvements in quality. These results may guide to other programs seeking to improve the quality of supervisor narrative comments.

Keywords: Competency-based medical education; Faculty development; Narrative comments; Quality.