Fixing the Rubber Band in Communication Skills Assessment of Interdepartmental OSCE Stations

Background. Comparing uncalibrated outcome of clinical skills assessment is challenging. This study proposes reliable and valid comparison of communication skills. Our aim is to determine the differences in communication... [ view full abstract ]

Background. Comparing uncalibrated outcome of clinical skills assessment is challenging. This study proposes reliable and valid comparison of communication skills. Our aim is to determine the differences in communication skills (CS) of the year 4 OSCEs in an undergraduate Irish medical school to identify the characteristics of each test using calibrated assessment forms.
Method. Three academic terms of the year 4 OSCE (n=380) were analysed retrospectively. The OSCE contains of interdepartmental Psychiatry and General Practice stations designed. The MAAS-Global caliber is the percentage of station checklist items that can be considered as ‘true’ CS. Initial station scores were multiplied by the caliber of MAAS-Global items and acknowledged as MAAS-Global scores. The reliability of the OSCE was calculated with G-Theory analysis whereas nested ANOVA was used to compare mean scores of all years.
Results. The calibrated outcome of Psychiatry stations shows higher Maas-Global scores than General Practice stations. The portion of section 3 of the MAAS-Global was larger than section 1 and 2. Average raw OSCE scores of six circuits over the past three years of General Practice station scores ranged from 58(sd=6) to 64(sd= 6). MAAS-Global scores ranged from 44(sd=4) to 48(sd=5). For Psychiatry stations scores ranged from 61(sd=8) to 70(sd=10) whereas MAAS-Global scores ranged from 59(sd=8) to 66(sd=8). MAAS-Global CS scores in Pyschiatry stations are significantly higher (p<0.03).
Conclusion. The higher students' scores in Psychiatry stations is a direct result of higher MAAS-Global caliber scores compared to the General Practice stations. In this study, the comparison of outcome measurement between different interdepartmental station checklists can be considered as valid and reliable. The most striking finding is the fact that we replaced a rubber band measure with true characteristics of CS measures for individual station. The MAAS-Global is used as a single validated instrument and is suggested as gold standard.