Lessons learned from an end-of-year OSCE assessment of freshman medical student skills following the inaugural year of a limited ultrasound curriculum

INTRODUCTION: Ultrasound education and training is gaining widespread acceptance as a valuable addition to an undergraduate medical curriculum. The University of Arkansas for Medical Sciences just completed the inaugural year... [ view full abstract ]

INTRODUCTION: Ultrasound education and training is gaining widespread acceptance as a valuable addition to an undergraduate medical curriculum. The University of Arkansas for Medical Sciences just completed the inaugural year of a longitudinal ultrasound curriculum designed for first year medical students. The goal of the ultrasound curriculum was to introduce students to sonographic imaging, reinforce important anatomical relationships, view dynamic physiologic responses, and begin to develop sonographic skills. The curriculum also provided additional opportunities for enhancing standardized patient (SP) interaction skills. The students utilized hand held GE Vscan dual probe devices to initially scan themselves and then SPs during five separate faculty guided one-hour sessions scheduled throughout the year (5 students/device/SP). An end-of-the-year pass/fail ultrasound objective structured clinical exam (OSCE) was introduced in order to assess student learning and evaluate program effectiveness.

METHODS: Students were asked to prepare for a single station OSCE that would include one of four potential scanning targets: 1) parasternal long axis view of the heart; 2) thyroid gland and carotid sheath contents; 3) right kidney/liver relationship; or 4) popliteal fossa contents. Students were provided 24-hour on campus access to two dual probe devices and offered an optional OSCE review session held in our active learning center during which they could scan multiple SPs for each potential target. Written instructions outlined performance expectations including: greeting the patient, describing the procedure, positioning and draping the patient, properly holding the probe, obtaining and saving an appropriate scan and identification of a list of structures for each scanning target. A silent evaluator in the room during the OSCE used a 10-item grading rubric to evaluate student performance (1 point given for complete/accurate performance, a half point given for incomplete/suboptimal performance and no points given for a failed/missed step). Anonymous student evaluations included a 14 question pre-OSCE paper survey and an online end-of-year ultrasound curriculum survey.

RESULTS: Although some students took advantage of the ultrasound devices on reserve, more than half of the class attended the optional OSCE review session. The pre-OSCE survey indicated that students generally felt prepared for the OSCE (4.13; LIKERT; n=134/172 students) and were glad it was only pass/fail (4.56). Students felt best prepared to obtain a scan of the popliteal fossa (4.40) and thyroid gland and carotid sheath contents (actual OSCE target) (4.31) versus the heart (4.09) or kidney/liver (3.96) targets. All students finished the OSCE within the required 5-minute time limit with 76% finishing before the one-minute warning. The average score on the OSCE was 90.2 ± 0.6 (mean ± SEM; n=172) with 40% of the students scoring 100% and only two students falling below 70%. Evaluators indicated that less than 15% of the students appeared tentative or nervous. Evaluators had difficulty confirming students actually saved an image as required and post-OSCE inspection of the devices indicated that 18% of them failed to do so. A quarter of the class held the probe incorrectly, 20% were unable to obtain a proper scan of the thyroid and carotid sheath contents, and nearly 30% failed to correctly identify all five required structures. An analysis of student performance revealed that the inclusion of non-ultrasound related patient interaction items in the OSCE grading rubric helped a significant portion (~20%) of students pass the OSCE. Preliminary responses from an ongoing end-of-the-year survey indicate students enjoyed the ultrasound OSCE (3.74) and thought it was a good learning experience (4.24).

CONCLUSIONS: Our results confirm that the use of an end-of-year OSCE is a viable option following a limited exposure ultrasound curriculum. Students viewed the pass/fail examination positively. Performance on the OSCE highlighted deficiencies in the level of student skills that will form the basis for curricular redesign and future educational targets for the evolving inaugural M2 curriculum. An end-of-year OSCE will be included next years curriculum for both the M1 and M2 students.