Effect of rater training on scoring performance & scale-specific expertise amongst occupational therapists: a single-group pre-post-test study

Background: When occupational therapists (OTs) use the McGill Ingestive Skills Assessment (MISA), they observe and score dysphagic clients’ mealtime performance according to the criteria of 36 MISA-items. This is a highly... [ view full abstract ]

Background:
When occupational therapists (OTs) use the McGill Ingestive Skills Assessment (MISA), they observe and score dysphagic clients’ mealtime performance according to the criteria of 36 MISA-items. This is a highly complex task, which might introduce unwanted variability in measurement scores. In order to enhance the quality of the data collected in a validation study of a Danish version of MISA, the author developed a rater training programme [Helker & Stange 2013]. The purpose of the present study was to evaluate the effect of the training on scoring performance and scale-specific expertise amongst 81 OTs.
Method:
The training effects were evaluated pre- to post-training using percentage exact agreement (PA) of scored MISA items of a case-vignette and a Likert scale self-report of scale-specific expertise.
Results:
PA increased significantly from pre- to post-training (Z = 4.404, p<0.001), although items for which the case-vignette reflected severely deficient mealtime performance appeared most difficult to score. The OTs scale specific expertise improved significantly (knowledge: Z =7.857, p<0.001 and confidence: Z =7.838, p<0.001).
Conclusion:
Rater training improved OTs scoring performance when using the Danish MISA as well as their perceived scale-specific expertise. However, further studies addressing different training approaches and durations are warranted.
Application to Practice:
The training programme could be used in undergraduate and postgraduate dysphagia education initiatives to help OTs understanding of the content and the scoring criteria for each aspect of dysphagic clients’ mealtime performance, thus developing observation skills as well as recognizing and avoiding the most common errors in measurement scores.