Reliability of evaluation for communication competency in pharmaceutical objective structured clinical examination (OSCE).

ABSTRACT

In the objective structured clinical examination (OSCE), which for pharmaceutical students training, adequate methods should be used for evaluating a student's skill and aptitude for good communication in a medical interview. However, the reliability of the evaluation methods used in the pharmaceutical OSCE has not been investigated sufficiently. In this study, we reviewed the evaluation scores and video recordings obtained in a pharmaceutical OSCE trial, and examined the reasons for disagreement in the scores between two raters. We had two experienced raters in medical communication re-evaluate the students using the vide images, and compared their scores with those on the examination day. The ratio of disagreement was 14.5% (87/600 items in 30 students), and the reason for disagreement could not be identified for 63 items that evaluated communication skills such as 'actively listen' and 'empathy'. A comparison of the scores on examination day and those on re-evaluation revealed a possible reason for the disagreement; the use of a checklist, i.e. binary scores, with criteria that differed between the raters. We suggest that the items used for a detailed performance evaluation be selected carefully and that rating scales be used in order to perform an adequate evaluation, especially regarding communication skill and aptitude.