Development and testing of the KERNset: an instrument to assess the quality of telephone triage in out-of-hours primary care services.

BACKGROUND: Telephone triage is a core but vulnerable part of the care process at out-of-hours general practitioner (GP) cooperatives. In the Netherlands, different instruments have been used for assessing the quality of telephone triage. These instruments focussed mainly on communicational aspects, and less on the medical quality of triage decisions. Our aim was to develop and test a minimum set of items to assess the quality of telephone triage. METHODS: A national survey among all GP cooperatives in the Netherlands was performed to examine the most important aspects of telephone triage. Next, corresponding items from existing instruments were searched on these topics. Subsequently, an expert panel judged these items on importance, completeness and formulation. The concept KERNset consisted of 24 items about the telephone conversation: 13 medical, ten communicational and one regarding both types. It was pilot tested on measurement characteristics, reliability, validity and variation between triagists. In this pilot study, 114 anonymous calls from four GP cooperatives spread across the Netherlands were judged by three out of eight raters, both internal and external raters. RESULTS: Cronbach's alpha was .94 for the medical items and .75 for the communicational items. Inter-rater reliability: complete agreement between the external raters was 45% and reasonable agreement 73% (difference of maximally one point on the five-point scale). Intra-rater reliability: complete agreement within raters was 55% and reasonable agreement 84%. There were hardly any differences between internal and external raters, but there were differences in strictness between individual raters. The construct validity was confirmed by the high correlation between the general impression of the call and the items of the KERNset. Of the differences within items 19% could be explained by differences between triage nurses, which means the KERNset is able to demonstrate differences between triage nurses. CONCLUSIONS: The KERNset can be used to assess the quality of telephone triage. The validity is good and differences between calls and between triage nurses can be measured. A more intensive training for raters could improve the reliability.

Should we assess clinical performance in single patient encounters or consistent behaviors of clinical performance over a series of encounters? A qualitative exploration of narrative trainee profiles.

BACKGROUND: A variety of tools have been developed to assess performance which typically use a single clinical encounter as a source for making competency inferences. This strategy may miss consistent behaviors. We therefore explored experienced clinical supervisors' perceptions of behavioral patterns that potentially exist in postgraduate general practice trainees expressed as narrative profiles to aid the grading of clinical performance. METHODS: We conducted semistructured interviews with clinical supervisors who had frequently observed clinical performance in trainees. Supervisors were asked to describe which behavioral patterns they had discerned in excellent and underperforming trainees, during different stages of training, in their careers as clinical supervisor. We analyzed the interviews using a grounded theory approach. RESULTS: The analysis resulted in a conceptual framework that distinguishes between desirable and undesirable narrative profiles. The framework consists of two dimensions: doctor-patient interaction and medical expertise. Personal values appear to be a moderating factor. CONCLUSIONS: According to experienced clinical supervisors, consistent behaviors do exist in GP trainees when observing clinical performance over time. The conceptual framework has to be validated by further observational studies to assess its potential for making robust and fair assessments of clinical performance and monitor the development of consultation performance over time.

Exploring typologies of consultation performance using authentic clinical experiences to support learning and assessment in postgraduate medical training.

OBJECTIVE: To validate and refine typologies of consultation performance from previous research to identify learning needs associated with each typology. METHODS: We performed a qualitative study in a General Practice Specialty Training programme, using a two-stage design. First, we selected four exemplars from 80 videotaped consultations of 7 first-year and 6 third-year trainees that reflected the four typologies. We subsequently held individual interviews with clinical supervisors (N = 20) who observed these consultations to identify recurrent trainee behaviours. RESULTS: The 'doctor-patient interaction' dimension from previous research was specified to encompass relationship-building, exploring, structuring, and shared decision-making competencies. Medical expertise was a moderating factor. The attitude and consultation behaviours included in the typologies were validated and we formulated directions for learning based on learning needs identified per typology. CONCLUSION: Supervisors have a shared frame of reference for the behaviours reflecting proficient consultation performance. Serving as a developmental road map, all learning needs emphasised contextual adaptation, calling for an improved balance between patient-centred relationship building and application of medical expertise. PRACTICE IMPLICATIONS: By providing rich and tailored feedback on consultation performance, the refined typologies - albeit subject to additional refinement in future research - may promote the monitoring of individual competence development over time.

Reliability of consultation skills assessments using standardised versus real patients.

OBJECTIVES: Training in and assessment of consultation skills are high on the agenda of vocational training institutes for postgraduate training. There is a need to establish valid and reliable instruments to assess consultation skills in authentic settings. We investigated the number of assessors and observations needed to achieve reliable assessments of the consultation skills of general practice trainees (GPTs) using a communication instrument (MAAS-Global) and either standardised patient (SP) encounters or videotaped real patient (RP) encounters. METHODS: Eight teachers at the Vrije Universiteit (VU) University Medical Centre in Amsterdam attended a training course on the use of the MAAS-Global instrument, which they subsequently used to assess the consultation skills of 53 GPTs in 176 videotaped consultations (102 with SPs, 74 with RPs). All consultations were randomly allocated and assessed by two teachers independently. The reliability of the ratings was estimated using generalisability theory. RESULTS: It was easier to obtain acceptable reliability using RP consultations than SP consultations. Two assessors and five consultations were required to achieve minimal reliability (generalisability coefficient 0.7) with RPs, whereas three assessors and 30 consultations were needed to achieve minimal reliability with SPs. CONCLUSIONS: Inter-observer and context variability in the assessment of the consultation skills of GPTs remains high. To achieve acceptable levels of reliability, large samples of observations are required in both formats, but, interestingly, RP encounters require a smaller sample than SP encounters.

Assessing communication skills of clinical call handlers working at an out-of-hours centre: development of the RICE rating scale.

BACKGROUND: Out-of-hours centres provide telephone support to patients with medical problems. In most of these centres specially-trained nurses handle incoming telephone calls. They assess patients' needs, the degree of urgency, and determine the level of care required. Assessment of the medical problem and the quality of 'care-by-phone' depend on the medical and communication skills of the call handlers. AIM: To develop a valid, reliable, and practical rating scale to evaluate the communication skills of call handlers working at an out-of-hours centre and to improve quality of communication. DESIGN OF STUDY: Qualitative study with focus groups followed by validation of the rating scale and measurement of reliability (internal consistency). SETTING: Out-of-hours centres in the Netherlands. METHOD: A focus group developed the rating scale. Experts with experience in training and evaluating communication skills of medical students and GPs commented on the scale to ensure content validity. The reliability of the rating scale was tested in a pilot in which ten specially-trained assessors scored six telephone calls each. RESULTS: The scale, known as the RICE rating scale, has 17 items divided over four different phases of the telephone consultation: Reason for calling; Information gathering; Conclusion; and Evaluation (RICE). Content validity of the scale was assessed by two experts. Reliability of the scale tested in the pilot was 0.73 (Cronbach's alpha). CONCLUSION: Establishing a rating scale to assess the communication skills of call handlers which meets common scientific demands, such as content validity and reliability, proved successful. This instrument can be used to give feedback to call handlers.

Characteristics of communication guidelines that facilitate or impede guideline use: a focus group study.

BACKGROUND: The quality of doctor-patient communication has a major impact on the quality of medical care. Communication guidelines define best practices for doctor patient communication and are therefore an important tool for improving communication. However, adherence to communication guidelines remains low, despite doctors participating in intensive communication skill training. Implementation research shows that adherence is higher for guidelines in general that are user centred and feasible, which implies that they are consistent with users' opinions, tap into users' existing skills and fit into existing routines. Developers of communication guidelines seem to have been somewhat negligent with regard to user preferences and guideline feasibility. In order to promote the development of user centred and practicable communication guidelines, we elicited user preferences and identified which guideline characteristics facilitate or impede guideline use. METHODS: Seven focus group interviews were conducted with experienced GPs, communication trainers (GPs and behavioural scientists) and communication learners (GP trainees and medical students) and three focus group interviews with groups of GP trainees only. All interviews were transcribed and analysed qualitatively. RESULTS: The participants identified more impeding guideline characteristics than facilitating ones. The most important impeding characteristic was that guidelines do not easily fit into GPs' day-to-day practice. This is due to rigidity and inefficiency of communication guidelines and erroneous assumptions underpinning guideline development. The most important facilitating characteristic was guideline structure. Guidelines that were structured in distinct phases helped users to remain in control of consultations, which was especially useful in complicated consultations. CONCLUSION: Although communication guidelines are generally considered useful, especially for structuring consultations, their usefulness is impaired by lack of flexibility and applicability to practice routines. User centred and feasible guidelines should combine the advantages of helping doctors to structure consultations with flexibility to tailor communication strategies to specific contexts and situations.

Effective improvement of doctor-patient communication: a randomised controlled trial.

BACKGROUND: Doctor-patient communication is an essential component of general practice. Improvement of GPs' communication patterns is an important target of training programmes. Available studies have so far failed to provide conclusive evidence of the effectiveness of educational interventions to improve doctor-patient communication. AIM: To examine the effectiveness of a learner-centred approach that focuses on actual needs, to improve GPs' communication with patients. DESIGN OF STUDY: Randomised controlled trial. SETTING: One hundred volunteer GPs in the Netherlands. METHOD: The intervention identified individual GPs' deficiencies in communication skills by observing authentic consultations in their own surgery. This performance assessment was followed by structured activities in small group meetings, aimed at remedying the identified shortcomings. Outcomes were measured using videotaped consultations in the GPs' own surgery before and after the intervention. Communication skills were rated using the MAAS-Global, a validated checklist. RESULTS: The scores in the intervention group demonstrated a significant improvement compared with those of the control group (95% confidence interval = 0.04 to 0.75). The effect size was moderate to large (d-value = 0.66). The level of participation significantly contributed to the effectiveness. Largest improvement was found on patient-centred communication skills. CONCLUSION: The approach of structured individual improvement activities based on performance assessment is more effective in improving communication skills than current educational activities.

Communication guidelines as a learning tool: an exploration of user preferences in general practice.

OBJECTIVE: To explore characteristics of written communication guidelines that enhance the success of training aimed at the application of the recommendations in the guidelines. METHODS: Seven mixed focus groups were held consisting of communication skill teachers and communication skill learners and three groups with only learners. Analysis was done in line with principles of grounded theory. RESULTS: Five key attributes of guidelines for communication skill training were identified: complexity, level of detail, format and organization, type of information, and trustworthiness/validity. The desired use of these attributes is related to specific educational purposes and learners' expertise. The low complexity of current communication guidelines is appreciated, but seems ad odds with the wish for more valid communication guidelines. CONCLUSIONS: Which guideline characteristics are preferred by users depends on the expertise of the learners and the educational purpose of the guideline. PRACTICE IMPLICATIONS: Communication guidelines can be improved by modifying the key attributes in line with specific educational functions and learner expertise. For example: the communication guidelines used in GP training in the Netherlands, seem to offer an oversimplified model of doctor patient communication. This model may be suited for undergraduate learning, but does not meet the validity demands of physicians in training.

How doctors move from generic goals to specific communicative behavior in real practice consultations.

OBJECTIVE: To understand how recommendations for communication can be brought into alignment with clinical communication routines, we explored how doctors select communicative actions during consultations. METHODS: We conducted stimulated recall interviews with 15 GPs (general practitioners), asking them to comment on recordings of two consultations. The data analysis was based on the principles of grounded theory. RESULTS: A model describing how doctors select communicative actions during consultations was developed. This model illustrates how GPs constantly adapt their selection of communicative actions to their evaluation of the situation. These evaluations culminate in the selection of situation-specific goals. These multiple and often dynamic goals require constant revision and adaptation of communication strategies, leading to constant readjustments of the selection of communicative actions. When selecting consultation goals GPs weigh patients' needs and preferences as well as the medical situation and its consequences. CONCLUSIONS: GPs' selection of communicative actions during consultations is situational and goal driven. PRACTICE IMPLICATIONS: To help doctors develop communicative competence tailored to the specific situation of each consultation, holistic communication training courses, which pay attention to the selection of consultation goals and matching communication strategies besides training specific communication skills, seem preferable to current generic communication skills training.