A core questionnaire for the assessment of patient satisfaction in academic hospitals in The Netherlands: development and first results in a nationwide study.

BACKGROUND: Patient satisfaction is one of the relevant indicators of quality of care; however, measuring patient satisfaction had been criticised. A major criticism is that many instruments are not reliable and/or valid. The instruments should have enough discriminative power for benchmarking of the results. OBJECTIVE: To develop a "core questionnaire for the assessment of patient satisfaction in academic hospitals" (COPS) that is reliable and appropriate for benchmarking patient satisfaction results. RESEARCH DESIGN: The development of COPS, the testing of its psychometric quality and its use in eight Dutch academic hospitals in three national comparative studies in 2003, 2005 and 2007 are described in this study. Results were reported only if they were significant (p<0.05) and relevant (also Cohen d>0.2). RESULTS: The questionnaire was returned in 2003 by 40,678 patients (77,450 sent, 53%) and by 40,248 patients (75,423 sent, 53%) in 2005. In 2007, the questionnaire was returned by 45,834 patients (87,137, 53%). The six dimensions have good Cronbach α's, varying from 0.79 to 0.88.The results of every item were reported to the individual hospital. A benchmark overview showed the overall comparison of all specialties of the eight hospitals for the clinic and outpatient departments. The 2007 measurement showed relevant differences in satisfaction on two dimensions in the clinical setting. CONCLUSIONS: COPS is shown to be a feasible and reliable instrument to measure the satisfaction of patients in Dutch academic hospitals. It allows comparison of hospitals and gives benchmark information on a hospital as well as data on specialty levels and previous measurements, including best practices.

Setting priorities for improving the preoperative assessment clinic: the patients' and the professionals' perspective.

BACKGROUND: The quality of the preoperative assessment clinic (PAC) is determined by many factors. Patients' experiences are important indicators, but often overlooked. We prepare to set priorities to improve the PAC by obtaining detailed patients' feedback on the quality of the PAC, and establishing the value patients and professionals attach to different care aspects, using the Patient Experiences with the Preoperative Assessment Clinic questionnaire. METHODS: The PAC's standard of service was determined for five care aspects (dimensions), using patients' feedback. The importance of a dimension to patients was determined by calculating the effects of the dimensions on patients' overall appraisal. In addition, professionals were asked to rate the importance of the different care aspects. RESULTS: Patients had the most positive experiences with the nurse, and the least positive experiences with waiting. However, waiting was least important to patients. When combining the PAC's standard of service with the value given to the dimensions by patients and professionals separately, we found in both instances that waiting was in greatest need of improvement. This was followed by reception, the anaesthetist, remaining experiences, and finally the nurse. CONCLUSIONS: Quality improvement of the PAC can be achieved by obtaining patients' feedback on the quality, determine a PAC's standard of service, recognize service areas that require improvement, and identify actions appropriate to bring about improvement. The value patients and professionals attach to different aspects of care can then be used to prioritize improvements.

Patient Experiences with the Preoperative Assessment Clinic (PEPAC): validation of an instrument to measure patient experiences.

BACKGROUND: Presently, no comprehensive and validated questionnaire to measure patient experiences of the preoperative assessment clinic (PAC) is available. We developed and validated the Patient Experiences with the Preoperative Assessment Clinic (PEPAC) questionnaire, which can be used for quantitative measurements of patient experiences of the PAC. METHODS: We adapted the National Health Service outpatient questionnaire, incorporating questions specific for anaesthesiology. To make the PEPAC appropriate for quantitative measurements, dimensions and single items suitable for statistical analysis were constructed. Each dimension consists of multiple items measuring the same aspect of care. Reliability was established by computing Cronbach's alpha coefficients. Construct validity was assessed by correlating the dimensions with the patient's overall appraisal (Pearson's r). These dimensions should explain a substantial level of variance of the patients' overall appraisal; therefore, regression analysis was performed. RESULTS: After a pilot phase, the questionnaire was sent to 700 consecutive patients (response 74%). Five scales measuring five dimensions of patient experiences were constructed. Cronbach's alpha ranged from 0.56 to 0.84, supporting reliability of the PEPAC. Correlations between the dimensions and patients' overall appraisal ranged from 0.22 to 0.56. Collectively, the five scales explained 51% of patients' overall appraisal. CONCLUSIONS: The PEPAC is a comprehensive, reliable, and validated questionnaire to measure patient experiences with the PAC. It might be a useful tool to identify the service areas of the PAC that require improvement and to determine which actions can bring about improvement.

Using standardized video cases for assessment of medical communication skills: reliability of an objective structured video examination by computer.

OBJECTIVE: Using standardized video cases in a computerized objective structured video examination (OSVE) aims to measure cognitive scripts underlying overt communication behavior by questions on knowledge, understanding and performance. In this study the reliability of the OSVE assessment is analyzed using the generalizability theory. METHODS: Third year undergraduate medical students from the Academic Medical Center of the University of Amsterdam answered short-essay questions on three video cases, respectively about history taking, breaking bad news, and decision making. Of 200 participants, 116 completed all three video cases. Students were assessed in three shifts, each using a set of parallel case editions. About half of all available exams were scored independently by two raters using a detailed rating manual derived from the other half. Analyzed were the reliability of the assessment, the inter-rater reliability, and interrelatedness of the three types of video cases and their parallel editions, by computing a generalizability coefficient G. RESULTS: The test score showed a normal distribution. The students performed relatively well on the history taking type of video cases, and relatively poor on decision making and did relatively poor on the understanding ('knows why/when') type of questions. The reliability of the assessment was acceptable (G = 0.66). It can be improved by including up to seven cases in the OSVE. The inter-rater reliability was very good (G = 0.93). The parallel editions of the video cases appeared to be more alike (G = 0.60) than the three case types (G = 0.47). DISCUSSION: The additional value of an OSVE is the differential picture that is obtained about covert cognitive scripts underlying overt communication behavior in different types of consultations, indicated by the differing levels of knowledge, understanding and performance. The validation of the OSVE score requires more research. CONCLUSION AND PRACTICE IMPLICATIONS: A computerized OSVE has been successfully applied with third year undergraduate medical students. The test score meets psychometric criteria, enabling a proper discrimination between adequately and poorly performing students. The high inter-rater reliability indicates that a single rater is permitted.

Summative assessment of medical students' communication skills and professional attitudes through observation in clinical practice.

UNLABELLED: To establish medical students' professional competence for the medical profession, we designed a standardized observation procedure and the Amsterdam Attitude and Communication Scale (AACS) with nine five-point scale items, for summative assessment of their communication skills and professional attitudes. This study examines the reliability of the AACS assessment in clinical practice. In the Academic Medical Centre, Amsterdam, The Netherlands, the performance of 442 fifth year clinical students was judged six times in two settings: behaviour in clinical practice was judged independently twice by a doctor and a nurse; one videotaped patient interview was judged independently by a doctor and by a psychologist. The final mark was obtained by averaging ratings across all six assessments. Raters were 88 doctors, 29 nurses, and three psychologists. MAIN OUTCOME MEASURES: Standard errors (SEs) for absolute judgements indicate measurement precision. Precision of AACS scores is considered sufficient with SEs smaller than 0.25. Multi-disciplinary assessment of students' clinical performance using the AACS is feasible and sufficiently precise (with an overall mean of 3.97 and standard deviation of 0.55, the absolute SE is 0.21). Judgements of behaviour in the clinic were more precise (SEs range from 0.11 to 0.16) than judgements of videotaped interviews (SEs are 0.25 and 0.29). The procedure is sufficiently precise if five or six assessments are combined.

Assessment of medical communication skills by computer: assessment method and student experiences.

BACKGROUND: A computer-assisted assessment (CAA) program for communication skills designated ACT was developed using the objective structured video examination (OSVE) format. This method features assessment of cognitive scripts underlying communication behaviour, a broad range of communication problems covered in 1 assessment, highly standardised assessment and rating procedures, and large group assessments without complex organisation. SETTING: The Academic Medical Centre (AMC) at the University of Amsterdam, the Netherlands. Aims To describe the development of the AMC Communication Test (ACT); to describe our experiences with the examination and rating procedures; to present test score descriptives, and to present the students' opinions of ACT. DESIGN: The ACT presents films on history taking, breaking bad news and shared decision making. Each film is accompanied by 3 types of short essay questions derived from our assessment model: "knows", "knows why/when" and "knows how". Evaluation questions about ACT were integrated into the assessment. Participants A total of 210 third year medical undergraduates were assessed. This study reports on the 110 (53%) students who completed all evaluation questions. RESULTS: Marking 210 examinations took about 17 days. The test score matched a normal distribution and showed a good level of discrimination of the students. About 75% passed the examination. Some support for the validity of our assessment model was found in the students' differential performance on the 3 types of questions. The ACT was well received. Student evaluations confirmed our efforts to develop realistic films that related well to the communication training programme. CONCLUSIONS: The ACT is a useful assessment method which complements interpersonal assessment methods for the evaluation of the medical communication skills of undergraduates.

Timing of patient satisfaction assessment: effect on questionnaire acceptability, completeness of data, reliability and variability of scores.

The present study compared the performance of a multidimensional patient satisfaction questionnaire according to the timing of questionnaire administration. Comparisons were made in terms of: (a) the completeness and representativeness of the data set (number of missing questionnaires, number missing item responses, respondents' representativeness to the target population); (b) the questionnaire acceptability to respondents (time and difficulty to complete); (c) the questionnaire reliability; and (d) variability of scores. One hundred and ten consecutive breast cancer patients hospitalised for surgery were randomised between being sent the comprehensive assessment of satisfaction with care (CASC) at 2 weeks (T(2W)) or at 3 months (T(3M)) after hospital discharge. The time to complete the CASC was shorter at T(3M) than at T(2W) and the mean percentage of item omission was lower at T(3M) (1.68) than at T(2W) (3.82). However, the response rate was much higher at T(2W) (87%) than at T(3M) (66%), making item omission non-significant. At both times of questionnaire administration samples were equally biased towards patients having undergone a less invasive surgery. Moreover, the multi-item scales of the CASC demonstrated adequate internal consistency coefficients, except the general satisfaction scale at T(3M), and fairly symmetrical distribution of scores. Response rate should be considered in priority. This criteria favoured an administration of the CASC shortly after discharge. Besides in a cancer patient population care experience and perception may vary in a 6 weeks time lapse. The timing of assessment needs to be clearly specified in cancer patients satisfaction survey.

Assessment of medical students' communicative behaviour and attitudes: estimating the reliability of the use of the Amsterdam attitudes and communication scale through generalisability coefficients.

It is widely accepted that adequate attitudes and communicative skills are among the essential objectives in medical education. The Amsterdam attitude and communication scale (AACS) was developed to assess communicative skills and professional attitudes of medical students. More specifically, it was designed to evaluate the clinical behaviour of clerks to establish their suitability for the medical profession. The AACS covers nine dimensions. Moreover, an overall judgement of the student's performance is included. The present paper reports first results on the reliability of the use of the AACS. Data were collected in the course of an AACS training programme for future judges: senior medical and nursing staff members (N=98). Participants judged three videotapes of clerks interviewing patients at the bedside. For the assessment of videotapes, the first four dimensions of the AACS and the overall judgement are relevant. By applying Generalisability Theory to the training data we can forecast the reliability of the AACS in practice and gain insight in the number of raters that is needed to achieve sufficient reliability in clinical practice. If clerk behaviour is rated by six judges, summative assessment is sufficiently precise, i.e. <0.25. When using the full AACS, covering 10 items, the same number of judges is needed. Scores on individual AACS items are not sufficiently reliable. In conclusion, the results indicate that students' behaviour can be evaluated in a reliable manner using the AACS as long as enough judges and items are involved.

Assessment of quality of care in an oncology institute using information on patients' satisfaction.

OBJECTIVE: To evaluate the feasibility of conducting a patient satisfaction survey in the oncology hospital setting, using a multidimensional patient satisfaction questionnaire to be completed at home. METHODS: Socio-demographic and clinical data were collected for 133 consecutive patients. Patients were asked to complete the European Organisation for Research and Treatment of Cancer QLQ-C30 (version 2.0) just before hospital discharge and the Comprehensive Assessment of Satisfaction with Care at home 2 weeks after discharge. RESULTS: Respondents (73% of patients approached) were younger, hospitalized for a shorter time and presented less appetite loss, nausea and vomiting and better physical and role functioning than non-responders. The aspects of care for which patients wanted the most improvement were associated with the provision of medical information. In multivariate analyses, longer hospital stay was associated with higher satisfaction with all aspects of medical and nursing care, most probably because patients discharged early were not assured of continuity of care and lacked information regarding self-care at home. Higher global quality of life was associated with higher satisfaction with all aspects of care, suggesting the potential contribution of patient satisfaction to the patients' well-being. CONCLUSIONS: Conducting a patient satisfaction survey in an oncology hospital setting proved feasible; however, further surveys should attempt to obtain the opinion of patients with more severe physical conditions. The assessment of the patients' satisfaction provided indications for improvement of care in a particular hospital. Although the results of this study are specific to one hospital, the methods could be reproduced in other hospital settings, but may possibly lead to other conclusions.

A comprehensive assessment of satisfaction with care: preliminary psychometric analysis in French, Polish, Swedish and Italian oncology patients.

Satisfaction with care may be closely related to quality of life in cancer patients. This evaluation is especially relevant when quality of care is considered. The present study assessed whether equivalent scaling properties could be found in a comprehensive assessment of satisfaction with care (CASC) administered in cancer patients from French, Polish and Swedish oncology settings, in comparison to the scaling properties previously evidenced in the CASC with an Italian sample. A total of 140, 186 and 133 oncology patients were approached in France, Poland and Sweden, respectively. Specific items in the CASC were identified as consistently omitted across country samples. Multitrait scaling analysis on an item-grouping adapted for the French, Polish, Swedish and Italian samples provided excellent internal consistencies and convergent validity estimates. Discriminant validity proved less satisfactory, evidencing overlap between hypothesised care dimensions across country samples. The identification of omitted or overlapping items will lead to the design of a revised CASC version to further test in larger cross-cultural samples.