Evaluating a web-based health risk assessment with tailored feedback: what does an expert focus group yield compared to a web-based end-user survey?

BACKGROUND: Increasingly, Web-based health applications are developed for the prevention and management of chronic diseases. However, their reach and utilization is often disappointing. Qualitative evaluations post-implementation can be used to inform the optimization process and ultimately enhance their adoption. In current practice, such evaluations are mainly performed with end-user surveys. However, a review approach by experts in a focus group may be easier to administer and might provide similar results. OBJECTIVE: The aim of this study was to assess whether industrial design engineers in a focus group would address the same issues as end users in a Web-based survey when evaluating a commercial Web-based health risk assessment (HRA) with tailored feedback. METHODS: Seven Dutch companies used the HRA as part of their corporate health management strategy. Employees using the HRA (N=2289) and 10 independent industrial designers were invited to participate in the study. The HRA consisted of four components: (1) an electronic health questionnaire, (2) biometric measurements, (3) laboratory evaluation, and (4) individually tailored feedback generated by decision support software. After participating in the HRA as end users, both end users and designers evaluated the program. End users completed an evaluation questionnaire that included a free-text field. Designers participated in a focus group discussion. Constructs from user satisfaction and technology acceptance theories were used to categorize and compare the remarks from both evaluations. RESULTS: We assessed and qualitatively analyzed 294 remarks of 189 end users and 337 remarks of 6 industrial designers, pertaining to 295 issues in total. Of those, 137 issues were addressed in the end-user survey and 148 issues in the designer focus group. Only 7.3% (10/137) of the issues addressed in the survey were also addressed in the focus group. End users made more remarks about the usefulness of the HRA and prior expectations that were not met. Designers made more remarks about how the information was presented to end users, quality of the feedback provided by the HRA, recommendations on the marketing and on how to create more unity in the design of the HRA, and on how to improve the HRA based on these issues. CONCLUSIONS: End-user surveys should not be substituted for expert focus groups. Issues identified by end users in the survey and designers in the focus group differed considerably, and the focus group produced a lot of new issues. The issues addressed in the focus group often focused on different aspects of user satisfaction and technology acceptance than those addressed by the survey participants; when they did focus on the same aspects, then the nature of issues differed considerably in content.

Design and Perceived Value of a Novel Solution for Asynchronous Communication in Radiology.

RATIONALE AND OBJECTIVES: Communication with and within the Radiology Department is typically initiated over phone, face-to-face or general-purpose chat, causing frequent interruptions, additional mental workload, workflow inefficiencies and diagnostic errors. We developed and evaluated a new communication solution that aims to reduce avoidable interruptions caused by technologist-radiologist communication. MATERIALS AND METHODS: Following an iterative design process with future end users, a scalable web-based software solution, RadConnect, was developed enabling a chat-based communication workflow between a technologist and a radiologist. As a first experimental implementation, technologists can send categorized tickets to a radiology section account. Radiologists receive the tickets in a worklist that is prioritized by urgency. Consented radiologists and technologists performed scripted tasks in 2 hr sessions and completed a structured questionnaire on perceived value and comparison to standard communication modes. RESULTS: Of 17 participants from three academic European institutes, 65% (11/17) believed they would use RadConnect frequently; 53% (9/17) believed that it reduces phone calls >80%; and 88% (15/17) believed it adds value compared to general-purpose enterprise chat applications. DISCUSSION: Participants recognized the value of RadConnect especially its categorized tickets, prioritized worklist and role-based interaction model. Inter-institute differences in perceived value of RadConnect may have been caused by technologist-radiologist proximity and communication alternatives in the institutions. CONCLUSION: Chat-based role-based communication might be a viable mode of communication between technologists and radiologists to reduce avoidable interruptions. Tailoring the chat solution to the needs of and tightly integrated with the radiology workflow is valued by future end users after exposure to the tool in a simulated environment.

Evaluation of end-user satisfaction among employees participating in a web-based health risk assessment with tailored feedback.

BACKGROUND: Web technology is increasingly being used to provide individuals with health risk assessments (HRAs) with tailored feedback. End-user satisfaction is an important determinant of the potential impact of HRAs, as this influences program attrition and adherence to behavioral advice. OBJECTIVE: The aim of this study was to evaluate end-user satisfaction with a web-based HRA with tailored feedback applied in worksite settings, using mixed (quantitative and qualitative) methods. METHODS: Employees of seven companies in the Netherlands participated in a commercial, web-based, HRA with tailored feedback. The HRA consisted of four components: 1) a health and lifestyle assessment questionnaire, 2) a biometric evaluation, 3) a laboratory evaluation, and 4) tailored feedback consisting of a personal health risk profile and lifestyle behavior advice communicated through a web portal. HRA respondents received an evaluation questionnaire after six weeks. Satisfaction with different parts of the HRA was measured on 5-point Likert scales. A free-text field provided the opportunity to make additional comments. RESULTS: In total, 2289 employees participated in the HRA program, of which 637 (27.8%) completed the evaluation questionnaire. Quantitative analysis showed that 85.6% of the respondents evaluated the overall HRA positively. The free-text field was filled in by 29.7 % of the respondents (189 out of 637), who made 315 separate remarks. Qualitative evaluation of these data showed that these respondents made critical remarks. Respondents felt restricted by the answer categories of the health and lifestyle assessment questionnaire, which resulted in the feeling that the corresponding feedback could be inadequate. Some respondents perceived the personal risk profile as unnecessarily alarming or suggested providing more explanations, reference values, and a justification of the behavioral advice given. Respondents also requested the opportunity to discuss the feedback with a health professional. CONCLUSIONS: Most people were satisfied with the web-based HRA with tailored feedback. Sources of dissatisfaction were limited opportunities for providing additional health information outside of the predefined health and lifestyle assessment questionnaire and insufficient transparency on the generation of the feedback. Information regarding the aim and content of the HRA should be clear and accurate to prevent unrealistic expectations among end-users. Involving trusted health professionals in the implementation of web-based HRAs may enhance the use of and confidence in the HRA.

An online survey to study the relationship between patients' health literacy and coping style and their preferences for self-management-related information.

OBJECTIVE: To evaluate patients' preferences for message features and assess their relationships with health literacy, monitor-blunter coping style, and other patient-dependent characteristics. METHODS: Patients with coronary heart disease completed an internet-based survey, which assessed health literacy and monitor-blunter coping style, as well as various other patient characteristics such as sociodemographics, disease history, and explicit information preferences. To assess preferences for message features, nine text sets differing in one of nine message features were composed, and participants were asked to state their preferences. RESULTS: The survey was completed by 213 patients. For three of the nine text sets, a relationship was found between patient preference and health literacy or monitor-blunter coping style. Patients with low health literacy preferred the text based on patient experience. Patients with a monitoring coping style preferred information on short-term effects of their treatment and mentioning of explicit risks. Various other patient characteristics such as marital status, social support, disease history, and age also showed a strong association. CONCLUSION: Individual differences exist in patients' preferences for message features, and these preferences relate to patient characteristics such as health literacy and monitor-blunter coping style.

Using personas as an intermediate construct in the development of tailored messages: a case study.

Tailoring health education messages to patients' preferences for message style is believed to increase patients' susceptibility to the given advice. This paper presents a persona-centered approach towards creating tailored health messages for chronically ill patients. A case study of tailoring messages to the preferences of patients with coronary heart disease illustrates the approach and shows the need for patient-centered data collection so that personas reflect patients' preferences. Based on these personas, a manageable set of tailored messages can be created in a step-by-step approach.

Changes in the cardiac rehabilitation workflow process needed for the implementation of a self-management system.

E-health interventions are of a growing importance for self-management of chronic conditions. This study aimed to describe the process adaptions that are needed in cardiac rehabilitation (CR) to implement a self-management system, called MyCARDSS. We created a generic workflow model based on interviews and observations at three CR clinics. Subsequently, a workflow model of the ideal situation after implementation of MyCARDSS was created. We found that the implementation will increase the complexity of existing working procedures because 1) not all patients will use MyCARDSS, 2) there is a transfer of tasks and responsibilities from professionals to patients, and 3) information in MyCARDSS needs to be synchronized with the EPR system for professionals.

A qualitative participatory study to identify experiences of coronary heart disease patients to support the development of online self-management services.

OBJECTIVE: Web-based self-management services remain underutilized in current practice. Our aim was to gain insight into disease and self-management experiences of patients in early and progressive stages of coronary heart disease (CHD), to understand moderating effects of daily life experiences on the utilization of web-based self-management services and preconditions for use. METHODS: We applied generative research techniques, which stem from the field of product design and are characterized by the use of creative processes. Three groups of patients with CHD received a sensitizing package to document and reflect on their health, and were subsequently either interviewed or participated in a focus group session. RESULTS: In total, 23 patients participated in this study. Emerging themes were (1) fear for recurrent events, (2) experiences with professional care, (3) the perceived inability to prevent disease progression, (4) the desire to go on living without thinking about the disease every day, (5) the social environment as a barrier to or facilitator for self-management, and (6) the need for information tailored to personal preferences. CONCLUSION: How patients experience their disease varies between stable and post-acute stages, as well as between early and progressive stages of CHD. Patients in post-acute stages of the disease seem to be most amenable to support, while patients in stable stages want to live their life without being reminded of their disease. In the context of self-management, web-based services should be adapted to the variation in needs that occur in the different stages of CHD and new strategies to fit such services to these needs should be developed. Furthermore, they should be tailored to patients' individual health situation and preferences, support patient empowerment, and manage expectations regarding the progression of their disease.

Initiation of health-behaviour change among employees participating in a web-based health risk assessment with tailored feedback.

BACKGROUND: Primary prevention programs at the worksite can improve employee health and reduce the burden of cardiovascular disease. Programs that include a web-based health risk assessment (HRA) with tailored feedback hold the advantage of simultaneously increasing awareness of risk and enhancing initiation of health-behaviour change. In this study we evaluated initial health-behaviour change among employees who voluntarily participated in such a HRA program. METHODS: We conducted a questionnaire survey among 2289 employees who voluntarily participated in a HRA program at seven Dutch worksites between 2007 and 2009. The HRA included a web-based questionnaire, biometric measurements, laboratory evaluation, and tailored feedback. The survey questionnaire assessed initial self-reported health-behaviour change and satisfaction with the web-based HRA, and was e-mailed four weeks after employees completed the HRA. RESULTS: Response was received from 638 (28%) employees. Of all, 86% rated the program as positive, 74% recommended it to others, and 58% reported to have initiated overall health-behaviour change. Compared with employees at low CVD risk, those at high risk more often reported to have increased physical activity (OR 3.36, 95% CI 1.52-7.45). Obese employees more frequently reported to have increased physical activity (OR 3.35, 95% CI 1.72-6.54) and improved diet (OR 3.38, 95% CI 1.50-7.60). Being satisfied with the HRA program in general was associated with more frequent self-reported initiation of overall health-behaviour change (OR 2.77, 95% CI 1.73-4.44), increased physical activity (OR 1.89, 95% CI 1.06-3.39), and improved diet (OR 2.89, 95% CI 1.61-5.17). CONCLUSIONS: More than half of the employees who voluntarily participated in a web-based HRA with tailored feedback, reported to have initiated health-behaviour change. Self-reported initiation of health-behaviour change was more frequent among those at high CVD risk and BMI levels. In general employees reported to be satisfied with the HRA, which was also positively associated with initiation of health-behaviour change. These findings indicate that among voluntary participating employees a web-based HRA with tailored feedback may motivate those in greatest need of health-behaviour change and may be a valuable component of workplace health promotion programs.