Evaluating a web-based health risk assessment with tailored feedback: what does an expert focus group yield compared to a web-based end-user survey?

BACKGROUND: Increasingly, Web-based health applications are developed for the prevention and management of chronic diseases. However, their reach and utilization is often disappointing. Qualitative evaluations post-implementation can be used to inform the optimization process and ultimately enhance their adoption. In current practice, such evaluations are mainly performed with end-user surveys. However, a review approach by experts in a focus group may be easier to administer and might provide similar results. OBJECTIVE: The aim of this study was to assess whether industrial design engineers in a focus group would address the same issues as end users in a Web-based survey when evaluating a commercial Web-based health risk assessment (HRA) with tailored feedback. METHODS: Seven Dutch companies used the HRA as part of their corporate health management strategy. Employees using the HRA (N=2289) and 10 independent industrial designers were invited to participate in the study. The HRA consisted of four components: (1) an electronic health questionnaire, (2) biometric measurements, (3) laboratory evaluation, and (4) individually tailored feedback generated by decision support software. After participating in the HRA as end users, both end users and designers evaluated the program. End users completed an evaluation questionnaire that included a free-text field. Designers participated in a focus group discussion. Constructs from user satisfaction and technology acceptance theories were used to categorize and compare the remarks from both evaluations. RESULTS: We assessed and qualitatively analyzed 294 remarks of 189 end users and 337 remarks of 6 industrial designers, pertaining to 295 issues in total. Of those, 137 issues were addressed in the end-user survey and 148 issues in the designer focus group. Only 7.3% (10/137) of the issues addressed in the survey were also addressed in the focus group. End users made more remarks about the usefulness of the HRA and prior expectations that were not met. Designers made more remarks about how the information was presented to end users, quality of the feedback provided by the HRA, recommendations on the marketing and on how to create more unity in the design of the HRA, and on how to improve the HRA based on these issues. CONCLUSIONS: End-user surveys should not be substituted for expert focus groups. Issues identified by end users in the survey and designers in the focus group differed considerably, and the focus group produced a lot of new issues. The issues addressed in the focus group often focused on different aspects of user satisfaction and technology acceptance than those addressed by the survey participants; when they did focus on the same aspects, then the nature of issues differed considerably in content.

Effectiveness of a web-based health risk assessment with individually-tailored feedback on lifestyle behaviour: study protocol.

BACKGROUND: Physical inactivity, unhealthy dietary habits, smoking and high alcohol consumption are recognized risk factors for cardiovascular disease and cancer. Web-based health risk assessments with tailored feedback seem promising in promoting a healthy lifestyle. This study evaluates the effectiveness of a web-based health risk assessment with individually-tailored feedback on lifestyle behaviour, conducted in a worksite setting. METHODS/DESIGN: The web-based health risk assessment starts with a questionnaire covering socio-demographic variables, family and personal medical history, lifestyle behaviour and psychological variables. Prognostic models are used to estimate individual cardiovascular risks. In case of high risk further biometric and laboratory evaluation is advised. All participants receive individually-tailored feedback on their responses to the health risk assessment questionnaire. The study uses a quasi-experimental design with a waiting list control group. Data are collected at baseline (T0) and after six months (T1). Within each company, clusters of employees are allocated to either the intervention or the control group. Primary outcome is lifestyle behaviour, expressed as the sum of five indicators namely physical activity, nutrition, smoking behaviour, alcohol consumption, and symptoms of burnout. Multilevel regression analysis will be used to answer the main research question and to correct for clustering effects. Baseline differences between the intervention and control group in the distribution of characteristics with a potential effect on lifestyle change will be taken into account in further analyses using propensity scores. DISCUSSION: This study will increase insight into the effectiveness of health risk assessments with tailored feedback and into conditions that may modify the effectiveness. This information can be used to design effective interventions for lifestyle behaviour change among employees. TRIAL REGISTRATION: Dutch Trial Register NTR8148.

Evaluation of end-user satisfaction among employees participating in a web-based health risk assessment with tailored feedback.

BACKGROUND: Web technology is increasingly being used to provide individuals with health risk assessments (HRAs) with tailored feedback. End-user satisfaction is an important determinant of the potential impact of HRAs, as this influences program attrition and adherence to behavioral advice. OBJECTIVE: The aim of this study was to evaluate end-user satisfaction with a web-based HRA with tailored feedback applied in worksite settings, using mixed (quantitative and qualitative) methods. METHODS: Employees of seven companies in the Netherlands participated in a commercial, web-based, HRA with tailored feedback. The HRA consisted of four components: 1) a health and lifestyle assessment questionnaire, 2) a biometric evaluation, 3) a laboratory evaluation, and 4) tailored feedback consisting of a personal health risk profile and lifestyle behavior advice communicated through a web portal. HRA respondents received an evaluation questionnaire after six weeks. Satisfaction with different parts of the HRA was measured on 5-point Likert scales. A free-text field provided the opportunity to make additional comments. RESULTS: In total, 2289 employees participated in the HRA program, of which 637 (27.8%) completed the evaluation questionnaire. Quantitative analysis showed that 85.6% of the respondents evaluated the overall HRA positively. The free-text field was filled in by 29.7 % of the respondents (189 out of 637), who made 315 separate remarks. Qualitative evaluation of these data showed that these respondents made critical remarks. Respondents felt restricted by the answer categories of the health and lifestyle assessment questionnaire, which resulted in the feeling that the corresponding feedback could be inadequate. Some respondents perceived the personal risk profile as unnecessarily alarming or suggested providing more explanations, reference values, and a justification of the behavioral advice given. Respondents also requested the opportunity to discuss the feedback with a health professional. CONCLUSIONS: Most people were satisfied with the web-based HRA with tailored feedback. Sources of dissatisfaction were limited opportunities for providing additional health information outside of the predefined health and lifestyle assessment questionnaire and insufficient transparency on the generation of the feedback. Information regarding the aim and content of the HRA should be clear and accurate to prevent unrealistic expectations among end-users. Involving trusted health professionals in the implementation of web-based HRAs may enhance the use of and confidence in the HRA.