Evaluation Considerations for Secondary Uses of Clinical Data: Principles for an Evidence-based Approach to Policy and Implementation of Secondary Analysis.

Objectives: To set the scientific context and then suggest principles for an evidence-based approach to secondary uses of clinical data, covering both evaluation of the secondary uses of data and evaluation of health systems and services based upon secondary uses of data. Method: Working Group review of selected literature and policy approaches. Results: We present important considerations in the evaluation of secondary uses of clinical data from the angles of governance and trust, theory, semantics, and policy. We make the case for a multi-level and multi-factorial approach to the evaluation of secondary uses of clinical data and describe a methodological framework for best practice. We emphasise the importance of evaluating the governance of secondary uses of health data in maintaining trust, which is essential for such uses. We also offer examples of the re-use of routine health data to demonstrate how it can support evaluation of clinical performance and optimize health IT system design. Conclusions: Great expectations are resting upon "Big Data" and innovative analytics. However, to build and maintain public trust, improve data reliability, and assure the validity of analytic inferences, there must be independent and transparent evaluation. A mature and evidence-based approach needs not merely data science, but must be guided by the broader concerns of applied health informatics.

Improving Evaluation to Address the Unintended Consequences of Health Information Technology:. a Position Paper from the Working Group on Technology Assessment & Quality Development.

BACKGROUND AND OBJECTIVES: With growing use of IT by healthcare professionals and patients, the opportunity for any unintended effects of technology to disrupt care health processes and outcomes is intensified. The objectives of this position paper by the IMIA Working Group (WG) on Technology Assessment and Quality Development are to highlight how our ongoing initiatives to enhance evaluation are also addressing the unintended consequences of health IT. METHODS: Review of WG initiatives Results: We argue that an evidence-based approach underpinned by rigorous evaluation is fundamental to the safe and effective use of IT, and for detecting and addressing its unintended consequences in a timely manner. We provide an overview of our ongoing initiatives to strengthen study design, execution and reporting by using evaluation frameworks and guidelines which can enable better characterization and monitoring of unintended consequences, including the Good Evaluation Practice Guideline in Health Informatics (GEP-HI) and the Statement on Reporting of Evaluation Studies in Health Informatics (STARE-HI). Indicators to benchmark the adoption and impact of IT can similarly be used to monitor unintended effects on healthcare structures, processes and outcome. We have also developed EvalDB, a web-based database of evaluation studies to promulgate evidence about unintended effects and are developing the content for courses to improve training in health IT evaluation. CONCLUSION: Evaluation is an essential ingredient for the effective use of IT to improve healthcare quality and patient safety. WG resources and skills development initiatives can facilitate a proactive and evidence-based approach to detecting and addressing the unintended effects of health IT.

STARE-HI - Statement on Reporting of Evaluation Studies in Health Informatics: explanation and elaboration.

BACKGROUND: Improving the quality of reporting of evaluation studies in health informatics is an important requirement towards the vision of evidence-based health informatics. The STARE-HI - Statement on Reporting of Evaluation Studies in health informatics, published in 2009, provides guidelines on the elements to be contained in an evaluation study report. OBJECTIVES: To elaborate on and provide a rationale for the principles of STARE-HI and to guide authors and readers of evaluation studies in health informatics by providing explanatory examples of reporting. METHODS: A group of methodologists, researchers and editors prepared the present elaboration of the STARE-HI statement and selected examples from the literature. RESULTS: The 35 STARE-HI items to be addressed in evaluation papers describing health informatics interventions are discussed one by one and each is extended with examples and elaborations. CONCLUSION: The STARE-HI statement and this elaboration document should be helpful resources to improve reporting of both quantitative and qualitative evaluation studies. Evaluation manuscripts adhering to the principles will enable readers of such papers to better place the studies in a proper context and judge their validity and generalizability, and thus in turn optimize the exploitation of the evidence contained therein. LIMITATIONS: This paper is based on experiences of a group of editors, reviewers, authors of systematic reviews and readers of the scientific literature. The applicability of the details of these principles has to evolve as a function of their use in practice.

Evidence Based Health Informatics: 10 Years of Efforts to Promote the Principle. Joint Contribution of IMIA WG EVAL and EFMI WG EVAL.

OBJECTIVES: To present the importance of Evidence-based Health Informatics (EBHI) and the ethical imperative of this approach; to highlight the work of the IMIA Working Group on Technology Assessment and Quality Improvement and the EFMI Working Group on Assessment of Health Information Systems; and to introduce the further important evaluation and evidence aspects being addressed. METHODS: Reviews of IMIA, EFMA and other initiatives, together with literature reviews on evaluation methods and on published systematic reviews. RESULTS: Presentation of the rationale for the health informatics domain to adopt a scientific approach by assessing impact, avoiding harm, and empirically demonstrating benefit and best use; reporting of the origins and rationale of the IMIA- and EQUATOR-endorsed Statement on Reporting of Evaluation Studies in Health Informatics (STARE-HI) and of the IMIA WG's Guideline for Good Evaluation Practice in Health Informatics (GEP-HI); presentation of other initiatives for objective evaluation; and outlining of further work in hand on usability and indicators; together with the case for development of relevant evaluation methods in newer applications such as telemedicine. The focus is on scientific evaluation as a reliable source of evidence, and on structured presentation of results to enable easy retrieval of evidence. CONCLUSIONS: EBHI is feasible, necessary for efficiency and safety, and ethically essential. Given the significant impact of health informatics on health systems, care delivery and personal health, it is vital that cultures change to insist on evidence-based policies and investment, and that emergent global moves for this are supported.

eHealth in Europe - Status and Challenges.

OBJECTIVES: To present European reflections on the concept of eHealth and emerging challenges related to further development of eHealth in Europe. METHODS: A survey with 10 questions was distributed to representatives of the national member associations of the European Federation of Medical Informatics (EFMI). RESULTS: The results document a shift from a constricting ICT-orientation to development of the entire health system where eHealth strategies, organizational change, and appropriate technological infrastructure are singled out as important aspects. CONCLUSION: There are urgent needs to ensure that eHealth strategies and policies for further design and deployment of eHealth applications support sociable services and innovations in health care.

Systematic prioritization of the STARE-HI reporting items. An application to short conference papers on health informatics evaluation.

BACKGROUND: We previously devised and published a guideline for reporting health informatics evaluation studies named STARE-HI, which is formally endorsed by IMIA and EFMI. OBJECTIVE: To develop a prioritization framework of ranked reporting items to assist authors when reporting health informatics evaluation studies in space restricted conference papers and to apply this prioritization framework to measure the quality of recent health informatics conference papers on evaluation studies. METHOD: We deconstructed the STARE-HI guideline to identify reporting items. We invited a total of 111 authors of health informatics evaluation studies, reviewers and editors of health Informatics conference proceedings to score those reporting items on a scale ranging from "0 - not necessary in a conference paper" through to "10 - essential in a conference paper" by a web-based survey. From the responses we derived a mean priority score. All evaluation papers published in proceedings of MIE2006, Medinfo2007, MIE2008 and AMIA2008 were rated on these items by two reviewers. From these ratings a priority adjusted completeness score was computed for each paper. RESULTS: We identified 104 reporting items from the STARE-HI guideline. The response rate for the survey was 59% (66 out of 111). The most important reporting items (mean score ≥9) were "Interpret the data and give an answer to the study question - (in Discussion)", "Whether it is a laboratory, simulation or field study - (in Methods-study design)" and "Description of the outcome measure/evaluation criteria - (in Methods-study design)". Per reporting area the statistically more significant important reporting items were distinguished from less important ones. Four reporting items had a mean score ≤6. The mean priority adjusted completeness of evaluation papers of recent health informatics conferences was 48% (range 14-78%). CONCLUSION: We produced a ranked list of reporting items from STARE-HI according to their prioritized relevance for inclusion in space-limited conference papers. The priority adjusted completeness scores demonstrated room for improvement for the analyzed conference papers. We believe that this prioritization framework is an aid to improving the quality and utility of conference papers on health informatics evaluation studies.

Health Informatics 3.0 and other increasingly dispersed technologies require even greater trust: promoting safe evidence-based health informatics. Contribution of the IMIA Working Group on Technology Assessment & Quality Development in Health Informatics.

BACKGROUND: Health informatics is generally less committed to a scientific evidence-based approach than any other area of health science, which is an unsound position. Introducing the new Web 3.0 paradigms into health IT applications can unleash a further great potential, able to integrate and distribute data from multiple sources. The counter side is that it makes the user and the patient evermore dependent on the 'black box' of the system, and the re-use of the data remote from the author and initial context. Thus anticipatory consideration of uses, and proactive analysis of evidence of effects, are imperative, as only when a clinical technology can be proven to be trustworthy and safe should it be implemented widely - as is the case with other health technologies. OBJECTIVES: To argue for promoting evidence-based health informatics as systems become more powerful and pro-active yet more dispersed and remote; and evaluation as the means of generating the necessary scientific evidence base. To present ongoing IMIA and EFMI initiatives in this field. METHODS: Critical overview of recent developments in health informatics evaluation, alongside the precedents of other health technologies, summarising current initiatives and the new challenges presented by Health Informatics 3.0. RESULTS: Web 3.0 should be taken as an opportunity to move health informatics from being largely unaccountable to one of being an ethical and responsible science-based domain. Recent and planned activities of the EFMI and IMIA working groups have significantly progressed key initiatives. CONCLUSIONS: Concurrent with the emergence of Web 3.0 as a means of new-generation diffuse health information systems comes an increasing need for an evidence-based culture in health informatics.

Use of Headings and Classifications by Physicians in Medical Narratives of EHRs: An evaluation study in a Finnish hospital.

OBJECTIVE: The purpose of this study was to describe and evaluate patient care documentation by hospital physicians in EHRs and especially the use of national headings and classifications in these documentations. MATERIAL AND METHODS: The initial material consisted of a random sample of 3,481 medical narratives documented in EHRs during the period 2004-2005 in one department of a Finnish central hospital. The final material comprised a subset of 1,974 medical records with a focus on consultation requests and consultation responses by two specialist groups from 871 patients. This electronic documentation was analyzed using deductive content analyses and descriptive statistics. RESULTS: The physicians documented patient care in EHRs principally as narrative text. The medical narratives recorded by specialists were structured with headings in less than half of the patient cases. Consultation responses in general were more often structured with headings than consultation requests. The use of classifications was otherwise insignificant, but diagnoses were documented as ICD 10 codes in over 50% of consultation responses by both medical specialties. CONCLUSION: There is an obvious need to improve the structuring of narrative text with national headings and classifications. According to the findings of this study, reason for care, patient history, health status, follow-up care plan and diagnosis are meaningful headings in physicians' documentation. The existing list of headings needs to be analyzed within a consistent unified terminology system as a basis for further development. Adhering to headings and classifications in EHR documentation enables patient data to be shared and aggregated. The secondary use of data is expected to improve care management and quality of care.

STARE-HI -statement on reporting of evaluation studies in health informatics.

OBJECTIVE: Development of guidelines for publication of evaluation studies of Health Informatics applications. METHODS: An initial list of issues to be addressed in reports on evaluation studies was drafted based on experiences as editors and reviewers and as authors of systematic reviews , taking into account guidelines for reporting of medical research. This list has been discussed in several rounds by an increasing number of experts in Health Informatics evaluation during conferences and by using e-mail. RESULTS: A set of STARE-HI principles to be addressed in papers describing evaluations of Health Informatics interventions is presented. These principles include formulation of title and abstract, of introduction (e.g. scientific background, study objectives), study context (e.g. organizational setting, system details), methods (e.g. study design, outcome measures), results (e.g. study findings, unexpected observations) and discussion and conclusion. CONCLUSION: A comprehensive list of principles relevant for properly describing Health Informatics evaluations has been developed. When manuscripts submitted to Health Informatics journals and general medical journals adhere to these aspects, readers will be better positioned to place the studies in a proper context and judge their validity and generalisability. STARE-HI may also be used for study planning and hence positively influence the quality of evaluation studies in Health Informatics. We believe that better publication of (both quantitative and qualitative) evaluation studies is an important step toward the vision of evidence-based Health Informatics. LIMITATIONS: This study is based on experiences from editors, reviewers, authors of systematic reviews and readers of the scientific literature. The applicability of the principles has not been evaluated in real practice. Only when authors start to use these principles for reporting, shortcomings in the principles will emerge.

Cost-effectiveness of infliximab in the treatment of rheumatoid arthritis in clinical practice.

OBJECTIVE: We evaluated the cost-effectiveness of infliximab therapy in Finnish RA patients in a real-life clinical setting and identified factors influencing it, using the national register of biological treatment (ROB-FIN). METHODS: A cost-utility analysis was performed, derived from EQ-5D, and related to HAQ score and disease activity using multiple regression. QALYs were calculated based on these utilities, using patient-level data up to the last control registered. Cost-effectiveness analyses included costs per ACR50 responder, and costs per low DAS28 score (<3.2) achieved, in combination with a clinically significant improvement (>1.2). The costs considered were direct medical costs of infliximab and cost of intravenous infusion. Patient-level costs were calculated based on dose and dosage frequency, and were related to the difference in QALYs resulting from infliximab therapy. RESULTS: The 297 patients had been treated with infliximab for an average of 21 months. The HAQ score and patient's global assessment improved significantly on infliximab therapy. More than two-thirds of the patients achieved a clinically important improvement in HAQ. A QALY gain occurred in 76%. 35% of these had an incremental cost-effectiveness ratio of < or =40,000 Euro/QALY gained, the median cost being 51,884 Euro. The cost per QALY gained was significantly lower for patients achieving an ACR50 response at 3, 12 and 24 months. CONCLUSION: Treatment with infliximab and aiming at ACR50 response appears cost-effective, remembering the restrictions of an observational study set up. Current Care guidelines, which require sufficient disease control when deciding on continuing biological therapy, get support from these findings.

National health IT services in Finland.

OBJECTIVES: In 2002 a decision was reached to set up a nation-wide electronic health record system in Finland. The legal framework of actors with the necessary mandate was approved in the parliament in December 2006. A set of standards and norms have been selected that all health care actors need to follow. Functional specifications of the services were completed in 2006. Setting up the centralized health IT services begins in 2007. Centralization of patient record data allows the reorganization of health service providers to take place at local and regional levels according to need. The services allow users to access patient records securely from anywhere with the provision that they have the right to access private patient data. METHODS: The functionality of the services and the necessary infrastructure has been agreed to in projects and working groups involving users, experts, key stakeholders and vendors. RESULTS: The legal framework was approved in the parliament in December 2006. The functional specifications of the centralized health IT services were finalized in 2006. CONCLUSIONS: The implementation of the services will start in 2007.

Success and failure factors in the regional health information system design process--results from a constructive evaluation study.

OBJECTIVES: To identify success and failure factors in the design process of a regional health information system. METHODS: A constructive evaluation study including interviews, observations, usability study and document analysis. RESULTS: Modelling was found to be a key element for the successful implementation of a health information system. The developed service chain model helped to define use cases and to implement seamless service chains. User participation in the design process was a success factor resulting in good user acceptance and signs of positive impacts on work practices. Evaluation study also helped system developers to guide the system's further development. An important failure factor identified was the lack of semantic interoperability of the system components. CONCLUSIONS: The results emphasize the socio-technical nature of health information systems. The starting point for development should be thorough insight into the health care work practices where the information systems are to be used. Successful system design should start from modelling of work processes, data and information flows and definition of concepts and their relations. Health informatics as a scientific discipline provides theories and models for the design and development process.

Factors influencing success and failure of health informatics systems--a pilot Delphi study.

OBJECTIVES: The aim is to gain information on factors influencing success and failure for Health Informatics applications from a group of medical informaticians. METHODS: Based on the presentations at a special topic conference on success and failure in Health ICT and analysis of the proceedings, we conducted a Delphi study on success and failure aspects. RESULTS: A total of 110 success factors and 27 failure criteria were identified, distributed on categories like functional, organizational, behavioral, technical, managerial, political, cultural, legal, strategy, economy, education and user acceptance. These factors and criteria were rated for six different system types. Unanimously it was agreed that "collaboration and co-operation" and "setting goals and courses" are "essential for the success" of clinical systems, and "user acceptance" for educational systems. Similarly, the score "essential in order to avoid a failure" were given unanimously on clinical systems for "response rate and other performance measures" and on administrative systems for "not understanding the organizational context" with "not understanding or foreseeing the extent to which the new IT-system affects the organization, its structure and/or work procedures" as the highest scoring sub-item. CONCLUSIONS: All success factors and failure criteria were considered relevant by the Delphi expert panel. There is no small set of relevant factors or indicators, but success or failure of a Health ICT depends on a large set of issues. Further, clinical systems and decision support systems depend on more factors than other systems.

Citric acid as a pH-regulating additive in granules and the tablet matrix in enteric-coated formulations for colon-specific drug delivery.

Colon-specific drug-delivery systems have been extensively investigated over the last decade. The aim of the study reported here was to investigate whether times of commencement of drug liberation and absorption could be controlled by varying the amount of citric acid in granule cores or in the tablet matrix in enteric-coated multiple-unit tablets. One of the most important aims was to determine the optimal amounts and locations of citric acid in formulations intended as drugs targeted at the colon. Ibuprofen was used as the model drug. Drug release rates were studied in phosphate buffer at pH 6.8 and 7.4. A gradient dissolution study at pH 1.2, 6.8 and 7.4 was undertaken with two formulations. Drug absorption was studied by means of bioavailability tests. We concluded that the drug release rate could be controlled in vitro by changing the amount of citric acid in granule cores or the tablet matrix. In vivo tests confirmed that between 10 and 15% citric acid in the tablet matrix delayed the commencement of drug absorption most. This kind of formulations could be suitable for preparation of colon-specific dosage forms. It is probably unnecessary to include citric acid in granule cores. No logical correlation between in vitro and in vivo results was obtained.

Citric acid as excipient in multiple-unit enteric-coated tablets for targeting drugs on the colon.

Delivery of drugs to the large bowel has been extensively investigated during the last decade. The aim of this study was to investigate whether enteric-coated tablets could be made from enteric-coated matrix granules and drug release targeted to the colon. Whether in vitro drug release rate and in vivo absorption could be delayed by adding citric acid to the granules and/or to the tablet matrix was also studied. Ibuprofen was used as model drug because it is absorbed throughout the gastrointestinal tract. Eudragit S and Aqoat AS-HF were used as enteric polymers. Drug release rates were studied at different pH levels and drug absorption was studied in bioavailability tests. The conclusion was that citric acid retarded in vitro drug release when used in multiple-unit tablets. In vivo absorption of ibuprofen was markedly delayed when citric acid was included in both granules and tablet matrix. Further studies are needed to determine the optimal amount of citric acid in formulations.

On conceptualization of a decision support system in health informatics.

A decision support system can be approached from two major disciplinary perspectives, those of information systems science (IS) and artificial intelligence (AI). We present in this study an extended ontology for a decision support system in health informatics, which is founded on experience from related research fields as well as being informed by our case studies. The ontology emphasises the need to cover environmental and contextual variables as an integral part of a decision support systems development methodology. With the addition of these variables, the focus in decision support systems development shifts from a task ontology towards a domain ontology. The results of this study help the system developers to take the system's context into account through the set of defined variables that are linked to the application domain. These variables explicate relevant constructs and present a vocabulary for a decision support system. However, applying the ontology requires a more thorough analysis of the domain and therefore more qualified resources for systems development. This indicates the need to focus more on education and training in health informatics.

Roles of the heat shock transcription factors in regulation of the heat shock response and beyond.

The heat shock response, characterized by increased expression of heat shock proteins (Hsps) is induced by exposure of cells and tissues to extreme conditions that cause acute or chronic stress. Hsps function as molecular chaperones in regulating cellular homeostasis and promoting survival. If the stress is too severe, a signal that leads to programmed cell death, apoptosis, is activated, thereby providing a finely tuned balance between survival and death. In addition to extracellular stimuli, several nonstressful conditions induce Hsps during normal cellular growth and development. The enhanced heat shock gene expression in response to various stimuli is regulated by heat shock transcription factors (HSFs). After the discovery of the family of HSFs (i.e., murine and human HSF1, 2, and 4 and a unique avian HSF3), the functional relevance of distinct HSFs is now emerging. HSF1, an HSF prototype, and HSF3 are responsible for heat-induced Hsp expression, whereas HSF2 is refractory to classical stressors. HSF4 is expressed in a tissue-specific manner; similar to HSF1 and HSF2, alternatively spliced isoforms add further complexity to its regulation. Recently developed powerful genetic models have provided evidence for both cooperative and specific functions of HSFs that expand beyond the heat shock response. Certain specialized functions of HSFs may even include regulation of novel target genes in response to distinct stimuli.

Genomic organization and promoter analysis of the human heat shock factor 2 gene.

Heat shock factor 2 (HSF2) is a member of the heat shock transcription factor family, which appears to be activated during differentiation and development rather than on cellular stress. Here we report the isolation and characterization of the human hsf2 gene and its 5'-flanking region. The transcription unit of the human hsf2 gene consists of 13 exons dispersed over 33 kbp of genomic DNA on chromosome 6. The hsf2 mRNA is transcribed from multiple start sites, and initiation from the major site results in a transcript of 2.45 kb. A functional promoter, as determined by the ability to direct expression of a transiently transfected luciferase reporter gene, resides in a 950-bp upstream region of the human hsf2 gene. Examination of the core promoter sequence revealed a high GC content and lack of a canonical TATA box. This feature seems to be common among various species, as comparison of the hsf2 proximal promoter sequences from human, mouse, and rat showed distinct conserved regions. Moreover, the overall architecture of the human hsf2 gene is similar to its mouse counterpart. A comparison between human hsf2 gene and other hsf genes showed striking similarities in exon size. However, the exons are assembled in an hsf-specific manner.

Organic acids as excipients in matrix granules for colon-specific drug delivery.

Interest exists in developing site-specific systems for release of a drug in the lower part of the small intestine or in the colon. The aim of this study was to investigate whether drug release rates from enteric matrix granules could be influenced by using organic acids as excipients. Ibuprofen was used as a model drug and Eudragit S and Aqoat AS-HF as enteric polymers. The dissolution rates of the drug were investigated at different levels of pH (5.8, 6.8 and 7. 4). Drug absorption was studied in bioavailability tests in healthy volunteers. In vitro/in vivo correlation was also investigated. It was concluded that although inclusion of an organic acid in a formulation retarded in vitro release of the model drug, no corresponding effect was evident in in vivo studies. Bioavailability tests are therefore important early on during development of new dosage forms or formulations. Although no correlation between in vitro and in vivo results was generally evident correlation could be demonstrated for individual formulations following mathematical transformation of data.

Differentiation lineage-specific expression of human heat shock transcription factor 2.

Differentiation of multipotential hematopoietic cells into lineage-committed precursors involves the selection and maintenance of appropriate programs of gene expression, regulated by specific transcription factors. Using human K562 erythroleukemia cells capable of differentiating along erythroid and megakaryocytic lineages, we explore the differentiation-related role of heat shock transcription factor 2 (HSF2), which belongs to a family of transcription factors generally known to regulate heat shock gene expression. We demonstrate that enhanced HSF2 expression and the acquisition of HSF2 DNA binding activity are strictly specific for erythroid characteristics of K562 cells. Our results reveal a multistep regulatory process of HSF2 gene expression. In K562 cells undergoing hemin-mediated erythroid differentiation, the increase in HSF2 protein levels is preceded by transcriptional induction of the HSF2 gene, accompanied by increased HSF2 mRNA stability. In contrast, during megakaryocytic differentiation induced by the phorbol ester TPA, expression of HSF2 is rapidly down-regulated, leading to a complete loss of the HSF2 protein. These results indicate that the determination of HSF2 expression occurs at the early stages of lineage commitment. Taken together, our data suggest that HSF2 could function as a lineage-restricted transcription factor during differentiation of K562 cells along either the erythroid or the megakaryocytic pathway.