The need to strengthen the evaluation of the impact of Artificial Intelligence-based decision support systems on healthcare provision.

Despite the renewed interest in Artificial Intelligence-based clinical decision support systems (AI-CDS), there is still a lack of empirical evidence supporting their effectiveness. This underscores the need for rigorous and continuous evaluation and monitoring of processes and outcomes associated with the introduction of health information technology. We illustrate how the emergence of AI-CDS has helped to bring to the fore the critical importance of evaluation principles and action regarding all health information technology applications, as these hitherto have received limited attention. Key aspects include assessment of design, implementation and adoption contexts; ensuring systems support and optimise human performance (which in turn requires understanding clinical and system logics); and ensuring that design of systems prioritises ethics, equity, effectiveness, and outcomes. Going forward, information technology strategy, implementation and assessment need to actively incorporate these dimensions. International policy makers, regulators and strategic decision makers in implementing organisations therefore need to be cognisant of these aspects and incorporate them in decision-making and in prioritising investment. In particular, the emphasis needs to be on stronger and more evidence-based evaluation surrounding system limitations and risks as well as optimisation of outcomes, whilst ensuring learning and contextual review. Otherwise, there is a risk that applications will be sub-optimally embodied in health systems with unintended consequences and without yielding intended benefits.

Social Determinants of Mobile Health App Adoption - A Qualitative Study of Older Adults' Perceptions in Australia.

Mobile health applications (mHealth apps) can provide effective self-management instruments for patients and offer advanced approaches to treatment. However, little is known about how the older population perceives the opportunity of using mHealth apps as a non-drug intervention. We aimed to identify the opinions and experiences of older people in Australia and gain new insights into their engagement with this modern approach to health treatment. We conducted a qualitative study with 21 Participants to explore users' perspectives on adopting and using mHealth apps and their awareness of the social factors influencing their uptake. The results show that a trusting doctor-patient relationship positively affects older adults' perceptions of mHealth apps. Consequently, the social influence of the General Practitioner (GP) plays a crucial role in the use of mHealth apps, while the social influence through family and friends seems to be less critical.

The Role of Formative Evaluation in Promoting Digitally-based Health Equity and Reducing Bias for Resilient Health Systems: The Case of Patient Portals.

OBJECTIVES: Patient portals are increasingly implemented to improve patient involvement and engagement. We here seek to provide an overview of ways to mitigate existing concerns that these technologies increase inequity and bias and do not reach those who could benefit most from them. METHODS: Based on the current literature, we review the limitations of existing evaluations of patient portals in relation to addressing health equity, literacy and bias; outline challenges evaluators face when conducting such evaluations; and suggest methodological approaches that may address existing shortcomings. RESULTS: Various stakeholder needs should be addressed before deploying patient portals, involving vulnerable groups in user-centred design, and studying unanticipated consequences and impacts of information systems in use over time. CONCLUSIONS: Formative approaches to evaluation can help to address existing shortcomings and facilitate the development and implementation of patient portals in an equitable way thereby promoting the creation of resilient health systems.

Telehealth utilisation in residential aged care facilities during the COVID-19 pandemic: A retrospective cohort study in Australian general practice.

INTRODUCTION: Our earlier analysis during the COVID-19 surges in 2020 showed a reduction in general practitioner (GP) in-person visits to residential aged care facilities (RACFs) and increased use of telehealth. This study assessed how sociodemographic characteristics affected telehealth utilisation. METHODS: This retrospective cohort consists of 27,980 RACF residents aged 65 years and over, identified from general practice electronic health records in Victoria and New South Wales during March 2020-August 2021. Residents' demographic characteristics, including age, sex, region, and pension status, were analysed to estimate the odds ratio (OR) and 95% confidence interval (CI) for the associations with telehealth utilisation (telephone/video vs. in-person consultations) and with video versus telephone consultations, in mixed-effects multiple level regression models. RESULTS: Of 32,330 median monthly GP consultations among 21,987 residents identified in 2020, telehealth visits accounted for 17% of GP consultations, of which 93% were telephone consults. In 2021, of 32,229 median monthly GP consultations among 22,712 residents, telehealth visits accounted for 11% of GP consultations (97% by telephone). Pension holders (OR: 1.14; 95% CI: 1.10, 1.17) and those residing in rural areas (OR: 1.72; 95% CI: 1.57, 1.90) were more likely to use telehealth. However, residents in rural areas were less likely to use video than telephone in GP consultations (OR: 0.41; 95% CI: 0.29, 0.57). Results were similar in separate analyses for each COVID surge. DISCUSSION: Telephone was primarily used in telehealth consultations among pension holders and rural residents in RACFs. Along with the limited use of video in virtual care in rural RACFs, the digital divide may imply potential healthcare disparities in socially disadvantaged patients.

Incidence of adverse incidents in residential aged care.

Objective Adverse incident research within residential aged care facilities (RACFs) is increasing and there is growing awareness of safety and quality issues. However, large-scale evidence identifying specific areas of need and at-risk residents is lacking. This study used routinely collected incident management system data to quantify the types and rates of adverse incidents experienced by residents of RACFs. Methods A concurrent mixed-methods design was used to examine 3 years of incident management report data from 72 RACFs in New South Wales and the Australian Capital Territory. Qualitative thematic analysis of free-text incident descriptions was undertaken to group adverse incidents into categories. The rates and types of adverse incidents based on these categories were calculated and then compared using incidence rate ratios (IRRs). Results Deidentified records of 11 987 permanent residents (aged ≥65 years; mean (±s.d.) age 84 ± 8 years) from the facilities were included. Of the 60 268 adverse incidents, falls were the most common event (36%), followed by behaviour-related events (33%), other impacts and injuries (22%) and medication errors (9%). The number of adverse incidents per resident ranged from 0 (42%) to 171, with a median of 2. Women (IRR 0.804; P P Conclusion This study demonstrates that data already collected within electronic management systems can provide crucial baseline information about the risk levels that adverse incidents pose to older Australians living in RACFs. What is known about the topic? To date, research into aged care adverse incidents has typically focused on single incident types in small studies involving mitigation strategies. Little has been published quantifying the multiple adverse incidents experienced by residents of aged care facilities or reporting organisation-wide rates of adverse incidents. What does this paper add? This paper adds to the growing breadth of Australian aged care research by providing baseline information on the rates and types of adverse incidents in RACFs across a large and representative provider. What are the implications for practitioners? This research demonstrates that the wealth of data captured by aged care facilities' incident management information systems can be used to provide insight into areas of commonly occurring adverse incidents. Better use of this information could greatly enhance strategic planning of quality improvement activities and the care provided to residents.

Diagnostic Informatics-The Role of Digital Health in Diagnostic Stewardship and the Achievement of Excellence, Safety, and Value.

Diagnostic investigations (pathology laboratory and medical imaging) aim to: increase certainty of the presence or absence of disease by supporting the process of differential diagnosis; support clinical management; and monitor a patient's trajectory (e. g., disease progression or response to treatment). Digital health can be defined as the collection, storage, retrieval, transmission, and utilization of data, information, and knowledge to support healthcare. Digital health has become an essential component of the diagnostic process, helping to facilitate the accuracy and timeliness of information transfer and enhance the effectiveness of decision-making processes. Digital health is also important to diagnostic stewardship, which involves coordinated guidance and interventions to ensure the appropriate utilization of diagnostic tests for therapeutic decision-making. Diagnostic stewardship and informatics are thus important in efforts to establish shared decision-making. This is because they contribute to the establishment of shared information platforms (enabling patients to read, comment on, and share in decisions about their care) based on timely and meaningful communication. This paper will outline key diagnostic informatics and stewardship initiatives across three interrelated fields: (1) diagnostic error and the establishment of outcomes-based diagnostic research; (2) the safety and effectiveness of test result management and follow-up; and (3) digitally enhanced decision support systems.

Co-designing a dashboard of predictive analytics and decision support to drive care quality and client outcomes in aged care: a mixed-method study protocol.

INTRODUCTION: There is a clear need for improved care quality and quality monitoring in aged care. Aged care providers collect an abundance of data, yet rarely are these data integrated and transformed in real-time into actionable information to support evidence-based care, nor are they shared with older people and informal caregivers. This protocol describes the co-design and testing of a dashboard in residential aged care facilities (nursing or care homes) and community-based aged care settings (formal care provided at home or in the community). The dashboard will comprise integrated data to provide an 'at-a-glance' overview of aged care clients, indicators to identify clients at risk of fall-related hospitalisations and poor quality of life, and evidence-based decision support to minimise these risks. Longer term plans for dashboard implementation and evaluation are also outlined. METHODS: This mixed-method study will involve (1) co-designing dashboard features with aged care staff, clients, informal caregivers and general practitioners (GPs), (2) integrating aged care data silos and developing risk models, and (3) testing dashboard prototypes with users. The dashboard features will be informed by direct observations of routine work, interviews, focus groups and co-design groups with users, and a community forum. Multivariable discrete time survival models will be used to develop risk indicators, using predictors from linked historical aged care and hospital data. Dashboard prototype testing will comprise interviews, focus groups and walk-through scenarios using a think-aloud approach with staff members, clients and informal caregivers, and a GP workshop. ETHICS AND DISSEMINATION: This study has received ethical approval from the New South Wales (NSW) Population & Health Services Research Ethics Committee and Macquarie University's Human Research Ethics Committee. The research findings will be presented to the aged care provider who will share results with staff members, clients, residents and informal caregivers. Findings will be disseminated as peer-reviewed journal articles, policy briefs and conference presentations.

Managing Pandemic Responses with Health Informatics - Challenges for Assessing Digital Health Technologies.

OBJECTIVES: To highlight the role of technology assessment in the management of the COVID-19 pandemic. METHOD: An overview of existing research and evaluation approaches along with expert perspectives drawn from the International Medical Informatics Association (IMIA) Working Group on Technology Assessment and Quality Development in Health Informatics and the European Federation for Medical Informatics (EFMI) Working Group for Assessment of Health Information Systems. RESULTS: Evaluation of digital health technologies for COVID-19 should be based on their technical maturity as well as the scale of implementation. For mature technologies like telehealth whose efficacy has been previously demonstrated, pragmatic, rapid evaluation using the complex systems paradigm which accounts for multiple sociotechnical factors, might be more suitable to examine their effectiveness and emerging safety concerns in new settings. New technologies, particularly those intended for use on a large scale such as digital contract tracing, will require assessment of their usability as well as performance prior to deployment, after which evaluation should shift to using a complex systems paradigm to examine the value of information provided. The success of a digital health technology is dependent on the value of information it provides relative to the sociotechnical context of the setting where it is implemented. CONCLUSION: Commitment to evaluation using the evidence-based medicine and complex systems paradigms will be critical to ensuring safe and effective use of digital health technologies for COVID-19 and future pandemics. There is an inherent tension between evaluation and the imperative to urgently deploy solutions that needs to be negotiated.

Clinical thresholds for diagnosing iron deficiency: comparison of functional assessment of serum ferritin to population based centiles.

Low serum ferritin is diagnostic of iron deficiency, yet its published lower cut-off values are highly variable, particularly for pediatric populations. Lower cut-off values are commonly reported as 2.5th percentiles, and is based on the variation of ferritin values in the population. Our objective was to determine whether a functional approach based on iron deficient erythropoiesis could provide a better alternative. Utilizing 64,443 ferritin test results from pediatric electronic health records, we conducted various statistical techniques to derive 2.5th percentiles, and also derived functional reference limits through the association between ferritin and erythrocyte parameters: hemoglobin, mean corpuscular volume, mean cell hemoglobin concentration, and red cell distribution width. We find that lower limits of reference intervals derived as centiles are too low for clinical interpretation. Functional limits indicate iron deficiency anemia starts to occur when ferritin levels reach 10 µg/L, and are largely similar between genders and age groups. In comparison, centiles (2.5%) presented with lower limits overall, with varying levels depending on age and gender. Functionally-derived limits better reflects the underlying physiology of a patient, and may provide a basis for deriving a threshold related to treatment of iron deficiency and any other biomarker with functional outcomes.

The co-design of timely and meaningful information needed to enhance social participation in community aged care services: Think tank proceedings.

OBJECTIVE: To determine what information from community aged care social participation and quality of life assessments needs to be captured, and meaningfully utilised as part of an integrated information and communication technology system. METHODS: Two think tank sessions comprised of community aged care staff and researchers (n = 9) were conducted over 5 weeks. The sessions were guided by the Continuous Quality Improvement framework. Thematic analysis was used to categorise the think tank data. RESULTS: To monitor progress over time, participants needed more contextual information captured in the assessment forms, such as client goals and outcomes of assessments. The aged care provider agreed to embed outcome measure score and action following assessment into its information and communication technology system. CONCLUSION: Collaboration between aged care staff and researchers resulted in adjustments to the aged care provider's information and communication technology system to better target the monitoring and planning of its clients' psychosocial needs.

Diagnostic Informatics: Its Role in Enhancing Clinical Excellence, Patient Safety and the Value of Care.

Diagnostic informatics encompasses the role of information technology in key areas of the diagnostic testing (pathology and medical imaging) process, including the selection of appropriate tests and interpretation and follow-up of test results. We present three case studies employing diagnostic informatics methodologies to demonstrate their potential use and value in health services research: (1) Data analytics applied to diagnostic data linked with patient outcome data as a means of enhanancing the monitoring of the quality and appropriateness of diagnostic test choices; (2) Business process modelling which can help to highlight healthcare processes in the diagnostic pathway as a means of improving safety and performance, and (3) Consumer involvement in the diagnostic research process to assist in the establishment of person-centred test result management systems. The case studies provide evidence of the role that diagnostic informatics can have in improving the quality and safety of patient care.

The use of physiotherapy in nursing homes internationally: A systematic review.

BACKGROUND: Physiotherapy can improve functional ability, prevent falls and reduce pain for older adults in nursing homes. However, there are no legislations or guidelines that specify the parameters of physiotherapy required in nursing homes. With the increasing healthcare demands of ageing populations worldwide, it is important to understand the current use of physiotherapy services to ensure they are both evidence-based and promote equity. OBJECTIVES: (1) When and how are physiotherapy services used by older adults living in nursing homes? (2) What are the factors associated with use of physiotherapy services in nursing homes? (3) How are physiotherapy services in nursing homes documented and monitored? METHODS: Several databases and grey literature (including MEDLINE, PubMed, Pedro and EMBASE) were searched following PRISMA guidelines in March 2018. Searches were limited to English language publications from 1997. Assessment for inclusion, data extraction and quality assessment were completed by two investigators independently using standardised forms. Studies were included if they considered any type of physiotherapy service that involved a qualified physiotherapist (such as exercise, massage and staff education) with older adults (aged 60 years and older) that were primarily permanent residents of a nursing home. Data extracted included proportion of clients that used physiotherapy services, type, frequency and duration of physiotherapy services, and factors associated with physiotherapy service use. RESULTS: Eleven studies were included. Between 10% and 67% of nursing home clients used physiotherapy services. Factors associated with greater use of physiotherapy services included larger size facilities, and if clients had a physical impairment and mild or no cognitive impairment. Types of physiotherapy services reported were pain management and pressure ulcer management. CONCLUSIONS: Physiotherapy service use in nursing homes varied widely. The development of physiotherapy benchmarks and quality standards are needed to support older adults in nursing homes. PROSPERO registration number: CRD42018082460.

Artificial Intelligence in Clinical Decision Support: Challenges for Evaluating AI and Practical Implications.

OBJECTIVES: This paper draws attention to: i) key considerations for evaluating artificial intelligence (AI) enabled clinical decision support; and ii) challenges and practical implications of AI design, development, selection, use, and ongoing surveillance. METHOD: A narrative review of existing research and evaluation approaches along with expert perspectives drawn from the International Medical Informatics Association (IMIA) Working Group on Technology Assessment and Quality Development in Health Informatics and the European Federation for Medical Informatics (EFMI) Working Group for Assessment of Health Information Systems. RESULTS: There is a rich history and tradition of evaluating AI in healthcare. While evaluators can learn from past efforts, and build on best practice evaluation frameworks and methodologies, questions remain about how to evaluate the safety and effectiveness of AI that dynamically harness vast amounts of genomic, biomarker, phenotype, electronic record, and care delivery data from across health systems. This paper first provides a historical perspective about the evaluation of AI in healthcare. It then examines key challenges of evaluating AI-enabled clinical decision support during design, development, selection, use, and ongoing surveillance. Practical aspects of evaluating AI in healthcare, including approaches to evaluation and indicators to monitor AI are also discussed. CONCLUSION: Commitment to rigorous initial and ongoing evaluation will be critical to ensuring the safe and effective integration of AI in complex sociotechnical settings. Specific enhancements that are required for the new generation of AI-enabled clinical decision support will emerge through practical application.

Ageing well: evaluation of social participation and quality of life tools to enhance community aged care (study protocol).

BACKGROUND: Several outcome measures can be utilised to measure social participation and Quality of Life (QoL) in research and clinical practice. However there have been few large-scale trials of these tools in community care to identify their value to clients and providers. This study aims to evaluate the implementation of the Australian Community Participation Questionnaire (ACPQ) and the ICEpop CAPability measure for Older people (ICECAP-O) as tools to measure social participation and QoL for clients receiving community aged care services. The specific research questions focus on determining: (1) the levels and predictors of social participation and QoL among older adults using community aged care services; (2) the acceptability and feasibility of implementation of ACPQ and ICECAP-O tools into routine community aged care assessments; (3) if implementation of the tools change service provision and outcomes for older adults receiving community aged care services. METHODS: A mixed method design will be used to collect data from a large Australian aged care provider. Community aged care clients' ACPQ and ICECAP-O scores, as well as other key outcomes (e.g. services used, hospitalisation and admission to permanent residential care), will be examined at baseline and 12-monthly follow-up assessments. Interviews and focus groups with community aged care clients and staff who administer the tools will also be completed. Descriptive statistics and multiple linear regression will be used to examine the levels and predictors of social participation and QoL. Thematic analysis of interviews and focus groups will be used to determine the acceptability and feasibility of implementing the ACPQ and ICECAP-O into routine needs assessments in community aged care. Case-controlled analyses will be used to determine whether the implementation of the ACPQ and ICECAP-O changes service use and outcomes. DISCUSSION: The novel use of the ACPQ and the ICECAP-O tools as part of routine needs assessments for community aged care clients has the potential to improve the quality and effectiveness of community aged care services and outcomes. TRIAL REGISTRATION: Australian and New Zealand clinical trial registry number: ACTRN12617001212347 . Registered 18/08/2017.

Health Information Infrastructure for People with Intellectual and Developmental Disabilities (I/DD) Living in Supported Accommodation: Communication, Co-Ordination and Integration of Health Information.

People with intellectual and/or developmental disability (I/DD) commonly have complex health care needs, but little is known about how their health information is managed in supported accommodation, and across health services providers. This study aimed to describe the current health information infrastructure (i.e., how data and information are collected, stored, communicated, and used) for people with I/DD living in supported accommodation in Australia. It involved a scoping review and synthesis of research, policies, and health documents relevant in this setting. Iterative database and hand searches were conducted across peer-reviewed articles internationally in English and grey literature in Australia (New South Wales) up to September 2015. Data were extracted from the selected relevant literature and analyzed for content themes. Expert stakeholders were consulted to verify the authors' interpretations of the information and content categories. The included 286 sources (peer-reviewed n = 27; grey literature n = 259) reflect that the health information for people with I/DD in supported accommodation is poorly communicated, coordinated and integrated across isolated systems. 'Work-as-imagined' as outlined in policies, does not align with 'work-as-done' in reality. This gap threatens the quality of care and safety of people with I/DD in these settings. The effectiveness of the health information infrastructure and services for people with I/DD can be improved by integrating the information sources and placing people with I/DD and their supporters at the centre of the information exchange process.

The Safe and Effective Use of Shared Data Underpinned by Stakeholder Engagement and Evaluation Practice.

OBJECTIVES: The paper draws attention to: i) key considerations involving the confidentiality, privacy, and security of shared data; and ii) the requirements needed to build collaborative arrangements encompassing all stakeholders with the goal of ensuring safe, secure, and quality use of shared data. METHOD: A narrative review of existing research and policy approaches along with expert perspectives drawn from the International Medical Informatics Association (IMIA) Working Group on Technology Assessment and Quality Development in Health Care and the European Federation for Medical Informatics (EFMI) Working Group for Assessment of Health Information Systems. RESULTS: The technological ability to merge, link, re-use, and exchange data has outpaced the establishment of policies, procedures, and processes to monitor the ethics and legality of shared use of data. Questions remain about how to guarantee the security of shared data, and how to establish and maintain public trust across large-scale shared data enterprises. This paper identifies the importance of data governance frameworks (incorporating engagement with all stakeholders) to underpin the management of the ethics and legality of shared data use. The paper also provides some key considerations for the establishment of national approaches and measures to monitor compliance with best practice. CONCLUSION: Data sharing endeavours can help to underpin new collaborative models of health care which provide shared information, engagement, and accountability amongst all stakeholders. We believe that commitment to rigorous evaluation and stakeholder engagement will be critical to delivering health data benefits and the establishment of collaborative models of health care into the future.

Delivering safe and effective test-result communication, management and follow-up: a mixed-methods study protocol.

INTRODUCTION: The failure to follow-up pathology and medical imaging test results poses patient-safety risks which threaten the effectiveness, quality and safety of patient care. The objective of this project is to: (1) improve the effectiveness and safety of test-result management through the establishment of clear governance processes of communication, responsibility and accountability; (2) harness health information technology (IT) to inform and monitor test-result management; (3) enhance the contribution of consumers to the establishment of safe and effective test-result management systems. METHODS AND ANALYSIS: This convergent mixed-methods project triangulates three multistage studies at seven adult hospitals and one paediatric hospital in Australia.Study 1 adopts qualitative research approaches including semistructured interviews, focus groups and ethnographic observations to gain a better understanding of test-result communication and management practices in hospitals, and to identify patient-safety risks which require quality-improvement interventions.Study 2 analyses linked sets of routinely collected healthcare data to examine critical test-result thresholds and test-result notification processes. A controlled before-and-after study across three emergency departments will measure the impact of interventions (including the use of IT) developed to improve the safety and quality of test-result communication and management processes.Study 3 adopts a consumer-driven approach, including semistructured interviews, and the convening of consumer-reference groups and community forums. The qualitative data will identify mechanisms to enhance the role of consumers in test-management governance processes, and inform the direction of the research and the interpretation of findings. ETHICS AND DISSEMINATION: Ethical approval has been granted by the South Eastern Sydney Local Health District Human Research Ethics Committee and Macquarie University. Findings will be disseminated in academic, industry and consumer journals, newsletters and conferences.

Reducing Unnecessary Imaging and Pathology Tests: A Systematic Review.

CONTEXT: Unnecessary imaging and pathology procedures represent low-value care and can harm children and the health care system. OBJECTIVE: To perform a systematic review of interventions designed to reduce unnecessary pediatric imaging and pathology testing. DATA SOURCES: We searched Medline, Embase, Cinahl, PubMed, Cochrane Library, and gray literature. STUDY SELECTION: Studies we included were: reports of interventions to reduce unnecessary imaging and pathology testing in pediatric populations; from developed countries; written in the English language; and published between January 1, 1996, and April 29, 2017. DATA EXTRACTION: Two researchers independently extracted data and assessed study quality using a Cochrane group risk of bias tool. Level of evidence was graded using the Oxford Centre for Evidence-Based Medicine grading system. RESULTS: We found 64 articles including 44 before-after, 14 interrupted time series, and 1 randomized controlled trial. More effective interventions were (1) multifaceted, with 3 components (mean relative reduction = 45.0%; SD = 28.3%) as opposed to 2 components (32.0% [30.3%]); or 1 component (28.6%, [34.9%]); (2) targeted toward families and clinicians compared with clinicians only (61.9% [34.3%] vs 30.0% [32.0%], respectively); and (3) targeted toward imaging (41.8% [38.4%]) or pathology testing only (48.8% [20.9%]), compared with both simultaneously (21.6% [29.2%]). LIMITATIONS: The studies we included were limited to the English language. CONCLUSIONS: Promising interventions include audit and feedback, system-based changes, and education. Future researchers should move beyond before-after designs to rigorously evaluate interventions. A relatively novel approach will be to include both clinicians and the families they manage in such interventions.

The health literacy demands of electronic personal health records (e-PHRs): An integrative review to inform future inclusive research.

OBJECTIVE: To integrate the findings of research on electronic personal health records (e-PHRs) for an understanding of their health literacy demands on both patients and providers. METHODS: We sought peer-reviewed primary research in English addressing the health literacy demands of e-PHRs that are online and allow patients any degree of control or input to the record. A synthesis of three theoretical models was used to frame the analysis of 24 studies. RESULTS: e-PHRs pose a wide range of health literacy demands on both patients and health service providers. Patient participation in e-PHRs relies not only on their level of education and computer literacy, and attitudes to sharing health information, but also upon their executive function, verbal expression, and understanding of spoken and written language. CONCLUSION: The multiple health literacy demands of e-PHRs must be considered when implementing population-wide initiatives for storing and sharing health information using these systems. PRACTICE IMPLICATIONS: The health literacy demands of e-PHRs are high and could potentially exclude many patients unless strategies are adopted to support their use of these systems. Developing strategies for all patients to meet or reduce the high health literacy demands of e-PHRs will be important in population-wide implementation.

The E-health Literacy Demands of Australia's My Health Record: A Heuristic Evaluation of Usability.

Background: My Health Record is Australia's electronic personal health record system, which was introduced in July 2012. As of August 2017, approximately 21 percent of Australia's total population was registered to use My Health Record. Internationally, usability issues have been shown to negatively influence the uptake and use of electronic health record systems, and this scenario may particularly affect people who have low e-health literacy. It is likely that usability issues are negatively affecting the uptake and use of My Health Record in Australia. Objective: To identify potential e-health literacy-related usability issues within My Health Record through a heuristic evaluation method. Methods: Between September 14 and October 12, 2016, three of the authors conducted a heuristic evaluation of the two consumer-facing components of My Health Record-the information website and the electronic health record itself. These two components were evaluated against two sets of heuristics-the Health Literacy Online checklist and the Monkman Heuristics. The Health Literacy Online checklist and Monkman Heuristics are evidence-based checklists of web design elements with a focus on design for audiences with low health literacy. During this heuristic evaluation, the investigators individually navigated through the consumer-facing components of My Health Record, recording instances where the My Health Record did not conform to the checklist criteria. After the individual evaluations were completed, the investigators conferred and aggregated their results. From this process, a list of usability violations was constructed. Results: When evaluated against the Health Literacy Online Checklist, the information website demonstrated violations in 12 of 35 criteria, and the electronic health record demonstrated violations in 16 of 35 criteria. When evaluated against the Monkman Heuristics, the information website demonstrated violations in 7 of 11 criteria, and the electronic health record demonstrated violations in 9 of 11 criteria. The identified violations included usability issues with the reading levels used within My Health Record, the graphic design elements, the layout of web pages, and a lack of images and audiovisual tools to support learning. Other important usability issues included a lack of translated resources, difficulty using accessibility tools, and complexity of the registration processes. Conclusion: My Health Record is an important piece of technology that has the potential to facilitate better communication between consumers and their health providers. However, this heuristic evaluation demonstrated that many usability-related elements of My Health Record cater poorly to users at risk of having low e-health literacy. Usability issues have been identified as an important barrier to use of personal health records internationally, and the findings of this heuristic evaluation demonstrate that usability issues may be substantial barriers to the uptake and use of My Health Record.