Clinical Quality Measure Exchange is Not Easy.

PURPOSE: The Trial of Aggregate Data Exchange for Maintenance of Certification and Raising Quality was a randomized controlled trial which first had to test whether quality reporting could be a by-product of clinical care. We report on the initial descriptive study of the capacity for and quality of exchange of whole-panel, standardized quality measures from health systems. METHODS: Family physicians were recruited from 4 health systems with mature quality measurement programs and agreed to submit standardized, physician-level quality measures for consenting physicians. Identified measure or transfer errors were captured and evaluated for root-cause problems. RESULTS: The health systems varied considerably by patient demographics and payer mix. From the 4 systems, 256 family physicians elected to participate. Of 19 measures negotiated for use, 5 were used by all systems. There were more than 15 types of identified errors including breaks in data delivery, changes in measures, and nonsensical measure results. Only 1 system had no identified errors. CONCLUSIONS: The secure transfer of standardized, physician-level quality measures from 4 health systems with mature measure processes proved difficult. There were many errors that required human intervention and manual repair, precluding full automation. This study reconfirms an important problem, namely, that despite widespread health information technology adoption and federal meaningful use policies, we remain far from goals to make clinical quality reporting a reliable by-product of care.

Assessing the Implementation and Effectiveness of the Electronic Patient-Reported Outcome Tool for Older Adults With Complex Care Needs: Mixed Methods Study.

BACKGROUND: Goal-oriented care is being adopted to deliver person-centered primary care to older adults with multimorbidity and complex care needs. Although this model holds promise, its implementation remains a challenge. Digital health solutions may enable processes to improve adoption; however, they require evaluation to determine feasibility and impact. OBJECTIVE: This study aims to evaluate the implementation and effectiveness of the electronic Patient-Reported Outcome (ePRO) mobile app and portal system, designed to enable goal-oriented care delivery in interprofessional primary care practices. The research questions driving this study are as follows: Does ePRO improve quality of life and self-management in older adults with complex needs? What mechanisms are likely driving observed outcomes? METHODS: A multimethod, pragmatic randomized controlled trial using a stepped-wedge design and ethnographic case studies was conducted over a 15-month period in 6 comprehensive primary care practices across Ontario with a target enrollment of 176 patients. The 6 practices were randomized into either early (3-month control period; 12-month intervention) or late (6-month control period; 9-month intervention) groups. The primary outcome measure of interest was the Assessment of Quality of Life-4D (AQoL-4D). Data were collected at baseline and at 3 monthly intervals for the duration of the trial. Ethnographic data included observations and interviews with patients and providers at the midpoint and end of the intervention. Outcome data were analyzed using linear models conducted at the individual level, accounting for cluster effects at the practice level, and ethnographic data were analyzed using qualitative description and framework analysis methods. RESULTS: Recruitment challenges resulted in fewer sites and participants than expected; of the 176 target, only 142 (80.6%) patients were identified as eligible to participate because of lower-than-expected provider participation and fewer-than-expected patients willing to participate or perceived as ready to engage in goal-setting. Of the 142 patients approached, 45 (32%) participated. Patients set a variety of goals related to self-management, mental health, social health, and overall well-being. Owing to underpowering, the impact of ePRO on quality of life could not be definitively assessed; however, the intervention group, ePRO plus usual care (mean 15.28, SD 18.60) demonstrated a nonsignificant decrease in quality of life (t24=-1.20; P=.24) when compared with usual care only (mean 21.76, SD 2.17). The ethnographic data reveal a complex implementation process in which the meaningfulness (or coherence) of the technology to individuals' lives and work acted as a key driver of adoption and tool appraisal. CONCLUSIONS: This trial experienced many unexpected and significant implementation challenges related to recruitment and engagement. Future studies could be improved through better alignment of the research methods and intervention to the complex and diverse clinical settings, dynamic goal-oriented care process, and readiness of provider and patient participants. TRIAL REGISTRATION: ClinicalTrials.gov NCT02917954; https://clinicaltrials.gov/ct2/show/NCT02917954.

Randomized Clinical Trial of a Clinical Decision Support Tool for Improving the Appropriateness Scores for Ordering Imaging Studies in Primary and Specialty Care Ambulatory Clinics.

OBJECTIVE. The objective of our study was to evaluate whether the use of a clinical decision support (CDS) tool improved the appropriateness scores of orders for advanced imaging in clinical practice. MATERIALS AND METHODS. We used a stepped-wedge, cluster randomized clinical trial to evaluate the effectiveness of a CDS tool in an integrated health care system. Clinicians entered structured indications for each CT and MRI order, and the indications were electronically scored against appropriateness criteria to assign an appropriateness score. We compared the proportion of orders with adjusted appropriateness scores of 7 or greater (on a 1-9 scale) before and after activation of best practice alerts (BPAs) triggered for orders with low or marginal appropriateness scores. Secondary outcomes included the rate per month of orders for advanced imaging and the proportion of orders for which the radiology department requested changes. RESULTS. Between October 2015 and February 2016, 941 clinicians ordered 22,279 CT or MRI studies that met eligibility criteria. Before activation of the BPA, the mean proportion of appropriate orders (adjusted for time and clinic effect) was 77.0% (95% CI, 75.5-78.4%), which increased to 80.1% (95% CI, 78.7-81.5%) after activation (p = 0.001). There was no significant change in the rate of orders per month for advanced imaging. The proportion of order changes requested by the radiology department decreased from 5.7% (95% CI, 5.6-5.9%) before CDS implementation to 5.3% (95% CI, 5.1-5.5%) after CDS implementation (p < 0.001). CONCLUSION. Using an evidence-based CDS tool in clinical practice was associated with a modest but significant improvement in the appropriateness scores of advanced imaging orders.

Adjuvant gemcitabine-docetaxel chemotherapy for stage I uterine leiomyosarcoma: Trends and survival outcomes.

OBJECTIVE: To assess recent trends of administering adjuvant gemcitabine-docetaxel (GD) chemotherapy for Stage I uterine leiomyosarcoma, and to compare disease-free and overall survival between women who received and did not receive adjuvant GD chemotherapy. METHODS: All patients diagnosed with Stage I uterine leiomyosarcoma in a California-Colorado population-based health plan inclusive of 2006-2013 were included in a retrospective cohort. Adjuvant GD chemotherapy rates, clinico-pathologic characteristics and survival estimates were assessed. RESULTS: Of 111 women with Stage I uterine leiomyosarcoma, 33 received adjuvant GD (median 4cycles), 77 received no chemotherapy, and 1 patient excluded for non-GD chemotherapy. GD-chemotherapy and no-chemotherapy groups were similar with respect to age, stage (IA/IB), uterine weight, mitotic index, body mass index, and Charlson comorbidity score. Non-Hispanic white women were twice as likely to receive adjuvant chemotherapy as non-white or Hispanic women (37.7 vs. 17.1%, P=0.02). The proportion of women receiving adjuvant GD chemotherapy increased from 6.5% in 2006-2008 to 46.9% in 2009-2013 (P<0.001). There was no significance difference in unadjusted Kaplan-Meyer estimated disease-free (P=0.95) or overall survival (P=0.43) between GD-chemotherapy and no-chemotherapy cohorts. Corresponding adjusted Cox proportional hazard ratios for adjuvant GD chemotherapy compared to no chemotherapy were 1.01 (95% confidence interval [CI] 0.57-1.80, P=0.97) for recurrence and 1.28 (95% CI 0.69-2.36, P-0.48) for mortality. CONCLUSIONS: Use of adjuvant GD chemotherapy for Stage I uterine leiomyosarcoma has increased significantly in the last decade, despite unclear benefit. Compared to no chemotherapy, 4-6cycles of adjuvant GD chemotherapy does not appear to alter survival outcomes.

eHealth Advances in Support of People with Complex Care Needs: Case Examples from Canada, Scotland and the US.

Information technology (IT) in healthcare, also referred to as eHealth technologies, may offer a promising solution to the provision of better care and support for people who have multiple conditions and complex care needs, and their caregivers. eHealth technologies can include electronic medical records, telemonitoring systems and web-based portals, and mobile health (mHealth) technologies that enable information sharing between providers, patients, clients and their families. IT often acts as an enabler of improved care delivery, rather than being an intervention per se. But how are different countries seeking to leverage adoption of these technologies to support people who have chronic conditions and complex care needs? This article presents three case examples from Ontario (Canada), Scotland and Kaiser Permanente Colorado (United States) to identify how these jurisdictions are currently using technology to address multimorbidity. A SWOT (strengths, weaknesses, opportunities, threats) analysis is presented for each case and a final discussion addresses the future of eHealth for complex care needs. The case reports presented in this manuscript mark the foundational work of the Multi-National eHealth Research Partnership Supporting Complex Chronic Disease and Disability (the eCCDD Network); a CIHR-funded project intended to support the international development and uptake of eHealth tools for people with complex care needs.

Applying Implementation Science Principles to Systematize High-Quality Care for Potentially Significant Imaging Findings.

OBJECTIVE: Use principles of implementation science to improve the diagnosis and management of potentially significant imaging findings. METHODS: Multidisciplinary stakeholders codified the diagnosis and management of potentially significant imaging findings in eight organs and created a finding tracking management system that was embedded in radiologist workflows and IT systems. Radiologists were trained to use this system. An automated finding tracking management system was created to support consistent high-quality care through care pathway visualizations, increased awareness of specific findings in the electronic medical record, templated notifications, and creation of an electronic safety net. Primary outcome was the rate of quality reviews related to eight targeted imaging findings. Secondary outcome was radiologist use of the finding tracking management tool. RESULTS: In the 4 years after implementation, the tool was used to track findings in 7,843 patients who received 10,015 ultrasound, CT, MRI, x-ray, and nuclear medicine examinations that were interpreted by all 34 radiologists. Use of the tool lead to a decrease in related quality reviews (from 8.0% to 0.0%, P < .007). Use of the system increased from 1.7% of examinations in the early implementation phase to 3.1% (+82%, P < .00001) in the postimplementation phase. Each radiologist used the tool on an average of 294.6 unique examinations (SD 404.8). Overall, radiologists currently use the tool approximately 4,000 times per year. DISCUSSION: Radiologists frequently used a finding tracking management system to ensure effective communication and raise awareness of the importance of recommended future follow-up studies. Use of this system was associated with a decrease in the rate of quality review requests in this domain.

Comparing virtual consults to traditional consults using an electronic health record: an observational case-control study.

BACKGROUND: Patients have typically received health care through face-to-face encounters. However, expansion of electronic communication and electronic health records (EHRs) provide alternative means for patient and physicians to interact. Electronic consultations may complement regular healthcare by providing "better, faster, cheaper" processes for diagnosing, treating, and monitoring health conditions. Virtual consultation between physicians may provide a method of streamlining care, potentially saving patients the time and expense of added visits. The purpose of this study was to compare physician usage and patient satisfaction with virtual consultations (VCs) with traditional consultations (TCs) facilitated within an EHR. METHODS: We conducted an observational case-control survey study within Kaiser Permanente, Colorado. A sample of patients who had VCs requested by physicians (N = 270) were matched with patients who had TCs requested by physicians (N = 270), by patient age, gender, reason for the consult, and specialty department. These patients (VC and TC), were invited to participate in a satisfaction survey. In addition, 205 primary care physicians who submitted a VC or TC were surveyed. RESULTS: During the study period, 58,146 VC or TC were requested (TC = 96.3%). Patients who completed a satisfaction survey (267 out of 540 patients, 49.4% response rate) indicated they were satisfied with their care, irrespective of the kind of consult (mean 10-point Likert score of 8.5). 88 of 205 primary care physicians surveyed (42.9%) returned at least one survey; VC and TC survey response rates and consulted departments were comparable (p = 0.13). More TCs than VCs requested transfer of patient care (p = 0.03), assistance with diagnosis (p = 0.04) or initiating treatment (p =0.04). Within 3 weeks of the consultation request, 72.1% of respondents reported receiving information from VCs, compared with 33.9% of the TCs (p < 0.001). Utility of information provided by consultants and satisfaction with consultations did not differ between VCs and TCs. CONCLUSIONS: Referring physicians received information from consultants more quickly from VCs compared with TCs, but the value and application of information from both types of consultations were similar. VCs may decrease the need for face-to-face specialty encounters without a decrease in the patient's perception of care.

Association of online patient access to clinicians and medical records with use of clinical services.

CONTEXT: Prior studies suggest that providing patients with online access to health records and e-mail communication with physicians may substitute for traditional health care services. OBJECTIVE: To assess health care utilization by both users and nonusers of online access to health records before and after initiation of MyHealthManager (MHM), a patient online access system. DESIGN, SETTING, AND PARTICIPANTS: Retrospective cohort study of the use of health care services by members (≥18 years old) who were continuously enrolled for at least 24 months during the study period March 2005 through June 2010 in Kaiser Permanente Colorado, a group model, integrated health care delivery system. Propensity scores (using age, sex, utilization frequencies, and chronic illnesses) were used for cohort matching. Unadjusted utilization rates were calculated for both MHM users and nonusers and were the basis for difference-of-differences analyses. We also used generalized estimating equations to compare the adjusted rates of utilization of health care services before and after online access. MAIN OUTCOME MEASURES: Rates of office visits, telephone encounters, after-hours clinic visits, emergency department encounters, and hospitalizations between members with and without online access. RESULTS: Comparing the unadjusted rates for use of clinical services before and after the index date between the matched cohorts, there was a significant increase in the per-member rates of office visits (0.7 per member per year; 95% CI, 0.6-0.7; P < .001) and telephone encounters (0.3 per member per year; 95% CI, 0.2-0.3; P < .001). There was also a significant increase in per-1000-member rates of after-hours clinic visits (18.7 per 1000 members per year; 95% CI, 12.8-24.3; P < .001), emergency department encounters (11.2 per 1000 members per year; 95% CI, 2.6-19.7; P = .01), and hospitalizations (19.9 per 1000 members per year; 95% CI, 14.6-25.3; P < .001) for MHM users vs nonusers. CONCLUSION: Having online access to medical records and clinicians was associated with increased use of clinical services compared with group members who did not have online access.

Improving Quality and Safety for Patients After Hospital Discharge: Primary Care as the Lead Integrator in Postdischarge Care Transitions.

To understand current and idealized primary care-based care transition workflow processes for hospitalized patients, we conducted 133 interviews with staff at 9 primary care sites; community agency staff (n = 18); recently discharged patients (n = 33); and primary care thought leaders (n = 9). Current postdischarge workflows in primary care vary widely across settings, are often implemented inconsistently, and rarely involve communications with the patient or inpatient team during hospitalization. Based on these findings, we propose 5 principles for primary care practices to facilitate active involvement in postdischarge care, beginning during the hospital admission and extending until after the initial postdischarge primary care visit.

Characteristics of Patients and Proxy Caregivers Using Patient Portals in the Setting of Serious Illness and End of Life.

Background: There are few studies examining the usage and utility of patient portals among seriously ill and end-of-life populations and their caregivers. Objective: The aim of this study was to describe portal user characteristics among patients and their caregivers (proxy login) at two time points: (1) the 12 months following an electronic medical record flag for serious illness and (2) during the last 12 months of life. Methods: A retrospective cohort analysis of Kaiser Permanente Colorado (KPCO) patients with serious illness, as defined by Kaiser Permanente's prognostic algorithm, and their proxy caregivers was performed for the two time periods. Use was characterized as (1) the discrete number of days the portal was used and (2) the number of days that portal features were accessed. Differences in use by user characteristics were assessed. Results: Patients flagged for serious illness (N = 6129) were 70.4 ± 14.2 years of age, and used the portal on average 50.4 days. Patients (N = 6517) in the last year of life were 76.7 ± 13.7 years of age and used the portal on average 43 days. Caregiver proxy use of the portal was low in both cohorts. Patients who were older, female, non-White, and healthier were less likely to use the portal. Conclusions: In comparison with overall KPCO portal use and recent patient portal studies examining use patterns, patient portal use was high among patients flagged with serious illness and nearing the end of life. However, because use was associated with age, gender, and race, addressing barriers to portal adoption among underserved populations and caregiver proxies is key to better leveraging patient portal systems for palliative and end-of-life care.

Exploring the relationship between the usability of a goal-oriented mobile health application and non-usage attrition in patients with multimorbidity: A blended data analysis approach.

BACKGROUND: Mobile health applications are increasingly used to support the delivery of health care services to a variety of patients. Based on data obtained from a pragmatic trial of the electronic Patient Reported Outcome (ePRO) app designed to support goal-oriented care primary care, this study aims to (1) examine how patient-reported usability changed over the one-year intervention period, and (2) explore participant attrition rate of the electronic Patient Reported Outcome app over one year study period. METHODS: We performed a secondary analysis of 44 older adults with complex chronic needs enrolled in the electronic Patient Reported Outcome-digital health intervention. App usage and attrition were measured using device-generated usage logs; usability was measured using the patient-reported post-study system usability questionnaire collected at 3, 6, 9, and 12 months. Research memos were used to interpret potential contextual contributing factors to patients' overall usage and usability score pattern. A data triangulation method of both quantitative and qualitative data was used to analyze and interpret study findings. RESULTS: While there was gradual attrition in the use of the ePRO app, patients' usability scores remained consistent throughout the study period. Qualitative memos suggested patients' encounters with technical difficulties and relationship dynamics with primary providers influenced patients' adherence to the ePRO app. CONCLUSION: This study highlights that the patient-provider relationship is a key determining factor that influences complex patients' continued engagement with a Mobile health app. The finding calls attention to the measurement of usability of a Mobile health app, its impact on attrition, and contributing factors that influence patients' attrition. Trial registration: Clinicaltrials.gov Identified NCT02917954.

Evaluation of laboratory monitoring alerts within a computerized physician order entry system for medication orders.

BACKGROUND: Errors involving medication use are common. Computerized physician order entry (CPOE) can improve prescribing practices. Few studies have examined the effect of CPOE in combination with decision support tools on prescribing practices in the outpatient setting. Less is known about prescribers' adherence to laboratory monitoring recommendations. OBJECTIVE: To evaluate if reminders presented during CPOE for medications would increase physicians' compliance with guidelines for laboratory monitoring at initiation of therapy. STUDY DESIGN: Randomized prospective intervention study. METHODS: Two hundred seven primary care physicians in a group-model managed care organization were randomized to receive or not receive drug laboratory monitoring alerts within the CPOE system. Adherence to laboratory monitoring recommendations for patients prescribed selected medications was compared between physician groups. RESULTS: There was no significant difference between the control and intervention group physicians in the overall rate of compliance with ordering the recommended laboratory monitoring for patients prescribed study medications. Laboratory monitoring was performed as recommended 56.6% of the time in the intervention group compared with 57.1% of the time in the control group (P = .31). In cases in which a statistically significant difference was demonstrated, improved compliance favored the intervention group (eg, 71.2% vs 62.3% [P = .003] for gemfibrozil and 75.7% vs 73.9% [P = .05] for statins). CONCLUSIONS: As CPOE becomes more prevalent, additional research is needed to determine effective decision support tools. These findings then should be communicated to the developers and users of computerized medical record systems.

Supporting Goal-Oriented Primary Health Care for Seniors with Complex Care Needs Using Mobile Technology: Evaluation and Implementation of the Health System Performance Research Network, Bridgepoint Electronic Patient Reported Outcome Tool.

BACKGROUND: Older adults experiencing multiple chronic illnesses are at high risk of hospitalization and health decline if they are unable to manage the significant challenges posed by their health conditions. Goal-oriented care approaches can provide better care for these complex patients, but clinicians find the process of ascertaining goals "too complex and too-time consuming," and goals are often not agreed upon between complex patients and their providers. The electronic patient reported outcomes (ePRO) mobile app and portal offers an innovative approach to creating and monitoring goal-oriented patient-care plans to improve patient self-management and shared decision-making between patients and health care providers. The ePRO tool also supports proactive patient monitoring by the patient, caregiver(s), and health care provider. It was developed with and for older adults with complex care needs as a means to improve their quality of life. OBJECTIVE: Our proposed project will evaluate the use, effectiveness, and value for money of the ePRO tool in a 12-month multicenter, randomized controlled trial in Ontario; targeting individuals 65 or over with two or more chronic conditions that require frequent health care visits to manage their health conditions. METHODS: Intervention groups using the ePRO tool will be compared with control groups on measures of quality of life, patient experience, and cost-effectiveness. We will also evaluate the implementation of the tool. RESULTS: The proposed project presented in this paper will be funded through the Canadian Institute for Health Research (CIHR) eHealth Innovation Partnerships Program (eHIPP) program (CIHR-348362). The expected completion date of the study is November, 2019. CONCLUSIONS: We anticipate our program of work will support improved quality of life and patient self-management, improved patient-centered primary care delivery, and will encourage the adoption of goal-oriented care approaches across primary health care systems. We have partnered with family health teams and quality improvement organizations in Ontario to ensure that our research is practical and that findings are shared widely. We will work with our established international network to develop an implementation framework to support continued adaptation and adoption across Canada and internationally.

Correction of: Supporting Goal-Oriented Primary Health Care for Seniors with Complex Care Needs Using Mobile Technology: Evaluation and Implementation of the Health System Performance Research Network, Bridgepoint Electronic Patient Reported Outcome Tool.

[This corrects the article DOI: 10.2196/resprot.5756.].

Using a Handheld Device for Patient Data Collection: A Pilot for Medical Countermeasures Surveillance.

Medical countermeasures (MCMs) are medical products used during public health emergencies. This study, conducted within the Mini-Sentinel Initiative, sought to develop the patient identification and matching processes necessary to assess safety outcomes for MCMs. A handheld device was used to collect identifying information (e.g., name, birthdate, and sex) from the driver's licenses of 421 individuals presenting for routine care at their primary care medical office. Overall, 374 individuals (88.8%) could be linked to their electronic health data using driver's license information. The device was also pilot-tested at a seasonal influenza immunization clinic: detailed vaccine information (e.g., lot number and manufacturer) was captured with a high degree of accuracy. This investigation demonstrated that a handheld device is a feasible means of collecting patient identity and medical product receipt data. This capacity should be useful for safety surveillance of MCMs, particularly when dispensed in settings outside the traditional health-care delivery system.

Occult Uterine Sarcoma and Leiomyosarcoma: Incidence of and Survival Associated With Morcellation.

OBJECTIVE: To estimate the incidence of occult uterine sarcoma and leiomyosarcoma in hysterectomies for leiomyomas and the risk associated with their morcellation. METHODS: We conducted a population-based cohort study. All uterine sarcomas from 2006-2013 in an integrated health care system were identified. Age- and race-specific incidences of occult uterine sarcoma were calculated. Kaplan-Meier survival analysis was performed. Crude and adjusted risk ratios of recurrence and death associated with morcellation at 1, 2, and 3 years were estimated using Poisson regression with inverse probability weighting. RESULTS: There were 125 hysterectomies with occult uterine sarcomas identified among 34,728 hysterectomies performed for leiomyomas. The incidence of occult uterine sarcoma and leiomyosarcoma was 1 of 278 or 3.60 (95% confidence interval [CI] 2.97-4.23) and 1 of 429 or 2.33 (95% CI 1.83-2.84) per 1,000 hysterectomies. For stage I leiomyosarcoma (n=111), eight (7.2%) were power and 27 (24.3%) nonpower-morcellated. The unadjusted 3-year probability of disease-free survival for no morcellation, power and nonpower morcellation was 0.54, 0.19, and 0.51, respectively (P=.15); overall survival was 0.64, 0.75, and 0.68, respectively (P=.97). None of the adjusted risk ratios for recurrence or death were significant except for death at 1 year for power and nonpower morcellation groups combined (6/33) compared with no morcellation (4/76) (5.12, 95% CI 1.33-19.76, P=.02). We had inadequate power to infer differences for all other comparisons including 3-year survival and power morcellation. CONCLUSION: Morcellation is associated with decreased early survival of women with occult leiomyosarcomas. We could not accurately assess associations between power morcellation and 3-year survival as a result of small numbers.

Natural language processing in the electronic medical record: assessing clinician adherence to tobacco treatment guidelines.

BACKGROUND: Comprehensively assessing care quality with electronic medical records (EMRs) is not currently possible because much data reside in clinicians' free-text notes. METHODS: We evaluated the accuracy of MediClass, an automated, rule-based classifier of the EMR that incorporates natural language processing, in assessing whether clinicians: (1) asked if the patient smoked; (2) advised them to stop; (3) assessed their readiness to quit; (4) assisted them in quitting by providing information or medications; and (5) arranged for appropriate follow-up care (i.e., the 5A's of smoking-cessation care). DESIGN: We analyzed 125 medical records of known smokers at each of four HMOs in 2003 and 2004. One trained abstractor at each HMO manually coded all 500 records according to whether or not each of the 5A's of smoking cessation care was addressed during routine outpatient visits. MEASUREMENTS: For each patient's record, we compared the presence or absence of each of the 5A's as assessed by each human coder and by MediClass. We measured the chance-corrected agreement between the human raters and MediClass using the kappa statistic. RESULTS: For "ask" and "assist," agreement among human coders was indistinguishable from agreement between humans and MediClass (p>0.05). For "assess" and "advise," the human coders agreed more with each other than they did with MediClass (p<0.01); however, MediClass performance was sufficient to assess quality in these areas. The frequency of "arrange" was too low to be analyzed. CONCLUSIONS: MediClass performance appears adequate to replace human coders of the 5A's of smoking-cessation care, allowing for automated assessment of clinician adherence to one of the most important, evidence-based guidelines in preventive health care.