Impact of moral injury and posttraumatic stress disorder on health care utilization and suicidality in rural and urban veterans.

This study explored the impact of moral injury (MI) and posttraumatic stress disorder (PTSD) on health care utilization, mental health complexity, and suicidality in rural and urban veterans. Analyses combined data from the Salt Lake City PTSD Clinic Intake Database and the Department of Veterans Affairs Corporate Data Warehouse. Participants (N = 1,545; Mage = 45.9 years) were predominately male (88.3%) and White (87.8%). Adjusted analyses indicated associations between a 1-unit increase in Moral Injury Events Scale (MIES) score and increased mental health complexity, RR = 1.01, 95% CI [1.01, 1.02], p < .001; psychotropic medication utilization, RR = 1.01, 95% CI [1.01, 1.03], p < .001; VA drug class count, RR = 1.01, 95% CI [1.00, 1.01], p = .030; outpatient utilization, RR = 1.01, 95% CI [1.01, 1.02], p < .001; and mental health outpatient utilization, RR = 1.01, 95% CI [1.00, 1.03], p < .001. For the MIES x PTSD interaction, all associations remained statistically significant with similar estimated effects. However, for rural veterans, this interaction did not significantly affect utilization. Among those with PTSD, a 1-unit MIES increase was associated with an increased risk of suicidality, OR = 1.02, 95% CI [1.01, 1.04], and psychiatric admission, OR = 1.02, 95% CI [1.00, 1.04]. Findings suggest that higher MIES scores predict increased health care utilization and mental health complexity. Further, PTSD combined with higher MIES scores may increase the risk of suicidality and psychiatric admission. Rural veterans with PTSD and higher MIES scores may require additional outreach and intervention.

Usability Testing a Potentially Inappropriate Medication Dashboard: A Core Component of the Dashboard Development Process.

BACKGROUND: With the increased usage of dashboard reporting systems to monitor and track patient panels by clinical users, developers must ensure that the information displays they produce are accurate and intuitive. When evaluating usability of a clinical dashboard among potential end users, developers oftentimes rely on methods such as questionnaires as opposed to other, more time-intensive strategies that incorporate direct observation. OBJECTIVES: Prior to release of the potentially inappropriate medication (PIM) clinical dashboard, designed to facilitate completion of a quality improvement project by clinician scholars enrolled in the Veterans Affairs (VA) workforce development Geriatric Scholars Program (GSP), we evaluated the usability of the system. This article describes the process of usability testing a dashboard reporting system with clinicians using direct observation and think-aloud moderating techniques. METHODS: We developed a structured interview protocol that combines virtual observation, think-aloud moderating techniques, and retrospective questioning of the overall user experience, including use of the System Usability Scale (SUS). Thematic analysis was used to analyze field notes from the interviews of three GSP alumni. RESULTS: Our structured approach to usability testing identified specific functional problems with the dashboard reporting system that were missed by results from the SUS. Usability testing lead to overall improvements in the intuitive use of the system, increased data transparency, and clarification of the dashboard's purpose. CONCLUSION: Reliance solely on questionnaires and surveys at the end stages of dashboard development can mask potential functional problems that will impede proper usage and lead to misinterpretation of results. A structured approach to usability testing in the developmental phase is an important tool for developers of clinician friendly systems for displaying easily digested information and tracking outcomes for the purpose of quality improvement.

Risk of Total Wrist Arthrodesis or Reoperation Following 4-Corner Arthrodesis or Proximal Row Carpectomy for Stage-II SLAC/SNAC Arthritis: A Propensity Score Analysis of 502 Wrists.

BACKGROUND: For stage-II scapholunate advanced collapse (SLAC) and scaphoid nonunion advanced collapse (SNAC) patterns of wrist arthritis, the optimal method of surgical treatment remains unclear. Previous literature has demonstrated similar clinical outcomes between proximal row carpectomy (PRC) and 4-corner arthrodesis (FCA), making the risk of reoperation a focus of particular interest. In the present study, the primary null hypothesis was that there would be no difference in the rate of conversion to total wrist arthrodesis between PRC and FCA. Additionally, we hypothesized that the rate of secondary surgical procedures would be similar between the 2 procedures. METHODS: The national Veterans Health Administration Corporate Data Warehouse was utilized to identify 2,449 patients who underwent either PRC or FCA between 1992 and 2016. With use of operative reports to identify the arthritis pattern, only cases of stage-II SLAC/SNAC were included. All complications and subsequent surgical procedures were confirmed by manual chart review. Propensity score analyses with matching weights were utilized to balance the PRC and FCA cohorts. The rates of conversion to wrist arthrodesis and secondary surgical procedures were calculated. RESULTS: Of the 1,168 patients with stage-II SLAC/SNAC arthritis, 933 wrists underwent PRC and 257 wrists underwent FCA. Ten-year survival free of total wrist arthrodesis in the matching PRC (251 procedures) and FCA (251 procedures) cohorts was 94.3% (95% confidence interval [CI], 92.3% to 96.3%) and 94.1% (95% CI, 90.8% to 97.4%), respectively. Survival free of a secondary surgical procedure other than wrist arthrodesis was 99.7% (95% CI, 99.3% to 100.0%) for PRC and 83.5% (95% CI, 78.2% to 88.8%) for FCA. CONCLUSIONS: PRC and FCA demonstrated similarly low rates of conversion to total wrist arthrodesis. In contrast, the rate of secondary surgical procedures following FCA was significantly higher compared with PRC. Given the historically similar clinical outcomes between PRC and FCA, the results of the present study show that PRC may be a preferable treatment for stage-II SLAC/SNAC wrist arthritis. LEVEL OF EVIDENCE: Therapeutic Level III. See Instructions for Authors for a complete description of levels of evidence.

VA Geriatric Scholars Program's impact on prescribing potentially inappropriate medications.

OBJECTIVES: The Veterans Affairs (VA) Geriatric Scholars Program (GSP) is a workforce development program to enhance skills and competencies among VA clinicians who provide healthcare for older veterans in VA primary care clinics. An intensive geriatrics didactics (IGD) course is a core element of this professional development program. The objective of this study was to evaluate the impact of completing the IGD course on providers' rates of prescribing definite potentially inappropriate medications (DPIMs) based on Beers Criteria from 2008 to 2016. STUDY DESIGN: We applied a longitudinal interrupted time series design to examine changes in DPIM prescribing rates for GSP participants before and after completing the IGD course. METHODS: The time series was divided into two 12-month periods, representing the preintervention period (ie, 12 months prior to completing the IGD course) and the postintervention period (ie, 12 months after completing the IGD course), and populated with pharmacy dispensing data from the VA's Corporate Data Warehouse. An adjusted slope impact model was developed to estimate the postintervention change in the proportion of the dispensed medications identified as DPIMs. RESULTS: After adjusting for case mix, we observed a statistically significant reduction in the proportion of DPIMs dispensed post IGD (slope change, 0.994; 95% CI, 0.991-0.997). This change in slope reflects a total decrease of 7971 DPIM dispensings during the postintervention period. This equates to an estimated 24 fewer DPIM dispensings per provider during the postintervention period. CONCLUSIONS: Although the size of the effect was modest, we found that participation in the GSP IGD course reduced prescribing of DPIMs for older veterans.

Validation of Algorithms Used to Identify Red Blood Cell Transfusion Related Admissions in Veteran Patients with End Stage Renal Disease.

BACKGROUND: The goal of this study was to compare the performance of several database algorithms designed to identify red blood cell (RBC) Transfusion Related hospital Admissions (TRAs) in Veterans with end stage renal disease (ESRD). METHODS: Hospitalizations in Veterans with ESRD and evidence of dialysis between 01/01/2008 and 12/31/2013 were screened for TRAs using a clinical algorithm (CA) and four variations of claims-based algorithms (CBA 1-4). Criteria were implemented to exclude patients with non-ESRD-related anemia (e.g., injury, surgery, bleeding, medications known to produce anemia). Diagnostic performance of each algorithm was delineated based on two clinical representations of a TRA: RBC transfusion required to treat ESRD-related anemia on admission regardless of the reason for admission (labeled as TRA) and hospitalization for the primary purpose of treating ESRD-related anemia (labeled TRA-Primary). The performance of all algorithms was determined by comparing each to a reference standard established by medical records review. Population-level estimates of classification agreement statistics were calculated for each algorithm using inverse probability weights and bootstrapping procedures. Due to the low prevalence of TRAs, the geometric mean was considered the primary measure of algorithm performance. RESULTS: After application of exclusion criteria, the study consisted of 12,388 Veterans with 26,672 admissions. The CA had a geometric mean of 90.8% (95% Confidence Interval: 81.8, 95.6) and 94.7% (95% CI: 80.5, 98.7) for TRA and TRA-Primary, respectively. The geometric mean for the CBAs ranged from 60.3% (95% CI: 53.2, 66.9) to 91.8% (95% CI: 86.9, 95) for TRA, and from 80.7% (95% CI: 72.9, 86.7) to 96.7% (95% CI: 94.1, 98.2) for TRA-Primary. The adjusted proportions of admissions classified as TRAs was 3.2% (95% CI: 2.8, 3.8) and TRA-Primary was 1.3% (95% CI: 1.1, 1.7). CONCLUSIONS: The CA and select CBAs were able to identify TRAs and TRA-primary with high levels of accuracy and can be used to examine anemia management practices in ESRD patients.

Cohort identification of axial spondyloarthritis in a large healthcare dataset: current and future methods.

BACKGROUND: Big data research is important for studying uncommon diseases in real-world settings. Most big data studies in axial spondyloarthritis (axSpA) have been limited to populations identified with billing codes for ankylosing spondylitis (AS). axSpA is a more inclusive concept, and reliance on AS codes does not produce a comprehensive axSpA study population. The first objective was to describe our process for establishing an appropriate sample of patients with and without axSpA for developing accurate axSpA identification methods. The second objective was to determine the classification performance of AS billing codes against the chart-reviewed axSpA reference standard. METHODS: Veteran Health Affairs clinical and administrative data, between January 2005 and June 2015, were used to randomly select patients with clinical phenotypes that represented high, moderate, and low likelihoods of an axSpA diagnosis. With chart review, the sampled patients were classified as Yes axSpA, No axSpA or Uncertain axSpA, and these classification assignments were used as the reference standard for determining the positive predictive value (PPV) and sensitivity of AS ICD-9 codes for axSpA. RESULTS: Six hundred patients were classified as Yes axSpA (26.8%), No axSpA (68.3%), or Uncertain axSpA (4.8%). The PPV and sensitivity of an AS ICD-9 code for axSpA were 83.3% and 57.3%, respectively. CONCLUSIONS: Standard methods of identifying axSpA patients in a large dataset lacked sensitivity. An appropriate sample of patients with and without axSpA was established and characterized for developing novel axSpA identification methods that are anticipated to enable previously impractical big data research.

Predicting Psychiatric Hospitalizations among Elderly Veterans with a History of Mental Health Disease.

INTRODUCTION: Patient Aligned Care Team (PACT) care managers are tasked with identifying aging Veterans with psychiatric disease in attempt to prevent psychiatric crises. However, few resources exist that use real-time information on patient risk to prioritize coordinating appropriate care amongst a complex aging population. OBJECTIVE: To develop and validate a model to predict psychiatric hospital admission, during a 90-day risk window, in Veterans ages 65 or older with a history of mental health disease. METHODS: This study applied a cohort design to historical data available in the Veterans Affairs (VA) Corporate Data Warehouse (CDW). The Least Absolute Shrinkage and Selection Operator (LASSO) regularization regression technique was used for model development and variable selection. Individual predicted probabilities were estimated using logistic regression. A split-sample approach was used in performing external validation of the fitted model. The concordance statistic (C-statistic) was calculated to assess model performance. RESULTS: Prior to modeling, 61 potential candidate predictors were identified and 27 variables remained after applying the LASSO method. The final model's predictive accuracy is represented by a C-statistic of 0.903. The model's predictive accuracy during external validation is represented by a C-statistic of 0.935. Having a previous psychiatric hospitalization, psychosis, bipolar disorder, and the number of mental-health related social work encounters were strong predictors of a geriatric psychiatric hospitalization. CONCLUSION: This predictive model is capable of quantifying the risk of a geriatric psychiatric hospitalization with acceptable performance and allows for the development of interventions that could potentially reduce such risk.

Persistence With Conventional Triple Therapy Versus a Tumor Necrosis Factor Inhibitor and Methotrexate in US Veterans With Rheumatoid Arthritis.

OBJECTIVE: To compare persistence and adherence to triple therapy with the nonbiologic disease-modifying antirheumatic drugs (DMARDs) methotrexate (MTX), hydroxychloroquine, and sulfasalazine, versus a tumor necrosis factor inhibitor (TNFi) plus MTX in patients with rheumatoid arthritis (RA). METHODS: Administrative and laboratory data were analyzed for US Veterans with RA initiating triple therapy or TNFi + MTX between January 2006 and December 2012. Treatment persistence 365 days postindex was calculated using 3 definitions. Definition 1 required no gap in therapy of ≥90 days for any drug in the original combination. Definition 2 required no added or switched DMARD, no decrease to nonbiologic DMARD monotherapy, and no termination of all DMARD therapies. Definition 3 was similar to definition 2 but allowed a switch to another drug within the same class. Adherence used a proportion of days covered of ≥80%. Propensity-weighted analysis with matched weights was used to balance covariates. RESULTS: The analysis included 4,364 RA patients (TNFi + MTX, n = 3,204; triple therapy, n = 1,160). In propensity-weighted analysis, patients in the TNFi + MTX group were significantly more likely than patients in the triple therapy group to satisfy all persistence criteria in definition 1 (risk difference [RD] 13.1% [95% confidence interval (95% CI) 9.2-17.0]), definition 2 (RD 6.4% [95% CI 2.3-10.5]), and definition 3 (RD 9.5% [95% CI 5.5-13.6]). Patients in the TNFi + MTX group also exhibited higher adherence during the first year (RD 7.2% [95% CI 3.8-10.5]). CONCLUSION: US Veterans with RA were significantly more likely to be persistent and adherent to combination therapy with TNFi + MTX than triple therapy with nonbiologic DMARDs.

Identifying Axial Spondyloarthritis in Electronic Medical Records of US Veterans.

OBJECTIVE: Large database research in axial spondyloarthritis (SpA) is limited by a lack of methods for identifying most types of axial SpA. Our objective was to develop methods for identifying axial SpA concepts in the free text of documents from electronic medical records. METHODS: Veterans with documents in the national Veterans Health Administration Corporate Data Warehouse between January 1, 2005 and June 30, 2015 were included. Methods were developed for exploring, selecting, and extracting meaningful terms that were likely to represent axial SpA concepts. With annotation, clinical experts reviewed sections of text containing the meaningful terms (snippets) and classified the snippets according to whether or not they represented the intended axial SpA concept. With natural language processing (NLP) tools, computers were trained to replicate the clinical experts' snippet classifications. RESULTS: Three axial SpA concepts were selected by clinical experts, including sacroiliitis, terms including the prefix spond*, and HLA-B27 positivity (HLA-B27+). With supervised machine learning on annotated snippets, NLP models were developed with accuracies of 91.1% for sacroiliitis, 93.5% for spond*, and 97.2% for HLA-B27+. With independent validation, the accuracies were 92.0% for sacroiliitis, 91.0% for spond*, and 99.0% for HLA-B27+. CONCLUSION: We developed feasible and accurate methods for identifying axial SpA concepts in the free text of clinical notes. Additional research is required to determine combinations of concepts that will accurately identify axial SpA phenotypes. These novel methods will facilitate previously impractical observational research in axial SpA and may be applied to research with other diseases.

The use of natural language processing on narrative medication schedules to compute average weekly dose.

PURPOSE: Medications with non-standard dosing and unstandardized units of measurement make the estimation of prescribed dose difficult from pharmacy dispensing data. A natural language processing tool named the SIG extractor was developed to identify and extract elements from narrative medication instructions to compute average weekly doses (AWDs) for disease-modifying antirheumatic drugs. The goal of this paper is to evaluate the performance of the SIG extractor. METHOD: This agreement study utilized Veterans Health Affairs pharmacy data from 2008 to 2012. The SIG extractor was designed to extract key elements from narrative medication schedules (SIGs) for 17 select medications to calculate AWD, and these medications were categorized by generic name and route of administration. The SIG extractor was evaluated against an annotator-derived reference standard for accuracy, which is the fraction of AWDs accurately computed. RESULTS: The overall accuracy was 89% [95% confidence interval (CI) 88%, 90%]. The accuracy was ≥85% for all medications and route combinations, except for cyclophosphamide (oral) and cyclosporine (oral), which were 79% (95%CI 72%, 85%) and 66% (95%CI 58%, 73%), respectively. CONCLUSIONS: The SIG extractor performed well on the majority of medications, indicating that AWD calculated by the SIG extractor can be used to improve estimation of AWD when dispensed quantity or days' supply is questionable or improbable. The working model for annotating SIGs and the SIG extractor are generalized and can easily be applied to other medications. Copyright © 2016 John Wiley & Sons, Ltd.

Normalizing acronyms and abbreviations to aid patient understanding of clinical texts: ShARe/CLEF eHealth Challenge 2013, Task 2.

BACKGROUND: The ShARe/CLEF eHealth challenge lab aims to stimulate development of natural language processing and information retrieval technologies to aid patients in understanding their clinical reports. In clinical text, acronyms and abbreviations, also referenced as short forms, can be difficult for patients to understand. For one of three shared tasks in 2013 (Task 2), we generated a reference standard of clinical short forms normalized to the Unified Medical Language System. This reference standard can be used to improve patient understanding by linking to web sources with lay descriptions of annotated short forms or by substituting short forms with a more simplified, lay term. METHODS: In this study, we evaluate 1) accuracy of participating systems' normalizing short forms compared to a majority sense baseline approach, 2) performance of participants' systems for short forms with variable majority sense distributions, and 3) report the accuracy of participating systems' normalizing shared normalized concepts between the test set and the Consumer Health Vocabulary, a vocabulary of lay medical terms. RESULTS: The best systems submitted by the five participating teams performed with accuracies ranging from 43 to 72 %. A majority sense baseline approach achieved the second best performance. The performance of participating systems for normalizing short forms with two or more senses with low ambiguity (majority sense greater than 80 %) ranged from 52 to 78 % accuracy, with two or more senses with moderate ambiguity (majority sense between 50 and 80 %) ranged from 23 to 57 % accuracy, and with two or more senses with high ambiguity (majority sense less than 50 %) ranged from 2 to 45 % accuracy. With respect to the ShARe test set, 69 % of short form annotations contained common concept unique identifiers with the Consumer Health Vocabulary. For these 2594 possible annotations, the performance of participating systems ranged from 50 to 75 % accuracy. CONCLUSION: Short form normalization continues to be a challenging problem. Short form normalization systems perform with moderate to reasonable accuracies. The Consumer Health Vocabulary could enrich its knowledge base with missed concept unique identifiers from the ShARe test set to further support patient understanding of unfamiliar medical terms.

Performance of a Natural Language Processing (NLP) Tool to Extract Pulmonary Function Test (PFT) Reports from Structured and Semistructured Veteran Affairs (VA) Data.

INTRODUCTION/OBJECTIVE: Pulmonary function tests (PFTs) are objective estimates of lung function, but are not reliably stored within the Veteran Health Affairs data systems as structured data. The aim of this study was to validate the natural language processing (NLP) tool we developed-which extracts spirometric values and responses to bronchodilator administration-against expert review, and to estimate the number of additional spirometric tests identified beyond the structured data. METHODS: All patients at seven Veteran Affairs Medical Centers with a diagnostic code for asthma Jan 1, 2006-Dec 31, 2012 were included. Evidence of spirometry with a bronchodilator challenge (BDC) was extracted from structured data as well as clinical documents. NLP's performance was compared against a human reference standard using a random sample of 1,001 documents. RESULTS: In the validation set NLP demonstrated a precision of 98.9 percent (95 percent confidence intervals (CI): 93.9 percent, 99.7 percent), recall of 97.8 percent (95 percent CI: 92.2 percent, 99.7 percent), and an F-measure of 98.3 percent for the forced vital capacity pre- and post pairs and precision of 100 percent (95 percent CI: 96.6 percent, 100 percent), recall of 100 percent (95 percent CI: 96.6 percent, 100 percent), and an F-measure of 100 percent for the forced expiratory volume in one second pre- and post pairs for bronchodilator administration. Application of the NLP increased the proportion identified with complete bronchodilator challenge by 25 percent. DISCUSSION/CONCLUSION: This technology can improve identification of PFTs for epidemiologic research. Caution must be taken in assuming that a single domain of clinical data can completely capture the scope of a disease, treatment, or clinical test.

Treatment patterns and annual biologic costs in US veterans with rheumatic conditions or psoriasis.

OBJECTIVE: To determine annual biologic drug and administration costs to the US Veterans Health Administration (VHA) per treated patient with rheumatoid arthritis (RA), psoriasis (PsO), psoriatic arthritis (PsA), or ankylosing spondylitis (AS) who received abatacept, adalimumab, certolizumab pegol, etanercept, golimumab, infliximab, rituximab, tocilizumab, or ustekinumab. METHODS: Adults with at least one biologic claim between January 1, 2008 and December 31, 2011 were included. Evidence of enrollment in the VHA was required from 365 days before (pre-index) to 360 days after (post-index) the date of the first biologic claim (index date). Included patients had pre-index diagnoses of RA, PsO, PsA, and/or AS. Drug costs were from Federal Supply Schedule or 'Big Four' in November 2014. Administration costs were VHA fixed costs for infused ($169) and subcutaneous ($25) biologics. RESULTS: Of the 20,465 patients in the analysis, 10,711 received etanercept, 7838 received adalimumab, and 1196 received infliximab as the index biologic. In these patients, across all uses studied, the VHA incurred greater annual cost per treated patient for infliximab ($18,066) compared with adalimumab ($16,523) and etanercept ($16,526). In the first year post-index, ∼80% of patients were either persistent on these index biologics or re-started these index biologics after a ≥45-day treatment gap. Other biologics comprised <5% of the study population, with sample sizes ranging from 3-374 patients each. Cost by indication for biologics used by >20 patients ranged from $15,056 (etanercept) to $17,050 (abatacept) for RA; $16,697 (adalimumab) to $33,163 (ustekinumab) for PsO; $15,035 (etanercept) to $20,465 (infliximab) for PsA; and $14,239 (etanercept) to $18,536 (infliximab) for AS. LIMITATIONS: The model was limited to the VHA. Results for biologics other than adalimumab, etanercept, and infliximab were difficult to interpret because of small sample sizes. CONCLUSIONS: Infliximab has higher cost to the VHA than adalimumab or etanercept.

The use of natural language processing of infusion notes to identify outpatient infusions.

PURPOSE: Outpatient infusions are commonly missing in Veterans Health Affairs (VHA) pharmacy dispensing data sets. Currently, Healthcare Common Procedure Coding System (HCPCS) codes are used to identify outpatient infusions, but concerns exist if they correctly capture all infusions and infusion-related data such as dose and date of administration. We developed natural language processing (NLP) software to extract infusion information from medical text infusion notes. The objective was to compare the sensitivity of three approaches to identify infliximab administration dates and infusion doses against a reference standard established from the Veterans Affairs rheumatoid arthritis (VARA) registry. METHODS: We compared the sensitivity and positive predictive value (PPV) of NLP to that of HCPCS codes in identifying the correct date and dose of infliximab infusions against a human extracted reference standard. RESULTS: The sensitivity was 0.606 (0.585-0.627) for HCPCS alone, 0.858 (0.842-0.873) for NLP alone, and 0.923 (0.911-0.934) for the two methods combined, with a PPV of 0.735 (0.716-0.754), 0.976 (0.969-0.983), and 0.957 (0.948-0.965) for each method, respectively. The mean dose of infliximab was 433 mg in the reference standard, 337 mg from HCPCS, 434 mg from NLP, and 426 mg from the combined method. CONCLUSIONS: HCPCS codes alone are not sufficient to accurately identify infliximab infusion dates and doses in the VHA system. The use of NLP significantly improved the sensitivity and PPV for estimating infusion dates and doses, especially when combined with HCPCS codes.

Evaluating the effects of machine pre-annotation and an interactive annotation interface on manual de-identification of clinical text.

The Health Insurance Portability and Accountability Act (HIPAA) Safe Harbor method requires removal of 18 types of protected health information (PHI) from clinical documents to be considered "de-identified" prior to use for research purposes. Human review of PHI elements from a large corpus of clinical documents can be tedious and error-prone. Indeed, multiple annotators may be required to consistently redact information that represents each PHI class. Automated de-identification has the potential to improve annotation quality and reduce annotation time. For instance, using machine-assisted annotation by combining de-identification system outputs used as pre-annotations and an interactive annotation interface to provide annotators with PHI annotations for "curation" rather than manual annotation from "scratch" on raw clinical documents. In order to assess whether machine-assisted annotation improves the reliability and accuracy of the reference standard quality and reduces annotation effort, we conducted an annotation experiment. In this annotation study, we assessed the generalizability of the VA Consortium for Healthcare Informatics Research (CHIR) annotation schema and guidelines applied to a corpus of publicly available clinical documents called MTSamples. Specifically, our goals were to (1) characterize a heterogeneous corpus of clinical documents manually annotated for risk-ranked PHI and other annotation types (clinical eponyms and person relations), (2) evaluate how well annotators apply the CHIR schema to the heterogeneous corpus, (3) compare whether machine-assisted annotation (experiment) improves annotation quality and reduces annotation time compared to manual annotation (control), and (4) assess the change in quality of reference standard coverage with each added annotator's annotations.