Validating estimates of prevalence of non-communicable diseases based on household surveys: the symptomatic diagnosis study.

BACKGROUND: Easy-to-collect epidemiological information is critical for the more accurate estimation of the prevalence and burden of different non-communicable diseases around the world. Current measurement is restricted by limitations in existing measurement systems in the developing world and the lack of biometry tests for non-communicable diseases. Diagnosis based on self-reported signs and symptoms ("Symptomatic Diagnosis," or SD) analyzed with computer-based algorithms may be a promising method for collecting timely and reliable information on non-communicable disease prevalence. The objective of this study was to develop and assess the performance of a symptom-based questionnaire to estimate prevalence of non-communicable diseases in low-resource areas. METHODS: As part of the Population Health Metrics Research Consortium study, we collected 1,379 questionnaires in Mexico from individuals who suffered from a non-communicable disease that had been diagnosed with gold standard diagnostic criteria or individuals who did not suffer from any of the 10 target conditions. To make the diagnosis of non-communicable diseases, we selected the Tariff method, a technique developed for verbal autopsy cause of death calculation. We assessed the performance of this instrument and analytical techniques at the individual and population levels. RESULTS: The questionnaire revealed that the information on health care experience retrieved achieved 66.1% (95% uncertainty interval [UI], 65.6-66.5%) chance corrected concordance with true diagnosis of non-communicable diseases using health care experience and 0.826 (95% UI, 0.818-0.834) accuracy in its ability to calculate fractions of different causes. SD is also capable of outperforming the current estimation techniques for conditions estimated by questionnaire-based methods. CONCLUSIONS: SD is a viable method for producing estimates of the prevalence of non-communicable diseases in areas with low health information infrastructure. This technology can provide higher-resolution prevalence data, more flexible data collection, and potentially individual diagnoses for certain conditions.

A shortened verbal autopsy instrument for use in routine mortality surveillance systems.

BACKGROUND: Verbal autopsy (VA) is recognized as the only feasible alternative to comprehensive medical certification of deaths in settings with no or unreliable vital registration systems. However, a barrier to its use by national registration systems has been the amount of time and cost needed for data collection. Therefore, a short VA instrument (VAI) is needed. In this paper we describe a shortened version of the VAI developed for the Population Health Metrics Research Consortium (PHMRC) Gold Standard Verbal Autopsy Validation Study using a systematic approach. METHODS: We used data from the PHMRC validation study. Using the Tariff 2.0 method, we first established a rank order of individual questions in the PHMRC VAI according to their importance in predicting causes of death. Second, we reduced the size of the instrument by dropping questions in reverse order of their importance. We assessed the predictive performance of the instrument as questions were removed at the individual level by calculating chance-corrected concordance and at the population level with cause-specific mortality fraction (CSMF) accuracy. Finally, the optimum size of the shortened instrument was determined using a first derivative analysis of the decline in performance as the size of the VA instrument decreased for adults, children, and neonates. RESULTS: The full PHMRC VAI had 183, 127, and 149 questions for adult, child, and neonatal deaths, respectively. The shortened instrument developed had 109, 69, and 67 questions, respectively, representing a decrease in the total number of questions of 40-55%. The shortened instrument, with text, showed non-significant declines in CSMF accuracy from the full instrument with text of 0.4%, 0.0%, and 0.6% for the adult, child, and neonatal modules, respectively. CONCLUSIONS: We developed a shortened VAI using a systematic approach, and assessed its performance when administered using hand-held electronic tablets and analyzed using Tariff 2.0. The length of a VA questionnaire was shortened by almost 50% without a significant drop in performance. The shortened VAI developed reduces the burden of time and resources required for data collection and analysis of cause of death data in civil registration systems.

Improving performance of the Tariff Method for assigning causes of death to verbal autopsies.

BACKGROUND: Reliable data on the distribution of causes of death (COD) in a population are fundamental to good public health practice. In the absence of comprehensive medical certification of deaths, the only feasible way to collect essential mortality data is verbal autopsy (VA). The Tariff Method was developed by the Population Health Metrics Research Consortium (PHMRC) to ascertain COD from VA information. Given its potential for improving information about COD, there is interest in refining the method. We describe the further development of the Tariff Method. METHODS: This study uses data from the PHMRC and the National Health and Medical Research Council (NHMRC) of Australia studies. Gold standard clinical diagnostic criteria for hospital deaths were specified for a target cause list. VAs were collected from families using the PHMRC verbal autopsy instrument including health care experience (HCE). The original Tariff Method (Tariff 1.0) was trained using the validated PHMRC database for which VAs had been collected for deaths with hospital records fulfilling the gold standard criteria (validated VAs). In this study, the performance of Tariff 1.0 was tested using VAs from household surveys (community VAs) collected for the PHMRC and NHMRC studies. We then corrected the model to account for the previous observed biases of the model, and Tariff 2.0 was developed. The performance of Tariff 2.0 was measured at individual and population levels using the validated PHMRC database. RESULTS: For median chance-corrected concordance (CCC) and mean cause-specific mortality fraction (CSMF) accuracy, and for each of three modules with and without HCE, Tariff 2.0 performs significantly better than the Tariff 1.0, especially in children and neonates. Improvement in CSMF accuracy with HCE was 2.5%, 7.4%, and 14.9% for adults, children, and neonates, respectively, and for median CCC with HCE it was 6.0%, 13.5%, and 21.2%, respectively. Similar levels of improvement are seen in analyses without HCE. CONCLUSIONS: Tariff 2.0 addresses the main shortcomings of the application of the Tariff Method to analyze data from VAs in community settings. It provides an estimation of COD from VAs with better performance at the individual and population level than the previous version of this method, and it is publicly available for use.

Using verbal autopsy to measure causes of death: the comparative performance of existing methods.

BACKGROUND: Monitoring progress with disease and injury reduction in many populations will require widespread use of verbal autopsy (VA). Multiple methods have been developed for assigning cause of death from a VA but their application is restricted by uncertainty about their reliability. METHODS: We investigated the validity of five automated VA methods for assigning cause of death: InterVA-4, Random Forest (RF), Simplified Symptom Pattern (SSP), Tariff method (Tariff), and King-Lu (KL), in addition to physician review of VA forms (PCVA), based on 12,535 cases from diverse populations for which the true cause of death had been reliably established. For adults, children, neonates and stillbirths, performance was assessed separately for individuals using sensitivity, specificity, Kappa, and chance-corrected concordance (CCC) and for populations using cause specific mortality fraction (CSMF) accuracy, with and without additional diagnostic information from prior contact with health services. A total of 500 train-test splits were used to ensure that results are robust to variation in the underlying cause of death distribution. RESULTS: Three automated diagnostic methods, Tariff, SSP, and RF, but not InterVA-4, performed better than physician review in all age groups, study sites, and for the majority of causes of death studied. For adults, CSMF accuracy ranged from 0.764 to 0.770, compared with 0.680 for PCVA and 0.625 for InterVA; CCC varied from 49.2% to 54.1%, compared with 42.2% for PCVA, and 23.8% for InterVA. For children, CSMF accuracy was 0.783 for Tariff, 0.678 for PCVA, and 0.520 for InterVA; CCC was 52.5% for Tariff, 44.5% for PCVA, and 30.3% for InterVA. For neonates, CSMF accuracy was 0.817 for Tariff, 0.719 for PCVA, and 0.629 for InterVA; CCC varied from 47.3% to 50.3% for the three automated methods, 29.3% for PCVA, and 19.4% for InterVA. The method with the highest sensitivity for a specific cause varied by cause. CONCLUSIONS: Physician review of verbal autopsy questionnaires is less accurate than automated methods in determining both individual and population causes of death. Overall, Tariff performs as well or better than other methods and should be widely applied in routine mortality surveillance systems with poor cause of death certification practices.

Assessing quality of medical death certification: Concordance between gold standard diagnosis and underlying cause of death in selected Mexican hospitals.

BACKGROUND: In Mexico, the vital registration system relies on information collected from death certificates to generate official mortality figures. Although the death certificate has high coverage across the country, there is little information regarding its validity. The objective of this study was to assess the concordance between the underlying cause of death in official statistics obtained from death certificates and a gold standard diagnosis of the same deaths derived from medical records of hospitals. METHODS: The study sample consisted of 1,589 deaths that occurred in 34 public hospitals in the Federal District and the state of Morelos, Mexico in 2009. Neonatal, child, and adult cases were selected for causes of death that included infectious diseases, noncommunicable diseases, and injuries. We compared the underlying cause of death, obtained from medical death certificates, against a gold standard diagnosis derived from a review of medical records developed by the Population Health Metrics Research Consortium. We used chance-corrected concordance and accuracy as metrics to evaluate the quality of performance of the death certificate. RESULTS: Analysis considering only the underlying cause of death resulted in a median chance-corrected concordance between the cause of death in medical death certificates versus the gold standard of 54.3% (95% uncertainty interval [UI]: 52.2, 55.6) for neonates, 38.5% (37.0, 40.0) for children, and 66.5% (65.9, 66.9) for adults. The accuracy resulting from the same analysis was 0.756 (0.747, 0.769) for neonates, 0.683 (0.663, 0.701) for children, and 0.780 (0.774, 0.785) for adults. Median chance-corrected concordance and accuracy increased when considering the mention of any cause of death in the death certificate, not just the underlying cause. Concordance varied substantially depending on cause of death, and accuracy varied depending on the true cause-specific mortality fraction composition. CONCLUSIONS: Although we cannot generalize our conclusions to Mexico as a whole, the results demonstrate important problems with the quality of the main source of information for causes of death used by decision-makers in settings with highly technological vital registration systems. It is necessary to improve death certification procedures, especially in the case of child and neonatal deaths. This requires an important commitment from the health system and health institutions.

Population Health Metrics Research Consortium gold standard verbal autopsy validation study: design, implementation, and development of analysis datasets.

BACKGROUND: Verbal autopsy methods are critically important for evaluating the leading causes of death in populations without adequate vital registration systems. With a myriad of analytical and data collection approaches, it is essential to create a high quality validation dataset from different populations to evaluate comparative method performance and make recommendations for future verbal autopsy implementation. This study was undertaken to compile a set of strictly defined gold standard deaths for which verbal autopsies were collected to validate the accuracy of different methods of verbal autopsy cause of death assignment. METHODS: Data collection was implemented in six sites in four countries: Andhra Pradesh, India; Bohol, Philippines; Dar es Salaam, Tanzania; Mexico City, Mexico; Pemba Island, Tanzania; and Uttar Pradesh, India. The Population Health Metrics Research Consortium (PHMRC) developed stringent diagnostic criteria including laboratory, pathology, and medical imaging findings to identify gold standard deaths in health facilities as well as an enhanced verbal autopsy instrument based on World Health Organization (WHO) standards. A cause list was constructed based on the WHO Global Burden of Disease estimates of the leading causes of death, potential to identify unique signs and symptoms, and the likely existence of sufficient medical technology to ascertain gold standard cases. Blinded verbal autopsies were collected on all gold standard deaths. RESULTS: Over 12,000 verbal autopsies on deaths with gold standard diagnoses were collected (7,836 adults, 2,075 children, 1,629 neonates, and 1,002 stillbirths). Difficulties in finding sufficient cases to meet gold standard criteria as well as problems with misclassification for certain causes meant that the target list of causes for analysis was reduced to 34 for adults, 21 for children, and 10 for neonates, excluding stillbirths. To ensure strict independence for the validation of methods and assessment of comparative performance, 500 test-train datasets were created from the universe of cases, covering a range of cause-specific compositions. CONCLUSIONS: This unique, robust validation dataset will allow scholars to evaluate the performance of different verbal autopsy analytic methods as well as instrument design. This dataset can be used to inform the implementation of verbal autopsies to more reliably ascertain cause of death in national health information systems.

Analysis of causes of death using verbal autopsies and vital registration in Hidalgo, Mexico.

INTRODUCTION: Verbal autopsy (VA) is a useful tool for evaluating causes of death, especially in places with limited or no vital registration systems. The Population Health Metrics Research Consortium (PHMRC) developed a validated questionnaire and a set of automated methods to determine the cause of death from a VA. However, the application of these methods needs to be tested in a community environment. OBJECTIVE: To estimate cause-specific mortality fractions (CSMFs) using VAs and compare them against those obtained in the vital statistics of the state of Hidalgo, Mexico. METHODS: A random sample of deaths occurred in 2009 was selected from vital statistics in the state of Hidalgo. The full PHMRC validated VA instrument was applied to the relatives of the deceased, and the cause of death was determined using Tariff's automated method. The causes of death were grouped into 34 causes for adults, 21 for children and 6 for newborns. Results were compared with cause of death on death certificates for all deaths. RESULTS: A total of 1,198 VAs were analyzed. The Tariff method was not able to assign a cause of death in only 9% of adults, 2% of children and 7% of neonatal deaths. The CSMFs obtained from the Tariff method were similar in some cases to those of vital statistics (e.g. cirrhosis), but different in others (e.g. sepsis). CONCLUSION: The application of VAs in a community sample, analyzed with the Tariff method, allowed assigning a cause of death to most of the cases, with results similar to those of vital statistics for most conditions. This tool can be useful to strengthen the quality of vital statistics.