Use of ROC curve analysis for prediction gives fallacious results: Use predictivity-based indices.

ABSTRACT: The area under the ROC curve is frequently used for assessing the predictive efficacy of a model, and the Youden index is commonly used to provide the optimal cut-off. Both are misleading tools for predictions. A ROC curve is drawn for the sensitivity of a quantitative test against its (1 - specificity) at different values of the test. Both sensitivity and specificity are retrospective in nature as these are indicators of correct classification of already known conditions. They are not indicators of future events and are not valid for predictions. Predictivity intimately depends on the prevalence which may be ignored by sensitivity and specificity. We explain this fallacy in detail and illustrate with several examples that the actual predictivity could differ greatly from the ROC curve-based predictivity reported by many authors. The predictive efficacy of a test or a model is best assessed by the percentage correctly predicted in a prospective framework. We propose predictivity-based ROC curves as tools for providing predictivities at varying prevalence in different populations. For optimal cut-off for prediction, in place of the Youden index, we propose a P-index where the sum of positive and negative predictivities is maximum after subtracting 1. To conclude, for correctly assessing adequacy of a prediction models, predictivity-based ROC curves should be used instead of the usual sensitivity-specificity-based ROC curves and the P-index should replace the Youden index.

The importance of small samples in medical research.

Almost all bio-statisticians and medical researchers believe that a large sample is always helpful in providing more reliable results. Whereas this is true for some specific cases, a large sample may not be helpful in more situations than we contemplate because of the higher possibility of errors and reduced validity. Many medical breakthroughs have occurred with self-experimentation and single experiments. Studies, particularly analytical studies, may provide more truthful results with a small sample because intensive efforts can be made to control all the confounders, wherever they operate, and sophisticated equipment can be used to obtain more accurate data. A large sample may be required only for the studies with highly variable outcomes, where an estimate of the effect size with high precision is required, or when the effect size to be detected is small. This communication underscores the importance of small samples in reaching a valid conclusion in certain situations and describes the situations where a large sample is not only unnecessary but may even compromise the validity by not being able to exercise full care in the assessments. What sample size is small depends on the context.

Aleatory and epistemic uncertainties can completely derail medical research results.

Aleatory uncertainties are generated by intrinsic factors such as studying a sample rather than the whole population and the source of epistemic uncertainties is extraneous such as limitations of knowledge. These uncertainties inflict all the findings in empirical medical research, but they are rarely appreciated. This article highlights these uncertainties and shows with the help of an example how apparently valid and reliable findings can completely derail due to these uncertainties. We conclude that aleatory and epistemic uncertainties should get due consideration while drawing conclusions and before the results are put into practice. Methods to reduce their impact on results are also presented.

Statistical medicine: An emerging medical specialty.

A large number of statistical tools are now used for medical decision in the core activities of diagnosis, treatment and prognosis. These tools provide undeniable help in improving medical outcomes. Prominent among them are uncertainty measurement by probability, medical indicators and indexes, reference ranges, and scoring systems. In addition are tools such as odds ratio, sensitivity, specificity and predictivities, area under the ROC curve, likelihood ratios, and cost-benefit analysis that are commonly applied in medical research but have implications for day-to-day clinical activities. These tools have so completely integrated into medical practice that statistical medicine by itself can stand alone as a medical specialty. Time has arrived to recognize statistical medicine as a medical specialty.

Reporting quality of multivariable logistic regression in selected Indian medical journals.

BACKGROUND: Use of multivariable logistic regression (MLR) modeling has steeply increased in the medical literature over the past few years. Testing of model assumptions and adequate reporting of MLR allow the reader to interpret results more accurately. AIMS: To review the fulfillment of assumptions and reporting quality of MLR in selected Indian medical journals using established criteria. SETTING AND DESIGN: Analysis of published literature. MATERIALS AND METHODS: Medknow.com publishes 68 Indian medical journals with open access. Eight of these journals had at least five articles using MLR between the years 1994 to 2008. Articles from each of these journals were evaluated according to the previously established 10-point quality criteria for reporting and to test the MLR model assumptions. STATISTICAL ANALYSIS: SPSS 17 software and non-parametric test (Kruskal-Wallis H, Mann Whitney U, Spearman Correlation). RESULTS: One hundred and nine articles were finally found using MLR for analyzing the data in the selected eight journals. The number of such articles gradually increased after year 2003, but quality score remained almost similar over time. P value, odds ratio, and 95% confidence interval for coefficients in MLR was reported in 75.2% and sufficient cases (>10) per covariate of limiting sample size were reported in the 58.7% of the articles. No article reported the test for conformity of linear gradient for continuous covariates. Total score was not significantly different across the journals. However, involvement of statistician or epidemiologist as a co-author improved the average quality score significantly (P=0.014). CONCLUSIONS: Reporting of MLR in many Indian journals is incomplete. Only one article managed to score 8 out of 10 among 109 articles under review. All others scored less. Appropriate guidelines in instructions to authors, and pre-publication review of articles using MLR by a qualified statistician may improve quality of reporting.

Statistical choropleth cartography in epidemiology.

BACKGROUND: The potential of maps in the study of regional variation and similarity in health and in understanding the underlying processes is being increasingly realized. It has thus become important that more care is exercised in drawing health maps and the subjective elements are minimized. Conventional choropleth maps based on qualitative data are mostly arbitrary with regard to the number of categories and the cutoff points. This can lead to substantially different pictures based on the same data set. METHODS: We suggest use of cluster methods to discover 'natural' groups of data points which to a large extent are suggested by the data themselves. These methods can determine not only the cutoff points but also the number of categories required to depict the variability in the data. The methods have natural extension to the multivariate set-up and thus can provide the strategy to construct integrated maps based on the simultaneous consideration of several variables. Since different cluster methods can yield different grouping we propose a simple method to identify cutoffs common to a majority of the methods. RESULTS: The details of the methods are explained on two real data sets. One is the indicators of mortality before one year of age in India and the other is years of life lost due to premature mortality in different countries. The maps obtained are compared with the conventional maps. CONCLUSION: The cutoff points obtained by a majority of cluster methods deserve attention for obtaining natural groups for choroplethic depiction. Maps based on such cutoffs seem to have promise for increasing the accuracy in perception and cognition of regional variation.

Development of an index of need for health resources for Indian States using factor analysis.

An index of need for health resources in the Indian States has been developed using the statistical technique of factor analysis. Data on seven selected indicators measuring various aspects of population health in 17 Indian States for the year 1981 were used to extract factors. Two factors were discovered which together explained 83% of the total variation. The first factor alone, identified as 'proximate determinants', explained 67% of the variation. The second, 'sociomedical background' accounted for 16%. Factor analysis results have been used to derive standardized indices which helped to compare quantitatively the health needs of the people of various States.

Rapid assessment of prevalence of cataract blindness at district level.

AIM: To find an optimal cluster size and number of clusters for a reasonable estimate of the prevalence of cataract blindness in people aged > or = 50 years in 19 rural districts of a state in India. MATERIALS: Cluster sampling methodology was used in 19 rural districts of Karnataka State, India. In each district, 15 clusters were randomly selected and 90 people aged > or = 50 years were examined in each cluster. As a result the visual acuity and lens status of a total of 22,218 people were assessed. METHODS: For each district, the design effect for cluster size ranging from 20 to 90 was calculated and the optimal cluster size and the required number of clusters to achieve an accuracy of 1% errors and 80% confidence was assessed. RESULTS: The age and gender adjusted prevalence of cataract blindness varied from 1.58% to 7.24%, which justifies district level surveys. The design effect is nearly 1.5 for clusters of sizes 30 and 40. With an average prevalence of 4.93% with 1% error and 80% confidence level, the optimal number of clusters is 37 and 28 for a cluster size of 30 and 40 respectively and the average sample size for a district around 1100. CONCLUSIONS: Rapid assessments for cataract blindness in those aged > or = 50 years can be conducted at district level in India with existing resources and at affordable costs. These provide reliable data, essential for effective monitoring and planning. Other parameters, for instance, surgical coverage can also be assessed. The availability of standardized software for data entry and analysis and strict adherence to survey procedures is essential.

Elements of medical research.

Most medical research is empirical based on evidence rather than hunches or preferences. It follows a series of specific steps. There are no short cuts. Collection of evidence and its analysis should follow a carefully drawn protocol. Most of the modern medical research requires biostatistical tools to reach to a valid and reliable conclusion. Researcher must have an adequate knowledge and skill to be really effective. The endeavours should be consistent with the accepted medical and research ethics. Medical research can provide immense satisfaction when conducted on scientific lines, and can be occasionally frustrating when years of efforts fail to produce expected results. This article focuses on aspects that can increase the credibility of research. It is addressed to all interested in medical research, and seeking answers to questions such as what actually is research, what are its types, what specific steps should be followed, what a research protocol should contain, and what makes research credible etc.

Is the heredity of reticulate acro pigmentation of Kitamura always autosomal dominant?

Reticulate acropigmentation of Kitamura (RAK) in three patients and their families is described. In one family, 2 women and 2 men were affected out of 9 individuals. In the other family, 6 women had lesions of reticulate acropigmentation out of total of 27 over three generations. In the third family, one man was affected. All of the cases had palmar pits; onset of lesions was after puberty in all the cases. The apparently different hereditary patterns in these three families are striking, and autosomal dominant inheritance appears unlikely in every case.

Clinical agreement in quantitative measurements.

With advances in medical technology, simpler and safer methods for diagnosis and therapy are increasingly replacing the old ones and it has become important that these be correctly assessed. When a measurement by a new method is the same as that using the old method, one frequently encounters the problem of assessing the agreement. Evidence such as a correlation equal to 1 or equality of means is known to be inadequate. However, two recent approaches--limits of agreement and intraclass correlation coefficient--have gained acceptance but each has its own merits and demerits. To help investigators choose a procedure which is appropriate and to help them use it properly, we provide a description of these two approaches and discuss their advantages and disadvantages, both clinical and statistical, using a real example.

Computer based statistical study of cartography in mortality upto age of one year.

Present cartography procedures for quantitative indicators are arbitrary on choice of the number of categories in which a particular area is to be divided. The choice of initial cutoff and the choice of the width of each category is also arbitrary. To remove this arbitrariness and thus to introduce objectivity, we propose use of a statistical procedure called cluster analysis. This procedure is easy to use on a computer. We also propose using computer based maps. We use these methods on mortality indicators upto age of one year for major states of India to devise objective maps. The terminology of mortality indicators upto age of one year has been used by UNICEF document(1). The mortality indicators analysed are infant mortality rate, neonatal mortality rate, postneonatal mortality rate, perinatal mortality rate and still birth rate. Different indicators reveal different pictures. In this paper, we also propose an innovation to obtain an integrated picture by simultaneously considering all the four indicators in a multivariate setting. Such mapping could help the health managers and planners to devise more effective strategies to control child mortality.

A comprehensive index for longitudinal monitoring of child health status.

The aim of this study was to develop a single comprehensive index of child mortality for longitudinal assessment of health status of children. The need for such a comprehensive index arose from conflicting trends in different child mortality indicators. The data for the study was taken from the Sample Registration System (SRS) reports of the Registrar General of India. SRS is known to provide reliable estimates of births and deaths at the State and the National level. The study included five child mortality indicators, namely, under five mortality rate (U5MR), infant mortality rate, neonatal mortality rate, perinatal mortality rate and still birth rate. These were available for fifteen states of India over the years 1972-1988. To develop this index we modified an earlier method based on factor analysis. Factor analysis of data on various indicators of child mortality revealed two factors which together explained 78% to 93% of the total variation in different years. The first factor was identified as representing mortality after birth and the second as before and during birth. The comprehensive index was obtained as a linear combination of these two factors. The resultant index thus fairly represented all five mortality indicators and provided a comprehensive and reasonably correct picture of child mortality. The lower the magnitude of this index, the better was the child health status. Trends in the index showed that the highest decline in the magnitude was in the state of Kerala followed by Punjab, Andhra Pradesh, Gujarat and Maharashtra in that order. This indicates steady improvement of the child health status over years in these states. In the State of Jammu and Kashmir, the index remained more or less constant over the years though the magnitude was low in the cross-sectional comparison with other states. Thus the comprehensive index developed by using factor analysis of the various mortality indicators can be used for the longitudinal monitoring of child health status in the states of India.

PIP: This study resulted in the development of a single comprehensive index of child mortality that can provide a longitudinal assessment of the health status of children. Rates of fetal, perinatal, neonatal, infant, and under age 5 mortality were obtained from the Registrar General of India's Sample Registration System for 15 Indian states during 1972-88. The comprehensive index was developed as a linear combination of two factors that explain 78-93% of the total variation in child mortality in different years: 1) mortality after birth and 2) mortality before and during birth. Child health status improves as the magnitude of this index declines. Application of the index revealed improved health status in the states of Kerala, Punjab, Andhra Pradesh, Gujarat, and Maharashtra (in declining order). Because the index of child mortality developed in this study gives differential weights to its several components, depending upon their overall contribution, simple summation of under age 5 mortality rates and the stillbirth rate would not be an adequate replacement for the statistical computation called for by the index. The minor drawbacks to use of the index are the tedious nature of the required calculations, its lack of an actual meaning in terms of mortality rates, and its limited applicability to settings where computed mortality rates of various segments are available. However, the index can be helpful in longitudinal monitoring of child health status and in providing comparisons between different states.

Childbirth spacing in a rural community of Delhi: profile in various marriage cohorts by decades.

Complete retrospective fertility histories of 843 ever married women of two villages in Delhi, obtained through house to house survey, were analysed to study various marriage cohorts by decades for trends of child birth spacing over a period of 60 years from 1921 to 1980. Spacing between consummation of marriage and first child birth gradually declined over the last 6 decades. For all other subsequent livebirths remained constant at an average of nearly 30 months. Irrespective of the reasons for such a trend, much more efforts are required to be put in to increase child spacing.

PIP: From March 1982 to April 1983, researchers from the University College of Medical Sciences in Delhi, India interviewed at least 843 women = or 15 years old and married between 1921-1980 from 2 villages near Delhi about their birth history and year of marriage. There were 3736 live births among the 843 women. 98.8% of the women married between 1921- 1970 had children. Only 182 of the 256 women married between 1971-1980 had already borne children. The spacing between consummation of marriage and the birth of the 1st child (1st birth order) decreased steadily over the 6 decades. For example, the average spacing fell from 44.86 months (1921-1930) to 40.11 (1951-1960) to 28.82 (1971-1980). During this time frame, the mean age at marriage rose gradually, but the mean age for 1st birth did not change. For each decade except 1971- 1980, the spacing for the 1st birth order remained significantly more than most of the other birth orders in that decade. The mean birth interval for all the marriage cohorts stood at 30 months. The average spacing of all births for the 1971-1980 marriage cohort was significantly lower than those of the 5 previous decades (p.05), however. This could be a result of the decline in breast feeding and sexual abstinence in urban areas of India. This trend appears to counteract the efforts of family planning programs. The results of this study indicate that the National Family Welfare Programme must more actively promote child spacing as a means to reduce family size.

Computer-assisted learning package for frequency distribution of physiological variables.

There are a variety of ways of picturing a frequency distribution type, viz, the histogram, the frequency polygon, the smoothed curve and the ogive. We have developed a computer package to demonstrate gradual change of a histogram into a curve. For a given set of data on frequency distribution of, say blood pressure levels in specified categories, this package helps the computer to draw bars which gradually rise to the level of the frequencies, and lateron are replaced by the polygon and finally by the frequency curve of the Gaussian type on the computer screen pixel by pixel. This thus demonstrates the meaning and genesis of frequency curves. This package could be very useful in learning the concept of frequency curves particularly the Gaussian form.

A study of urinary creatinine and creatinine coefficient in healthy young adults.

Daily urinary creatinine excretion and creatinine coefficient of eighty-four healthy medical students have been studied for three consecutive days. Daily creatinine excretion has been observed to be higher in the male subjects compared to the female counterparts. Creatinine excretion has been observed to be dependent on body weight of the subjects. The variability from subject to subject was higher than observed by most other workers. Day to day consistency within the same subject was rather low. Creatinine coefficient was computed per kg of body weight as well as per kg of lean body weight. In both the cases, it has been found to be higher in the males than in the females. Though an important factor, yet body weight of a subject alone could not explain satisfactorily wide variability of creatinine excretion in the present study. The possible influencing factors have been discussed along with clinical implications.

Effect of garlic on normal blood cholesterol level.

The effect of raw garlic on normal blood cholesterol level in males of the age group of 18-35 years was studied. The subjects, who never ingested garlic before, were given 10 g of garlic daily with their diet for two months. Fasting blood samples were investigated in respect of cholesterol before and after two months of garlic intake. Initially the blood cholesterol level ranged between 160-250 mg% which decreased significantly in all the subjects of experimental group after two months of ingestion of garlic. The slight decrease or increase in the blood cholesterol level of control group was not significant. The raw garlic can be advocated for daily ingestion in order to lower one's blood cholesterol level even if it is within normal limits.

Computer based application software for histopathological reporting system.

For better follow up of patient and the immediate retrieval of records, we have developed a computer based application software for histopathological reporting system (HIPRIS). With its help, among others, we can (i) retrieve the biopsy report of a patient from the accession number of the specimen; (ii) find out the number of cases for a particular period as well as can analyse cases by any relevant referral parameter like department, specialty and disease and (iii) find out the time gap between receiving the specimen and reporting of result. Our experience suggests that this system greatly improves the efficiency of the histopathological laboratory.

Smokeless tobacco use: a meta-analysis of risk and attributable mortality estimates for India.

BACKGROUND: Use of smokeless tobacco (SLT) is widely prevalent in India and Indian subcontinent. Cohort and case-control studies in India and elsewhere report excess mortality due to its use. OBJECTIVE: The aim was to estimate the SLT use-attributable deaths in males and females, aged 35 years and older, in India. MATERIALS AND METHODS: Prevalence of SLT use in persons aged 35 years and older was obtained from the Global Adult Tobacco Survey in India and population size and deaths in the relevant age-sex groups were obtained from UN estimates (2010 revision) for 2008. A meta-relative risk (RR) based population attributable fraction was used to estimate attributable deaths in persons aged 35 years and older. A random effects model was used in the meta-analysis on all-cause mortality from SLT use in India including four cohort and one case-control study. The studies included in the meta-analysis were adjusted for smoking, age and education. RESULTS: The prevalence of SLT use in India was 25.2% for men and 24.5% for women aged 35 years and older. RRs for females and males were 1.34 (1.27-1.42) and 1.17 (1.05-1.42), respectively. The number of deaths attributable to SLT use in India is estimated to be 368127 (217,076 women and 151,051 men), with nearly three-fifth (60%) of these deaths occurring among women. CONCLUSION: SLT use caused over 350,000 deaths in India in 2010, and nearly three-fifth of SLT use-attributable deaths were among women in India. This calls for targeted public health intervention focusing on SLT products especially among women.