Hierarchical Bayesian non-response models for error rates in forensic black-box studies.

Forensic science plays a critical role in the United States criminal legal system. Historically, however, most feature-based fields of forensic science, including firearms examination and latent print analysis, have not been shown to be scientifically valid. Recently, black-box studies have been proposed as a means of assessing whether these feature-based disciplines are valid, at least in terms of accuracy, reproducibility and repeatability. In these studies, forensic examiners frequently either do not respond to every test item or select an answer equivalent to 'don't know'. Current black-box studies do not account for these high levels of missingness in statistical analyses. Unfortunately, the authors of black-box studies typically do not share the data necessary to meaningfully adjust estimates for the high proportion of missing responses. Borrowing from work in the context of small area estimation, we propose the use of hierarchical Bayesian models that do not require auxiliary data to adjust for non-response. Using these models, we offer the first formal exploration of the impact that missingness is playing in error rate estimations reported in black-box studies. We show that error rates currently reported as low as 0.4% could actually be at least 8.4% in models accounting for non-response where inconclusive decisions are counted as correct, and over 28% when inconclusives are counted as missing responses. These proposed models are not the answer to the missingness problem in black-box studies. But with the release of auxiliary information, they can be the foundation for new methodologies to adjust for missingness in error rate estimations. This article is part of the theme issue 'Bayesian inference: challenges, perspectives, and prospects'.

Managing missing items in the Fagerström Test for Nicotine Dependence: a simulation study.

BACKGROUND: The Fagerström Test for Nicotine Dependence (FTND) is frequently used to assess the level of smokers' nicotine dependence; however, it is unclear how to manage missing items. The aim of this study was to investigate different methods for managing missing items in the FTND. METHODS: We performed a simulation study using data from the Arizona Smokers' Helpline. We randomly sampled with replacement from the complete data to simulate 1000 datasets for each parameter combination of sample size, proportion of missing data, and type of missing data (missing at random and missing not at random). Then for six methods for managing missing items on the FTND (two involving no imputation and four involving single imputation), we assessed the accuracy (via bias) and precision (via bias of standard error) of the total FTND score itself and of the regression coefficient for the total FTND score regressed on a covariate. RESULTS: When using the total FTND score as a descriptive statistic or in analysis for both types of missing data and for all levels of missing data, proration performed the best in terms of accuracy and precision. Proration's accuracy decreased with the amount of missing data; for example, at 9% missing data proration's maximum bias for the mean FTND was only - 0.3%, but at 35% missing data its maximum bias for the mean FTND increased to - 6%. CONCLUSIONS: For managing missing items on the FTND, we recommend proration, because it was found to be accurate and precise, and it is easy to implement. However, because proration becomes less accurate with more missing data, if more than ~ 10% of data are missing, we recommend performing a sensitivity analysis with a different method of managing missing data.

PROMIS Global Health item nonresponse: is it better to impute missing item responses before computing T-scores?

PURPOSE: Item response theory (IRT) scoring provides T-scores for physical and mental health subscales on the Patient-Reported Outcomes Measurement Information System Global Health questionnaire (PROMIS-GH) even when relevant items are skipped. We compared different item- and score-level imputation methods for estimating T-scores to the current scoring method. METHODS: Missing PROMIS-GH items were simulated using a dataset of complete PROMIS-GH scales collected at a single tertiary care center. Four methods were used to estimate T-scores with missing item scores: (1) IRT-based scoring of available items (IRTavail), (2) item-level imputation using predictive mean matching (PMM), (3) item-level imputation using proportional odds logistic regression (POLR), and (4) T-score-level imputation (IMPdirect). Performance was assessed using root mean squared error (RMSE) and mean absolute error (MAE) of T-scores and comparing estimated regression coefficients from the four methods to the complete data model. Different proportions of missingness and sample sizes were examined. RESULTS: IRTavail had lowest RMSE and MAE for mental health T-scores while PMM had lowest RMSE and MAE for physical health T-scores. For both physical and mental health T-scores, regression coefficients estimated from imputation methods were closer to those of the complete data model. CONCLUSIONS: The available item scoring method produced more accurate PROMIS-GH mental but less accurate physical T-scores, compared to imputation methods. Using item-level imputation strategies may result in regression coefficient estimates closer to those of the complete data model when nonresponse rate is high. The choice of method may depend on the application, sample size, and amount of missingness.

Mixing mixed-mode designs in a national health interview survey: a pilot study to assess the impact on the self-administered questionnaire non-response.

BACKGROUND: Many population health surveys consist of a mixed-mode design that includes a face-to-face (F2F) interview followed by a paper-and-pencil (P&P) self-administered questionnaire (SAQ) for the sensitive topics. In order to alleviate the burden of a supplementary P&P questioning after the interview, a mixed-mode SAQ design including a web and P&P option was tested for the Belgian health interview survey. METHODS: A pilot study (n = 266, age 15+) was organized using a mixed-mode SAQ design following the F2F interview. Respondents were invited to complete a web SAQ either immediately after the interview or at a later time. The P&P option was offered in case respondents refused or had previously declared having no computer access, no internet connection or no recent usage of computers. The unit response rate for the web SAQ and the overall unit response rate for the SAQ independent of the mode were evaluated. A logistic regression analysis was conducted to explore the association of socio-demographic characteristics and interviewer effects with the completed SAQ mode. Furthermore, a logistic regression analysis assessed the differential user-friendliness of the SAQ modes. Finally, a logistic multilevel model was used to evaluate the item non-response in the two SAQ modes while controlling for respondents' characteristics. RESULTS: Of the eligible F2F respondents in this study, 76% (107/140) agreed to complete the web SAQ. Yet among those, only 78.5% (84/107) actually did. At the end, the overall (web and P&P) SAQ unit response rate reached 73.5%. In this study older people were less likely to complete the web SAQ. Indications for an interviewer effect were observed as regard the number of web respondents, P&P respondents and respondents who refused to complete the SAQ. The web SAQ scored better in terms of user-friendliness and presented higher item response than the P&P SAQ. CONCLUSIONS: The web SAQ performed better regarding user-friendliness and item response than the P&P SAQ but the overall SAQ unit response rate was low. Therefore, future research is recommended to further assess which type of SAQ design implemented after a F2F interview is the most beneficial to obtain high unit and item response rates.

Handling missing data in an FFQ: multiple imputation and nutrient intake estimates.

OBJECTIVE: We aimed to examine missing data in FFQ and to assess the effects on estimating dietary intake by comparing between multiple imputation and zero imputation. DESIGN: We used data from the Okazaki Japan Multi-Institutional Collaborative Cohort (J-MICC) study. A self-administered questionnaire including an FFQ was implemented at baseline (FFQ1) and 5-year follow-up (FFQ2). Missing values in FFQ2 were replaced by corresponding FFQ1 values, multiple imputation and zero imputation. SETTING: A methodological sub-study of the Okazaki J-MICC study.ParticipantsOf a total of 7585 men and women aged 35-79 years at baseline, we analysed data for 5120 participants who answered all items in FFQ1 and at least 50% of items in FFQ2. RESULTS: Among 5120 participants, the proportion of missing data was 3·7%. The increasing number of missing food items in FFQ2 varied with personal characteristics. Missing food items not eaten often in FFQ2 were likely to represent zero intake in FFQ1. Most food items showed that the observed proportion of zero intake was likely to be similar to the probability that the missing value is zero intake. Compared with FFQ1 values, multiple imputation had smaller differences of total energy and nutrient estimates, except for alcohol, than zero imputation. CONCLUSIONS: Our results indicate that missing values due to zero intake, namely missing not at random, in FFQ can be predicted reasonably well from observed data. Multiple imputation performed better than zero imputation for most nutrients and may be applied to FFQ data when missing is low.

Effects of age and cognitive function on data quality of standardized surveys in nursing home populations.

BACKGROUND: Data quality is of special concern when it comes to survey research in nursing homes. Very little is known about specifics of cognitively impaired elderly in responding to survey questions. This study examines effects of cognitive impairment, age, gender, and interview duration on the data quality in a sample of 659 nursing home residents (NHR). METHODS: Within a cross-sectional design, survey methodology was used to evaluate the pain situation in 13 nursing homes. Residents were stratified into NHR with no/mild (Mini-Mental State Examination MMSE: 18-30) and NHR with moderate (MMSE: 10-17) cognitive impairment. Data quality is measured by item nonresponse (INR). Correlation analyses, ANCOVA, linear and logistic regression models are applied. RESULTS: Neither interview duration nor gender have effects on item nonresponse. Age accounts for higher INR (ß = 0.12, p < 0.001). Cognitive impairment strongly predicts INR (ß = - 0.40, p < 0.001). INR significantly differs between NHR with no/mild (3.98%) and moderate cognitive impairment (11.85%). The likelihood of INR > 5% for residents with moderate cognitive impairment is 3.8-times (p < 0.001) of that for those with no/mild impairment. CONCLUSIONS: Surveys are adequate for residents with no/mild cognitive impairment but data quality is threatened in residents with moderate impairments. Precision and validity of responses from NHR with progressed cognitive impairment are potentially limited and results may be biased. The results clearly do support the need for a multidisciplinary 'general theory' of the question-/answer-process which has to be also inclusive for cognitively impaired elderly persons.

Screening for Pregnancy Status in a Population-Based Sample: Characteristics Associated with Item Nonresponse.

OBJECTIVES: Population-based recruitment of a cohort of women who are currently pregnant or who may become pregnant in a given timeframe presents challenges unique to identifying pregnancy status or the likelihood of future pregnancy. Little is known about the performance of individual eligibility items on pregnancy screeners although they are critical to participant recruitment. This paper examined the patterns and respondent characteristics of key pregnancy screener items used in a large national study. METHODS: Cross-sectional analyses were conducted. Descriptive statistics and multivariable logistic regression models were used to examine nonresponse patterns to three questions (currently pregnant, trying to get pregnant and able to get pregnant). The questions were asked of 50,529 women in 17 locations across the US, as part of eligibility screening for the National Children's Study Vanguard Study household-based recruitment. RESULTS: Most respondents were willing to provide information about current pregnancy, trying, and able to get pregnant: 99.3% of respondents answered all three questions and 97.4% provided meaningful answers. Nonresponse ranged from 0.3 to 2.5% for individual items. Multivariable logistic regression results identified small but statistically significant differences in nonresponse by respondent age, marital status, race/ethnicity-language, and household-based recruitment group. CONCLUSIONS FOR PRACTICE: The high levels of response to pregnancy-related items are impressive considering that the eligibility questions were fairly sensitive, were administered at households, and were not part of a respondent-initiated encounter.

Missing data in FFQs: making assumptions about item non-response.

OBJECTIVE: FFQs are a popular method of capturing dietary information in epidemiological studies and may be used to derive dietary exposures such as nutrient intake or overall dietary patterns and diet quality. As FFQs can involve large numbers of questions, participants may fail to respond to all questions, leaving researchers to decide how to deal with missing data when deriving intake measures. The aim of the present commentary is to discuss the current practice for dealing with item non-response in FFQs and to propose a research agenda for reporting and handling missing data in FFQs. RESULTS: Single imputation techniques, such as zero imputation (assuming no consumption of the item) or mean imputation, are commonly used to deal with item non-response in FFQs. However, single imputation methods make strong assumptions about the missing data mechanism and do not reflect the uncertainty created by the missing data. This can lead to incorrect inference about associations between diet and health outcomes. Although the use of multiple imputation methods in epidemiology has increased, these have seldom been used in the field of nutritional epidemiology to address missing data in FFQs. We discuss methods for dealing with item non-response in FFQs, highlighting the assumptions made under each approach. CONCLUSIONS: Researchers analysing FFQs should ensure that missing data are handled appropriately and clearly report how missing data were treated in analyses. Simulation studies are required to enable systematic evaluation of the utility of various methods for handling item non-response in FFQs under different assumptions about the missing data mechanism.

The effect of sociodemographic (mis)match between interviewers and respondents on unit and item nonresponse in Belgium.

Interviewer characteristics affect nonresponse and measurement errors in face-to-face surveys. Some studies have shown that mismatched sociodemographic characteristics - for example gender - affect people's behavior when interacting with an interviewer at the door and during the survey interview, resulting in more nonresponse. We investigate the effect of sociodemographic (mis)matching on nonresponse in two successive rounds of the European Social Survey in Belgium. As such, we replicate the analyses of the effect of (mis)matching gender and age on unit nonresponse on the one hand, and of gender, age and education level (mis)matching on item nonresponse on the other hand. Recurring effects of sociodemographic (mis)match are found for both unit and item nonresponse.

A Test of Web and Mail Mode Effects in a Financially Sensitive Survey of Older Americans.

This study leverages a randomized experimental design of a mixed-mode mail- and web-based survey to examine mode effects separately from sample selectivity issues. Using data from the Cognitive Economics Study, which contains some sensitive financial questions, we analyze two sets of questions: fixed-choice questions posed nearly identically across mode, and dollar-value questions that exploit features available only on web mode. Focusing on differences in item nonresponse and response distributions, our results indicate that, in contrast to mail mode, web mode surveys display lower item nonresponse for all questions. While respondents appear to prefer providing financial information in ranges, use of reminder screens on the web version yields greater use of exact values without large sacrifices in item response. Still, response distributions for all questions are similar across mode, suggesting that data on sensitive financial questions collected from the two modes can be pooled.

Multiply Robust Bootstrap Variance Estimation in the Presence of Singly Imputed Survey Data.

Item nonresponse in surveys is usually dealt with through single imputation. It is well known that treating the imputed values as if they were observed values may lead to serious underestimation of the variance of point estimators. In this article, we propose three pseudo-population bootstrap schemes for estimating the variance of imputed estimators obtained after applying a multiply robust imputation procedure. The proposed procedures can handle large sampling fractions and enjoy the multiple robustness property. Results from a simulation study suggest that the proposed methods perform well in terms of relative bias and coverage probability, for both population totals and quantiles.

Who Can You Count On? Understanding The Determinants of Reliability.

Using reinterview data from the PATH Reliability and Validity (PATH-RV) study, we examine the characteristics of questions and respondents that predict the reliability of the answers. In the PATH-RV study, 524 respondents completed an interview twice, five to twenty-four days apart. We coded a number of question characteristics and used them to predict the gross discrepancy rates (GDRs) and kappas for each question. We also investigated respondent characteristics associated with reliability. Finally, we fitted cross-classified models that simultaneously examined a range of respondent and question characteristics. Although the different models yielded somewhat different conclusions, in general factual questions (especially demographic questions), shorter questions, questions that did not use scales, those with fewer response options, and those that asked about a noncentral topic produced more reliable answers than attitudinal questions, longer questions, questions using ordinal scales, those with more response options, and those asking about a central topic. One surprising finding was that items raising potential social desirability concerns yielded more reliable answers than items that did not raise such concerns. The respondent-level models and cross-classified models indicated that five adult respondent characteristics were associated with giving the same answer in both interviews-education, the Big Five trait of conscientiousness, tobacco use, sex, and income. Hispanic youths and non-Hispanic black youths were less likely to give the same answer in both interviews. The cross-classified model also found that more words were associated with less reliable answers. The results are mostly consistent with earlier findings but are nonetheless important because they are much less model-dependent than the earlier work. In addition, this study is the first to incorporate such personality traits as needed for cognition and the Big Five personality factors and to examine the relationships among reliability, item nonresponse, and response latency.

What They Expect Is What You Get: The Role of Interviewer Expectations in Nonresponse to Income and Asset Questions.

Personal income and assets are sensitive topics to discuss. This phenomenon is reflected in high rates of nonresponse to financial questions in surveys. In face-to-face surveys, item nonresponse is influenced by interviewers. Although interviewers are trained to conduct standardized interviews, some obtain a higher number of item nonresponses than others. This study examines interviewer effects on nonresponse to questions about household income, bank balances, and interest and dividend income in the Survey of Health, Ageing and Retirement in Europe (SHARE). It first investigates the extent to which interviewers affect nonresponse to income and asset questions and second whether interviewers' prior expectations regarding respondents' likelihood to provide information about their income predict actual nonresponse rates. Results of multilevel modeling show that interviewer influence on nonresponse to the income and asset questions was significant at the five percent level. In addition, interviewer expectations were significantly correlated with "don't know" responses and "refusals." These results indicate that interviewer expectations matter in the context of income and asset questions and that survey practitioners should take this into account when designing interviewer training.

Why Patients with Chronic Disease Keep Silent? Analysis of Item Nonresponse in Rural China.

BACKGROUND: This study aimed to identify the characteristics of item nonresponse and examine the factors affecting the refusal or failure to respond of patients with chronic disease in rural China. METHODS: A cross-sectional survey data from patients with chronic disease from rural China were analyzed. A total of 1,099 patients were enrolled. Chi-square test and cumulative logistic regression determined the predictors of having item nonresponse. RESULTS: The respondents in central provinces (OR = 2.311, 95%CI = 0.532∼1.144, P < 0.001) with over eight household members (OR = 0.067, 95%CI = -1.632∼-0.349, P = 0.002), multiple chronic diseases (OR = 0.301, 95%CI = -1.673∼-0.727, P < 0.001), and low health knowledge level (OR = 2.112, 95%CI = 0.405∼1.090, P < 0.001) had more item nonresponse numbers. Compared with the participants with high school education level and above, the item nonresponse number seemed to increase when the participants were illiterate (OR = 2.159, 95%CI = 0.254∼1.285, P = 0.003), had primary school education (OR = 2.161, 95%CI = 0.249∼1.294, P = 0.004) and junior school education (OR = 2.070, 95%CI = 0.160∼1.296, P = 0.012). CONCLUSION: This study indicates the influencing factors of the item nonresponse in survey of patients with chronic disease in rural China. This study contributes to investigation practice and highlights that health institutions should improve the quality of follow-up services. Moreover, the government should pay more attention to the care of vulnerable groups, especially patients with chronic disease in rural areas.

Asking about Sexual Identity on the National Health Interview Survey: Does Mode Matter?

Privacy, achieved through self-administered modes of interviewing, has long been assumed to be a necessary prerequisite for obtaining unbiased responses to sexual identity questions due to their potentially sensitive nature. This study uses data collected as part of a split-ballot field test embedded in the National Health Interview Survey (NHIS) to examine the association between survey mode (computer-assisted personal interviewing (CAPI) versus audio computer-assisted self-interviewing (ACASI)) and sexual minority identity reporting. Bivariate and multivariate quantitative analyses tested for differences in sexual minority identity reporting and non-response by survey mode, as well as for moderation of such differences by sociodemographic characteristics and interviewing environment. No significant main effects of interview mode on sexual minority identity reporting or nonresponse were found. Two significant mode effects emerged in subgroup analyses of sexual minority status out of 35 comparisons, and one significant mode effect emerged in subgroup analyses of item nonresponse. We conclude that asking the NHIS sexual identity question using CAPI does not result in estimates that differ systematically and meaningfully from those produced using ACASI.

Item nonresponse on the Myeloproliferative Neoplasms Symptom Assessment Form (MPN-SAF): a comparison of missing data strategies.

Administering questionnaires to patients is an efficient and effective method for assessing patients' symptoms. However, item nonresponse (skipped questions) potentially compromises the utility of these questionnaires. Using an international sample of 2,067 patients with myeloproliferative neoplasms, we evaluated the impact of item nonresponse on scoring of the Myeloproliferative Neoplasms Symptom Assessment Form Total Symptom Score (MPN-SAF TSS or MPN-10). We characterized item nonresponse on the MPN-10 and compared strategies for addressing item nonresponse (available-case analysis, proration, and multiple imputation) on the MPN-10 (multi-symptom assessment) and Brief Fatigue Inventory (BFI; single-symptom assessment). Characteristics of multi-symptom assessments would be expected to adversely affect proration, yet proration and multiple imputation provided very similar results for both the MPN-10 and BFI. This is likely because the MPN-10 item missing data rates were low, consistent with prior clinic- and internet-based studies. These results support the published scoring method for the MPN-10 (proration).

On the Importance of the Speed-Ability Trade-Off When Dealing With Not Reached Items.

In many applications of high- and low-stakes ability tests, a non-negligible amount of respondents may fail to reach the end of the test within the specified time limit. Since for respondents that ran out of time some item responses will be missing, this raises the question of how to best deal with these missing responses for the purpose of obtaining an optimal assessment of ability. Commonly, researchers consider three general solutions: ignore the missing responses, treat them as being incorrect, or treat the responses as missing but model the missingness mechanism. This paper approaches the issue of dealing with not reached items from a measurement perspective, and considers the question what the operationalization of ability should be in maximum performance tests that work with effective time limits. We argue that the target ability that the test attempts to measure is maximum performance when operating at the test-indicated speed, and that the test instructions should be taken to imply that respondents should operate at this target speed. The phenomenon of the speed-ability trade-off informs us that the ability that is measured by the test will depend on this target speed, as different speed levels will result in different levels of performance on the same set of items. Crucially, since respondents with not reached items worked at a speed level lower than this target speed, the level of ability that they have been able to display on the items that they did reach is higher than the level of ability that they would have displayed if they had worked at the target speed (i.e., higher than their level on the target ability). Thus, statistical methods that attempt to obtain unbiased estimates of the ability as displayed on the items that were reached will result in biased estimates of the target ability. The practical implications are studied in a simulation study where different methods of dealing with not reached items are contrasted, which shows that current methods result in biased estimates of target ability when a speed-ability trade-off is present. The paper concludes with a discussion of ways in which the issue can be resolved.

Tailored Explanation: A Strategy to Minimize Nonresponse in Demographic Items Among Low-income Racial and Ethnic Minorities.

There is a high rate of nonresponse for demographic items in survey research, particularly for racial and ethnic minority respondents. This present study examined whether providing an explanation to racial and ethnic minority respondents prior to asking a set of demographic questions would increase respondents' motivation to reduce nonresponse to gender, income, age, and race items. Using a cross-sectional, randomized comparison design, 99 respondents were randomly assigned to two groups. Group 1 did not receive an explanation for asking the demographic questions. Group 2 received an explanation designed to be relevant and meaningful to them concerning the significance and potential use of demographic information for racial and ethnic minority populations. A proportional difference test was used to calculate the differences in the proportion of respondents' motivation to complete demographic survey items between the groups. A proportional difference effect size (Cohen's h effect size) was used to determine the magnitude of difference between the two groups. Over 50% of respondents were African Americans. While none of the item non-responses for both groups is statistically significant in terms of proportional differences, there is small (Cohen's h= 0.184) to moderate (Cohen's h=0.342) effect in reducing demographic item non-response when an explanation was provided to respondents. Specifically, adding an explanation made the biggest improvement in reporting income. The study findings support the importance of providing participants with an explanation that is relevant and meaningful to increase motivation and thereby minimize item nonresponse.

Semiparametric fractional imputation using empirical likelihood in survey sampling.

The empirical likelihood method is a powerful tool for incorporating moment conditions in statistical inference. We propose a novel application of the empirical likelihood for handling item nonresponse in survey sampling. The proposed method takes the form of fractional imputation (Kim, 2011) but it does not require parametric model assumptions. Instead, only the first moment condition based on a regression model is assumed and the empirical likelihood method is applied to the observed residuals to get the fractional weights. The resulting semiparametric fractional imputation provides [Formula: see text]-consistent estimates for various parameters. Variance estimation is implemented using a jackknife method. Two limited simulation studies are presented to compare several imputation estimators.

Multiply robust imputation procedures for zero-inflated distributions in surveys.

Item nonresponse in surveys is usually treated by some form of single imputation. In practice, the survey variable subject to missing values may exhibit a large number of zero-valued observations. In this paper, we propose multiply robust imputation procedures for treating this type of variable. Our procedures may be based on multiple imputation models and/or multiple nonresponse models. An imputation procedure is said to be multiply robust if the resulting estimator is consistent when all models but one are misspecified. The variance of the imputed estimators is estimated through a generalized jackknife variance estimation procedure. Results from a simulation study suggest that the proposed procedures perform well in terms of bias, efficiency and coverage rate.