Identifying and Minimizing Errors in the Measurement of Early Childhood Development: Lessons Learned from the Cognitive Testing of the ECDI2030.

Challenges in measuring early childhood development (ECD) at scale have been documented, yet little is known about the specific difficulties related to questionnaire design and question interpretation. The purpose of this paper is to discuss the challenges of measuring ECD at scale in the context of household surveys and to show how to overcome them. The paper uses examples from the cognitive interviewing exercises that were conducted as part of the methodological work to develop a measure of ECD outcomes, the ECDI2030. It describes the methodological work carried out to inform the selection and improvement of question items and survey implementation tools as a fundamental step to reduce and mitigate systematic measurement error and improve data quality. The project consisted of a total of five rounds of testing, comprising 191 one-on-one, in-depth cognitive interviews across six countries (Bulgaria, India, Jamaica, Mexico, Uganda, and the USA). Qualitative data analysis methods were used to determine matches and mismatches between intention of items and false positives or false negative answers among subgroups of respondents. Key themes emerged that could potentially lead to systematic measurement error in population-based surveys on ECD: (1) willingness of child to perform task versus ability of child to perform task; (2) performing task versus performing task correctly; (3) identifying letters or numbers versus recognizing letters or numbers; (4) consistently performing task versus correctly performing task; (5) applicability of skills being asked versus observability of skills being asked; and (6) language production versus language comprehension. Through an iterative process of testing and subsequent revision, improvements were made to item wording, response options, and interviewer training instructions. Given the difficulties inherent in population-level data collection in the context of global monitoring, this study's findings confirm the importance of cognitive testing as a crucial step in careful, culturally relevant, and sensitive questionnaire design and as a means to reduce response bias in cross-cultural contexts.

Risks and rewards of using prepaid vs. postpaid incentive checks on a survey of physicians.

BACKGROUND: Survey researchers use monetary incentives as a strategy to motivate physicians' survey participation. Experiments from general population surveys demonstrate that prepaid incentives increase response rates and lower survey administration costs relative to postpaid incentives. Experiments comparing these two incentive strategies have rarely been attempted with physician samples. METHODS: A nationally representative sample of oncologists was recruited to participate in the National Survey of Precision Medicine in Cancer Treatment. To determine the optimal strategy for survey incentives, sample members were randomly assigned to receive a $50 prepaid incentive check or a $50 promised (postpaid) incentive check. Outcome measures for this incentives experiment include cooperation rates, speed of response, check-cashing behavior, and comparison of hypothetical costs for different incentive strategies. RESULTS: Cooperation rates were considerably higher for sample members in the prepaid condition (41%) than in the postpaid condition (29%). Similar differences in cooperation rates were seen for physicians when stratified by region, size of the physician's metropolitan statistical area, specialty, and gender by age. Survey responders in the prepaid condition responded earlier in the field period than those in the postpaid condition, thus requiring fewer contacts. In the prepaid group, 84% of sample members who responded with a completed survey cashed the incentive check and only 6% of nonresponders cashed the check. In the postpaid condition, 72% of survey responders cashed the check; nonresponders were not given a check. The relatively higher cooperation rates and earlier response of the responders in the prepaid condition was associated with a 30% cost savings for the prepaid condition compared to the postpaid incentive condition. CONCLUSIONS: The results of this study suggest that the rewards of offering physicians a prepaid incentive check outweigh the possible risks of nonresponders cashing the check. The relative cost benefit of this strategy is likely to vary depending on the amount of the incentive relative to the costs of additional contact attempts to nonresponders.

Awareness of the Food and Drug Administration's Bad Ad Program and Education Regarding Pharmaceutical Advertising: A National Survey of Prescribers in Ambulatory Care Settings.

The U.S. Food and Drug Administration's Bad Ad program educates health care professionals about false or misleading advertising and marketing and provides a pathway to report suspect materials. To assess familiarity with this program and the extent of training about pharmaceutical marketing, a sample of 2,008 health care professionals, weighted to be nationally representative, responded to an online survey. Approximately equal numbers of primary care physicians, specialists, physician assistants, and nurse practitioners answered questions concerning Bad Ad program awareness and its usefulness, as well as their likelihood of reporting false or misleading advertising, confidence in identifying such advertising, and training about pharmaceutical marketing. Results showed that fewer than a quarter reported any awareness of the Bad Ad program. Nonetheless, a substantial percentage (43%) thought it seemed useful and 50% reported being at least somewhat likely to report false or misleading advertising in the future. Nurse practitioners and physician assistants expressed more openness to the program and reported receiving more training about pharmaceutical marketing. Bad Ad program awareness is low, but opportunity exists to solicit assistance from health care professionals and to help health care professionals recognize false and misleading advertising. Nurse practitioners and physician assistants are perhaps the most likely contributors to the program.