Dealing with imperfect randomization: Inference for the highscope perry preschool program.

This paper considers the problem of making inferences about the effects of a program on multiple outcomes when the assignment of treatment status is imperfectly randomized. By imperfect randomization we mean that treatment status is reassigned after an initial randomization on the basis of characteristics that may be observed or unobserved by the analyst. We develop a partial identification approach to this problem that makes use of information limiting the extent to which randomization is imperfect to show that it is still possible to make nontrivial inferences about the effects of the program in such settings. We consider a family of null hypotheses in which each null hypothesis specifies that the program has no effect on one of several outcomes of interest. Under weak assumptions, we construct a procedure for testing this family of null hypotheses in a way that controls the familywise error rate - the probability of even one false rejection - in finite samples. We develop our methodology in the context of a reanalysis of the HighScope Perry Preschool program. We find statistically significant effects of the program on a number of different outcomes of interest, including outcomes related to criminal activity for males and females, even after accounting for the imperfectness of the randomization and the multiplicity of null hypotheses.

When Sarah Meets Lawrence: The Effects of Coeducation on Women's College Major Choices.

We leverage variation in the adoption of coeducation by U.S. women's colleges to study how exposure to a mixed-gender collegiate environment affects women's human capital investments. Our event-study analyses of newly collected historical data find a 3.0-3.5 percentage-point (30-33%) decline in the share of women majoring in STEM. While coeducation caused a large influx of male peers and modest increase in male faculty, we find no evidence that it altered the composition of the female student body or other gender-neutral inputs. Extrapolation of our main estimate suggests that coeducational environments explain 36% of the current gender gap in STEM.

Effective access to health care in Mexico.

OBJECTIVES: This paper assesses the impact of effective access on out-of-pocket health payments and catastrophic health expenditure. Effective access cannot be attained unless both health services and financial risk protection are accessible, affordable, and acceptable. Therefore, it represents a key determinant in the transition from fragmented health systems to universal coverage that many low- and middle-income countries face. METHODS: We use a definition of effective access as the utilization of health insurance when available. We conducted a cross-sectional analysis using the 2018 Mexican National Health Survey (ENSANUT) at the household level. The analysis is performed in two stages. The first stage is a multinomial analysis that captures the factor associated with choosing effective access against the alternative of paying privately. The second stage consists of an impact analysis regarding the decision of not choosing effective access in terms of out-of-pocket (OOP) health payments and catastrophic health expenditures (CHE). The analysis corrects for both the decision to buy insurance and the decision to pay for health care. RESULTS: We found that, on average, not choosing effective access increases OOP health payments by around 2300 pesos annually. Medicine payments are the most common factor in this increase. Nevertheless, outpatient and medicines health care are the main drivers of the increase in OOP health payments in all insurance beneficiaries. Not having effective access increases the probability of CHE health expenditures by 2.7 p.p. for the case of Social Security Insurance and 4.0 p.p. for Social Government insurance. Household enrolled in Prospera program for the poor are more likely to choose effective access while having household heads with more education and assets value does the opposite. Diabetes illnesses are associated with a higher probability of effective access. CONCLUSION: Improving effective access is a middle step that cannot be disregarded when seeking universal coverage because OOP health payments and catastrophic outcomes are direct consequences. Public insurance in general, has around 50% effective access which remains a challenge in terms of health services utilization and health public policy design, calling for the need of better coordination across insurance types and pooling mechanisms to increase sustainability of needed health services.

Short-run and Long-run Effects of Peers from Disrupted Families.

I study the short-run and long-run effects of exposure to peers from disrupted families in adolescence. Using the National Longitudinal Study of Adolescent to Adult Health (Add Health) data, I find that girls are mostly unaffected by peers from disrupted families, while boys exposed to more peers from disrupted families exhibit more school problems in adolescence and higher arrest probabilities, less stable jobs and higher probabilities of suffering from financial stress as young adults. These results suggest negative effects on non-cognitive skills but no effect on cognitive skills, as measured by academic performance. The dramatic increase in family disruption in the United States should thus receive more attention, as the intergenerational mobility and inequality consequences could be larger than anticipated as a result of classroom spillovers.

Causal Inference of Social Experiments using Orthogonal Designs.

Orthogonal Arrays are a powerful class of experimental designs that has been widely used to determine efficient arrangements of treatment factors in randomized controlled trials. Despite its popularity, the method is seldom used in social sciences. Social experiments must cope with randomization compromises such as noncompliance that often prevents the use of elaborate designs. We present a novel application of orthogonal designs that addresses the particular challenges arising in social experiments. We characterize the identification of counterfactual variables as a finite mixture problem in which choice incentives, rather than treatment factors, are randomly assigned. We show that the causal inference generated by an orthogonal array of incentives greatly outperforms a traditional design.

Teaching and Incentives: Substitutes or Complements?

Interventions to promote learning are often categorized into supply- and demand-side approaches. In a randomized experiment to promote learning about COVID-19 among Mozambican adults, we study the interaction between a supply and a demand intervention, respectively: teaching via targeted feedback, and providing financial incentives to learners. In theory, teaching and learner-incentives may be substitutes (crowding out one another) or complements (enhancing one another). Experts surveyed in advance predicted a high degree of substitutability between the two treatments. In contrast, we find substantially more complementarity than experts predicted. Combining teaching and incentive treatments raises COVID-19 knowledge test scores by 0.5 standard deviations, though the standalone teaching treatment is the most cost-effective. The complementarity between teaching and incentives persists in the longer run, over nine months post-treatment.

Education, Dietary Intakes and Exercise.

This paper examines the relationship between education and health behaviours, focusing on potential offsetting responses between calories in (i.e. dietary intakes) and calories out (i.e. physical activity). It exploits the 1972 British compulsory schooling law that raised the minimum school leaving age from 15 to 16 to estimate the effects of education on diet and exercise around middle age. Using a regression discontinuity design, the findings suggest that the reform led to a worsening of the quality of the diet, with increases in total calories, fats and animal proteins. However, I find that these changes are partially offset by a discontinuous increase in physical activity. Back-of-the-envelope calculations suggest little effect on the balance of calories. As such, the findings show that focusing on the two components of energy balance provides additional information that is concealed in analyses that only use a measure of obesity.

Do Financial Incentives Aimed at Decreasing Interhousehold Inequality Increase Intrahousehold Inequality?

Research has shown that giving disadvantaged families financial incentives to invest in their children could decrease socioeconomic inequality by enhancing human capital formation. Yet, within the household how are such gains achieved? We use a field experiment to investigate how parents allocate time when they receive financial incentives. We find that incentives increase investment in the target child. But, parents achieve these gains by substituting away from time spent with the child's sibling(s). An unintended consequence is that intrahousehold inequality increases and aggregate gains from the program are overstated when focusing only on target children.

Covid-19 shocks to education supply: how 200,000 U.S. households dealt with the sudden shift to distance learning.

Among the extraordinary shocks to household life caused by the Covid-19 pandemic was the sudden shift to distance learning in K-12 schools. Gone were Monday through Friday routines of school day, extracurricular activities, and evening homework; schools scrambled to launch alternative delivery systems, expecting parents to step in and spend significant amounts of time helping children continue to learn. This study examines the sudden shift to distance learning using data from U.S. Census Bureau's Household Pulse Survey. Conducted weekly from April through July 2020, the survey tracked COVID-related shocks to employment, health, food and housing security, and education in the U.S. population. We use Pulse data on 200,000 households with K-12 children to examine how school systems shifted, how parents stepped up and spent time helping children learn, how parental time inputs varied with parent education, and how education changes intersected with other pandemic shocks, including job loss and food insecurity. We find that parents and children spent significantly more time in learning activities when their schools provided diversified educational inputs, especially live contact time with teachers; live contact hours also facilitated children learning on their own. Given the type of alternative schooling, less educated parents spent no less time helping children than better educated parents, although they faced significantly more problems with computer and internet access. Thus, parents generally tried to help children continue learning in the pandemic, albeit with potentially wide variation in the resources they could supply to mitigate the drop in learning.

Prep School for Poor Kids: The Long-Run Impacts of Head Start on Human Capital and Economic Self-Sufficiency.

This paper evaluates the long-run effects of Head Start using large-scale, restricted administrative data. Using the county rollout of Head Start between 1965 and 1980 and age-eligibility cutoffs for school entry, we find that Head Start generated large increases in adult human capital and economic self-sufficiency, including a 0.65-year increase in schooling, a 2.7 percent increase in high school completion, an 8.5 percent increase in college enrollment, and a 39 percent increase in college completion. These estimates imply sizable, long-term returns to investments in means-tested, public preschool programs.

BD2K Training Coordinating Center's ERuDIte: the Educational Resource Discovery Index for Data Science.

Data science is a field that has developed to enable efficient integration and analysis of increasingly large data sets in many domains. In particular, big data in genetics, neuroimaging, mobile health, and other subfields of biomedical science, promises new insights, but also poses challenges. To address these challenges, the National Institutes of Health launched the Big Data to Knowledge (BD2K) initiative, including a Training Coordinating Center (TCC) tasked with developing a resource for personalized data science training for biomedical researchers. The BD2K TCC web portal is powered by ERuDIte, the Educational Resource Discovery Index, which collects training resources for data science, including online courses, videos of tutorials and research talks, textbooks, and other web-based materials. While the availability of so many potential learning resources is exciting, they are highly heterogeneous in quality, difficulty, format, and topic, making the field intimidating to enter and difficult to navigate. Moreover, data science is rapidly evolving, so there is a constant influx of new materials and concepts. We leverage data science techniques to build ERuDIte itself, using data extraction, data integration, machine learning, information retrieval, and natural language processing to automatically collect, integrate, describe, and organize existing online resources for learning data science.

Technology and educational choices: Evidence from a one-laptop-per-child program.

This paper provides the first causal estimates of the effect of children's access to computers and the internet on educational outcomes in early adulthood, such as schooling and choice of major. I exploit cross-cohort variation in access to technology among primary and middle school students in Uruguay, the first country to implement a nationwide one-laptop-per-child program. Despite a notable increase in computer access, educational attainment has not increased; the schooling gap between private and public school students has persisted, despite closing the technology gap. Among college students, those who had been exposed to the program as children were less likely to enroll in science and technology.

The educational and labor market returns to preschool attendance in Austria.

Preschool attendance is widely recognized as a key ingredient for later socioeconomic success, mothers' labor market participation, and leveling the playing field for children from disadvantaged backgrounds. However, the empirical evidence for these claims is still relatively scarce, particularly in Europe. Using data from the 2011 Austrian European Union Statistics of Income and Living Conditions (EU-SILC), we contribute to this literature by studying the effects of having attended preschool for the adult Austrian population. We find strong and positive effects of preschool attendance on later educational attainment, the probability of working full time, hourly wages, and the probability that the mother is in the labor market. Full time workers at the bottom and the top of the distribution benefit less than those in the middle. Women in particular benefit more in terms of years of schooling and the probability of working full time. Other disadvantaged groups (second generation migrants; people with less educated parents) also often benefit more in terms of education and work.

Evaluation of the Reggio Approach to Early Education.

We evaluate the Reggio Approach using non-experimental data on individuals from the cities of Reggio Emilia, Parma and Padova belonging to one of five age cohorts: ages 50, 40, 30, 18, and 6 as of 2012. The treated were exposed to municipally offered infant-toddler (ages 0-3) and preschool (ages 3-6) programs. The control group either did not receive formal childcare or were exposed to programs offered by the state or religious systems. We exploit the city-cohort structure of the data to estimate treatment effects using three strategies: difference-in-differences, matching, and matched-difference-in-differences. Most positive and significant effects are generated from comparisons of the treated with individuals who did not receive formal childcare. Relative to not receiving formal care, the Reggio Approach significantly boosts outcomes related to employment, socio-emotional skills, high school graduation, election participation, and obesity. Comparisons with individuals exposed to alternative forms of childcare do not yield strong patterns of positive and significant effects. This suggests that differences between the Reggio Approach and other alternatives are not sufficiently large to result in significant differences in outcomes. This interpretation is supported by our survey, which documents increasing similarities in the administrative and pedagogical practices of childcare systems in the three cities over time.

"It's hard to argue with a computer:" Investigating Psychotherapists' Attitudes towards Automated Evaluation.

We present CORE-MI, an automated evaluation and assessment system that provides feedback to mental health counselors on the quality of their care. CORE-MI is the first system of its kind for psychotherapy, and an early example of applied machine-learning in a human service context. In this paper, we describe the CORE-MI system and report on a qualitative evaluation with 21 counselors and trainees. We discuss the applicability of CORE-MI to clinical practice and explore user perceptions of surveillance, workplace misuse, and notions of objectivity, and system reliability that may apply to automated evaluation systems generally.

Unordered Monotonicity.

This paper defines and analyzes a new monotonicity condition for the identification of counterfactuals and treatment effects in unordered discrete choice models with multiple treatments, heterogenous agents and discrete-valued instruments. Unordered monotonicity implies and is implied by additive separability of choice of treatment equations in terms of observed and unobserved variables. These results follow from properties of binary matrices developed in this paper. We investigate conditions under which unordered monotonicity arises as a consequence of choice behavior. We characterize IV estimators of counterfactuals as solutions to discrete mixture problems.

Ubiquitous Event Mining to Enhance Personal Health.

Advances in user interfaces, pattern recognition, and ubiquitous computing continue to pave the way for better navigation towards our health goals. Quantitative methods which can guide us towards our personal health goals will help us optimize our daily life actions, and environmental exposures. Ubiquitous computing is essential for monitoring these factors and actuating timely interventions in all relevant circumstances. We need to combine the events recognized by different ubiquitous systems and derive actionable causal relationships from an event ledger. Understanding of user habits and health should be teleported between applications rather than these systems working in silos, allowing systems to find the optimal guidance medium for required interventions. We propose a method through which applications and devices can enhance the user experience by leveraging event relationships, leading the way to more relevant, useful, and, most importantly, pleasurable health guidance experience.

Do Single-Sex Schools Enhance Students' STEM (Science, Technology, Engineering, and Mathematics) Outcomes?

In many countries, males currently lag behind females in schooling attainment but females are still underrepresented in STEM studies. This pattern has raised renewed interest in the potential of single-sex schools for enhancing STEM outcomes. Utilizing the unique setting in Seoul, where assignment to single-sex or coeducational high schools is random, and with multiple years of administrative data from the national college entrance examinations and a longitudinal survey of high school seniors, we assess causal effects of single-sex schools on students' math test scores and choice of the science-math test. We also assess whether single-sex schools affect students' interests and self-efficacy in math and science, and expectations and actual choices of a STEM college major in university. We find significantly positive effects of all-boys schools consistently across different STEM outcomes but not for girls. We address one possible mechanism by conducting mediation analysis with the proportion of same-gender math teachers.

How much does teacher quality vary across teacher preparation programs? Reanalyses from six states.

At least sixteen US states have taken steps toward holding teacher preparation programs (TPPs) accountable for teacher value-added to student test scores. Yet it is unclear whether teacher quality differences between TPPs are large enough to make an accountability system worthwhile. Several statistical practices can make differences between TPPs appear larger and more significant than they are. We reanalyze TPP evaluations from 6 states-New York, Louisiana, Missouri, Washington, Texas, and Florida-using appropriate methods implemented by our new caterpillar command for Stata. Our results show that teacher quality differences between most TPPs are negligible-.01-0.03 standard deviations in student test scores-even in states where larger differences were reported previously. While ranking all a state's TPPs is not useful, in some states and subjects we can find a single TPP whose teachers are significantly above or below average. Such exceptional TPPs may reward further study.

Designing Contestability: Interaction Design, Machine Learning, and Mental Health.

We describe the design of an automated assessment and training tool for psychotherapists to illustrate challenges with creating interactive machine learning (ML) systems, particularly in contexts where human life, livelihood, and wellbeing are at stake. We explore how existing theories of interaction design and machine learning apply to the psychotherapy context, and identify "contestability" as a new principle for designing systems that evaluate human behavior. Finally, we offer several strategies for making ML systems more accountable to human actors.