EXAM

An experimental design problem consists of:

•Experimental subjects $$ i=1,\dots ,n$$.

•Experimental treatments $$ t_0,t_1,\dots ,t_m$$, where $$ t_0$$ is a control.

•Each subject $$ i$$’s preference or WTP $$ w_{it}\in \mathbb{R}$$ for treatment $$ t$$, where $$ w_{it}\ge w_{i{t}^{\prime }}$$ means subject $$ i$$ weakly prefers treatment $$ t$$ over $$ t^{\prime }$$.

•Each treatment $$ t$$’s predicted treatment effect $$ e_{ti}\in \mathbb{R}$$ for subject $$ i$$, where $$ e_{ti}\ge e_{t^{\prime }i}$$ means treatment $$ t$$ is predicted to have a weakly better effect than $$ t^{\prime }$$ for subject $$ i$$. When multiple outcomes matter, $$ e_{ti}$$ can be set to the predicted effect on a known function of these outcomes.

I assume $$ e_{ti}$$ and $$ w_{it}$$ to be deterministic for simplicity. Without loss of generality, I normalize $$ e_{ti}$$ and $$ w_{it}$$ by assuming $$ e_{t_{0}i}=w_{i{t}_0}=0$$ for every subject $$ i$$.

An experimental design specifies treatment-assignment probabilities $$ \left(p_{it}\right)$$, where $$ p_{it}$$ is the probability that subject $$ i$$ is assigned to treatment $$ t$$ under the experimental design. The benchmark design is the standard RCT, which assigns each subject $$ i$$ to each treatment $$ t$$ with the impersonal treatment-assignment probability $$ p_t^{RCT}$$, assumed to be written as $$ p_t^{RCT}=c_t/n$$ for some natural number $$ c_t<n$$. $$ c_t$$ is the quasi-capacity or supply of treatment $$ t$$. The vast majority of clinical trials use RCT.

This paper investigates welfare enhancement with a design that I call EXAM. The steps for implementing EXAM are as follows.

1)Obtain predicted effects $$ e_{ti}$$ if possible and relevant, as detailed in SI Appendix, section 1.B.

2)Obtain WTP $$ w_{it}$$ if possible and relevant, as described in SI Appendix, section 1.B.

3)Apply the following definition to the data from steps 1 and 2, producing EXAM’s assignment probabilities $$ p_{it}^{*}\left(ϵ\right)$$.

Definition 1 (EXAM):

In the experimenter’s computer, distribute any common artificial budget $$ b>0$$ to every subject. Find any price-discriminated competitive equilibrium, i.e., any treatment-assignment probabilities $$ \left(p_{it}^{*}\right)$$ and their prices $$ {\pi }_{te}$$ with the following properties:

•Effectiveness-discriminated treatment pricing: There exist $$ \alpha <0$$ and $$ {\beta }_t\in \mathbb{R}$$ for each treatment $$ t$$ such that the price of a unit of assignment probability to $$ t$$ for subjects with $$ e_{ti}=e\in \mathbb{R}$$ is

${\pi }_{te}=\alpha e+{\beta }_t.$

This price is decreasing in treatment-effect prediction $$ e$$, in order to assign each treatment with a higher probability to subjects who benefit more from it.

•Subject utility maximization: For each subject $$ i$$,

${\left(p_{it}^{*}\right)}_t\in \arg \hspace{0.17em}{max}_{p_i\in P}{\sum }_t{p}_{it}{w}_{it}\text{s.t.}{\sum }_t{p}_{it}{\pi }_{t{e}_{ti}}\le b,$

where $$ p_i\equiv {\left(p_{it}\right)}_t$$ and $$ P\equiv \left\{p_i\in {\mathbb{R}}^{m+1}|{\sum }_{t=t_0}^{t_m}{p}_{it}=1\text{and}|p_{it}|\le p\right\}$$, where $$ p$$ is a large enough number. $$ {\pi }_{t{e}_{ti}}$$ is the price of a unit of the assignment probability to treatment $$ t$$ for subject $$ i$$. EXAM breaks ties or indifferences so that every subject $$ i$$’s $$ p_i^{*}$$ solves the above problem with the minimum expenditure $$ {\sum }_t{p}_{it}{\pi }_{t{e}_{ti}}$$, while $$ {\left(p_{it}^{*}\right)}_t={\left(p_{jt}^{*}\right)}_t$$ for any subjects $$ i$$ and $$ j$$ with $$ w_i=w_j$$ and $$ e_i=e_j$$, where $$ w_i\equiv \left(w_{i{t}_1},\dots ,w_{i{t}_m}\right)$$ and $$ e_i\equiv \left(e_{t_{1}i},\dots ,e_{t_{m}i}\right)$$.

•Capacity constraints: $$ {\sum }_i{p}_{it}^{*}\le c_t$$ for every treatment $$ t=t_1,\dots ,t_m$$ and $$ {\sum }_i{p}_{it}^{*}<c_t$$ only if $$ {\pi }_{t{e}_{ti}}\le 0$$ for every $$ i$$.

Let $$ ϵ$$ be a nonnegative number such that the experimenter would like the assignment probabilities to be always within $$ \left[ϵ,1-ϵ\right]$$. Take any $$ ϵ\in \left[0,\overline{ϵ}\right]$$ as given, where $$ \overline{ϵ}\equiv \underset{t}{min}{p}_t^{RCT}$$ is the largest possible value of $$ ϵ$$. I define EXAM’s treatment-assignment probabilities as

A few remarks are in order. First, among the above steps, subjects only need to report their WTP $$ w_{it}$$. The remaining parts are run by the experimenter. Second, it is possible to let different subjects have different budgets. I make $$ b$$ the same for every subject, so that EXAM has the equality property that no subject strictly prefers anybody else’s treatment-assignment probabilities over her own. Finally, $$ \alpha ,{\beta }_t$$ and the resulting $$ p_{it}^{*}\left(ϵ\right)$$ are the equilibrium objects to be found by the experimenter, so as to satisfy the equilibrium constraints. SI Appendix, section 3.B provides an algorithm that finds equilibrium values of $$ \alpha $$ and $$ {\beta }_t$$ by adjusting their values so as to decrease excess demand or supply.

Information

Despite the welfare merit, EXAM also lets the experimenter estimate treatment effects as unbiasedly as they would do in RCTs. To spell it out, here, I switch back to any given finite problem with fixed WTP and predicted effects.

Suppose the experimenter is interested in the causal effect of each treatment on an outcome $$ Y_i$$. Let $$ Y_i\left(t\right)$$ denote subject $$ i$$’s potential outcome that would be observed if subject $$ i$$ receives treatment $$ t$$. Let $$ D_{it}$$ be the indicator that subject $$ i$$ is ex post assigned to treatment $$ t$$. The observed outcome is written as $$ Y_i={\sum }_t{D}_{it}{Y}_i\left(t\right)$$. While $$ Y_i\left(t\right)$$ is assumed to be fixed, $$ D_{it}$$ and $$ Y_i$$ are random variables, the distributions of which depend on the experimenter’s choice of an experimental design. Let $$ Y\equiv \left(Y_i\right)$$, $$ D_i\equiv {\left(D_{it}\right)}_t$$, and $$ D\equiv \left(D_i\right)$$.

The experimenter would like to learn any parameter of interest $$ \theta $$ of the distribution of potential outcomes $$ Y_i\left(t\right)$$’s, many of which are unobservable. Formally, $$ \theta $$ is any mapping $$ \theta :{\mathbb{R}}^{n\times \left(m+1\right)}\to \mathbb{R}$$ that maps each possible value of $$ \left(Y_i\left(t\right)\right)$$ into the corresponding value of the parameter. For example, $$ \theta $$ may be the ATE ($$ {\mathrm{A}\mathrm{T}\mathrm{E}}_t$$) of treatment $$ t$$ over control $$ t_0$$, $$ \frac{{\sum }_{i=1}^n\left(Y_i\left(t\right)-Y_i\left(t_0\right)\right)}{n}$$. The experimenter estimates $$ \theta $$ with an estimator $$ \widehat{\theta }\left(Y,D\right)$$, a function only of observed outcomes and treatment assignments. I say parameter $$ \theta $$ is estimable without bias with experimental design $$ p\equiv \left(p_{it}\right)$$ if there exists a “simple” estimator $$ \widehat{\theta }\left(Y,D\right)$$ (in the sense defined in SI Appendix, section 1.C) such that $$ E\left(\widehat{\theta }\left(Y,D\right)|\left(p_{it}\right)\right)=\theta $$, where $$ E\left(\cdot |\left(p_{it}\right)\right)$$ is expectation with respect to the distribution of $$ D_{it}$$ induced by experimental design $$ \left(p_{it}\right)$$.

EXAM turns out to be as informative as RCT in terms of the set of parameters estimable without bias with each experimental design.

Corollary 1.

The ATE and the treatment effect on the treated are estimable without bias with EXAM.

I use the ATE to illustrate the intuition for and implementation of Proposition 4 and Corollary 1. Why is ATE estimable without bias with EXAM? The reason is that once it is constructed, EXAM is a particular stratified experiment stratified on observable WTP and predicted effects. EXAM, therefore, produces treatment assignment that is independent from (unconfounded by) potential outcomes conditional on predicted effects and WTP, which are observable to the experimenter:

Conditional independence (3) implies that the same conditional independence holds conditional on the assignment probability $$ p_i^{*}\left(ϵ\right)\equiv {\left(p_{it}^{*}\left(ϵ\right)\right)}_t$$ (ref. 14, section 12.3), which is, again, known to the econometrician:

This conditionally independent treatment assignment allows the experimenter to unbiasedly estimate the conditional ATEs of each $$ t$$ over $$ t_0$$ conditional on observable propensity scores $$ p_i^{*}\left(ϵ\right)$$,

$\frac{{\sum }_{i=1}^{n}1\left\{p_i^{*}\left(ϵ\right)=p\right\}\left(Y_i\left(t\right)-Y_i\left(t_0\right)\right)}{{\sum }_{i=1}^{n}1\left\{p_i^{*}\left(ϵ\right)=p\right\}},$

which I denote by $$ CAT{E}_{pt}$$. By summing up conditional effects, the experimenter can back out the (unconditional) ATE, the single most important causal object identified and estimated by RCTs. That is, with weights $$ {\delta }_p\equiv {\sum }_{i=1}^{n}1\left\{p_i^{*}\left(ϵ\right)=p\right\}/n$$, I use $$ CAT{E}_{pt}$$’s to get ATE as follows: $$ {\sum }_p{\delta }_{p}CAT{E}_{pt}=AT{E}_t$$. The above estimability argument motivates a strategy to estimate ATE with EXAM’s data. As a warm-up, we focus on $$ \left\{i|p_i^{*}\left(ϵ\right)=p\right\}$$, the subpopulation of subjects with propensity vector $$ p$$, and consider this regression on the subpopulation:

$Y_i={\alpha }_p+\sum _{t=t_1}^{t_m}{\beta }_{pt}{D}_{it}+{ϵ}_i.$

By conditional independence (4), the ordinary least-squares (OLS) estimate $$ {\widehat{\beta }}_{pt}$$ from this regression is unbiased for $$ CAT{E}_{pt}$$ for each treatment $$ t\ne t_0$$. I then aggregate the resulting estimates $$ {\widehat{\beta }}_{pt}$$’s into $$ {\sum }_p{\delta }_p{\widehat{\beta }}_{pt}$$, which I denote by $$ {\widehat{\beta }}_t^{*}$$. This estimator $$ {\widehat{\beta }}_t^{*}$$ unbiasedly estimates the ATE with its variance in an analytical form, as shown in SI Appendix.

Alternatively, empirical researchers may prefer a single regression:

$Y_i=a+\sum _{t=t_1}^{t_m}{b}_t{D}_{it}+\sum _{t=t_1}^{t_m}{c}_t{p}_{it}^{*}\left(ϵ\right)+e_i,$ [3]

producing an alternative estimator $$ {\widehat{b}}_t^{*}$$. As verified in SI Appendix, $$ {\widehat{b}}_t^{*}$$ is an unbiased estimator of a differently weighted treatment effect:

$E\left({\widehat{b}}_t^{*}|p^{*}\left(ϵ\right)\right)=\frac{\sum _p{\lambda }_{pt}CAT{E}_{pt}}{\sum _p{\lambda }_{pt}}\text{with weights}{\lambda }_{pt}\equiv {\delta }_p{p}_t\left(1-p_t\right).$ [4]

Estimators like $$ {\widehat{b}}_t^{*}$$ and $$ {\widehat{\beta }}_t^{*}$$ allow the experimenter to unbiasedly estimate key causal effects with EXAM.

Empirical Application

My empirical test bed for EXAM is an application to a spring-protection experiment in Kenya. Waterborne diseases, especially diarrhea, remain the second leading cause of death among children, comprising about 17% of child deaths under age five (about 1.5 million deaths each year). The only quantitative United Nations Millennium Development Goal is in terms of “the proportion of the population without sustainable access to safe drinking water and basic sanitation,” such as protected springs. Yet, there is controversy about its health impacts. Experts argue that improving source-water quality may only have limited effects, since, for example, water is likely recontaminated in transport and storage.

This controversy motivated researchers to analyze randomized spring protection in Kenya (15). This experiment randomly selected springs to receive protection from the universe of 200 unprotected springs. The experimenter selected and followed a representative sample of about 1,500 households that regularly used some of the 200 springs before the experiment; these households are the experimental subjects. The researchers found that diarrhea among children in treatment households fell by about a quarter of the baseline level. I call this real experiment the “original experiment” and distinguish it from EXAM and RCT as formal concepts in my model.

I consolidate the original experimental data and my methodological framework to empirically evaluate EXAM. Applying the language and notation of my model, experimental subjects are households in the original experiment’s sample. The protection of the spring each household uses at baseline is a single treatment $$ t_1$$, while no protection is the control $$ t_0$$. Each household $$ i$$’s WTP for better water access $$ t_1$$ is denoted by $$ w_{i{t}_1}$$, which I estimate in SI Appendix, section 3A and Table S2. I also estimate the heterogeneous treatment effect $$ e_{t_{1}i}$$ of spring protection $$ t_1$$ on household $$ i$$’s child diarrhea outcome (SI Appendix, Table S1 and Fig. S1).

Given the estimates, imagine somebody is planning a new experiment to further investigate the same spring-protection treatment. What experimental design should she use? Specifically, which is better between RCT and EXAM? My approach is to use the estimated WTP $$ {ŵ}_{i{t}_1}$$ and predicted effects $$ {ê}_{t_{1}i}$$ to simulate EXAM and compare EXAM with RCT in terms of welfare, information, and incentive properties. I fix the set of subjects and treatments as in the original experiment. That is, there are 1,540 households as subjects to be assigned either to the water-source-protection treatment $$ t_1$$ or the control $$ t_0$$. Set the treatment capacity $$ c_{t_1}$$ to be the number of households assigned to the treatment $$ t_1$$ in the original experiment. I set the bound parameter $$ ϵ$$ to be 0.2. I fix predicted effects $$ e_{t_{1}i}$$ to their point estimate $$ {ê}_{t_{1}i}$$.

I start with evaluating EXAM’s welfare performance. I use EXAM’s treatment-assignment probabilities $$ p_{i{t}_1}^{*}\left(ϵ\right)$$ to calculate two welfare measures for each household $$ i$$:

I find that EXAM improves on RCT in terms of the welfare measures $$ w_i^{*}$$ and $$ e_i^{*}$$, a result reported in Fig. 1. The mean of average WTP $$ w_i^{*}$$ for assigned treatments is about 89% higher under EXAM than it is under RCT. Another interpretation of this WTP improvement is about 37% of the average WTP for the treatment. Similarly, EXAM improves the mean of $$ e_i^{*}$$ by about 0.8% absolute reduction in child diarrhea (equivalently, 42% reduction relative to RCT’s level). This predicted effect benefit amounts to about 17% of the ATE of the spring protection found by the original experiment.

Fig. 1.

EXAM vs RCT: Welfare. This figure shows the distribution of average subject welfare over 1,000 bootstrap simulations under each experimental design. A shows the average WTP for assigned treatments $$ w_i^{*}$$, where WTP is measured by the equivalent number of workdays, as described in SI Appendix, section 3.A. B shows the average predicted effects of assigned treatments $$ e_i^{*}$$. A dotted line indicates the distribution of each welfare measure for RCT, while a solid line indicates that for EXAM. Each vertical line represents mean. Kolmogorov–Smirnov tests find the EXAM and RCT distributions to be significantly different both for $$ w_i^{*}$$ and $$ e_i^{*}$$.

Data from EXAM also allow me to obtain more or less the same conclusion about treatment effects as RCT. To see this, I augment the above counterfactual simulation with ATE estimation as follows: I first simulate $$ w_{i{t}_1}$$ and run EXAM to get treatment-assignment probabilities $$ p_{it}^{*}\left(ϵ\right)$$. I use $$ p_{it}^{*}\left(ϵ\right)$$ to draw a final deterministic treatment assignment, denoted by a binary indicator $$ D_i$$ indicating that $$ i$$ is ex post assigned to $$ t_1$$. I then simulate counterfactual or predicted outcome $$ Y_i$$ under $$ D_i$$ by simulating the treatment-effect model I estimate in SI Appendix, section 3.A. Finally, I use the above simulated $$ Y_i$$ and $$ D_i$$ to estimate treatment effects with $$ {\widehat{b}}^{*}$$ from this OLS regression:

Program evaluation with EXAM is as unbiased and precise as that with RCT. Fig. 2A and SI Appendix, Fig. S2 plot the distribution of the resulting treatment effect estimates $$ {\widehat{b}}^{*}$$ and $$ {\widehat{\beta }}^{*}$$ over simulations. In line with Propositions 4 and 5 in SI Appendix, the means of $$ {\widehat{b}}^{*}$$ and $$ {\widehat{\beta }}^{*}$$ for EXAM are indistinguishable from those under RCT. Both designs successfully recover ref. 15’s ATE estimate (4.5% reduction in diarrhea; replicated in SI Appendix, Table S1).

Fig. 2.

EXAM vs RCT: Treatment-effect estimates. This figure compares EXAM and RCT’s causal-inference performance by showing the distribution of ATE estimates under each design. A shows the distribution of treatment-effect estimates $$ {\widehat{b}}^{*}$$, and B shows robust $$ P$$ values for $$ {\widehat{b}}^{*}$$. In A, blue bins indicate ATE estimates for RCT, while transparent bins with red outlines indicate those for EXAM. The solid vertical line indicates the mean for EXAM, while the dashed vertical line indicates that for RCT. In B, mean is represented by a solid line and median by a dashed line. The $$ P$$ values are based on robust standard errors. Blue bins indicate $$ P$$ values for RCT, while transparent bins with red outlines indicate those for EXAM. The solid vertical line indicates median for EXAM, while the dashed vertical line indicates that for RCT.

Perhaps more importantly, the distributions of $$ {\widehat{b}}^{*}$$ and $$ {\widehat{\beta }}^{*}$$ for EXAM have similar SDs as those for RCT. This means that the two experimental designs produce similar exact, finite-sample standard errors in their estimates $$ {\widehat{b}}^{*}$$ and $$ {\widehat{\beta }}^{*}$$. Variations of this observation are in Fig. 2B, which shows the distribution of the robust $$ P$$ values for the estimates $$ {\widehat{b}}^{*}$$. SI Appendix, Fig. S3 additionally shows $$ P$$ values based on exact, nonrobust, and finite population causal standard errors, where the exact standard error means the SD in the distribution of $$ {\widehat{b}}^{*}$$ in Fig. 2. RCT produces slightly smaller $$ P$$ values than EXAM, but the median $$ P$$ value is about 0.03 for RCT and about 0.04 for EXAM. Both EXAM and RCT, therefore, detect a significant ATE for a majority of cases. Overall, EXAM appears to succeed in its informational mission of eliminating selection bias and recovering ATE precisely enough.

EXAM’s WTP benefits can be regarded as welfare-relevant only if EXAM provides subjects with incentives to reveal their true WTP. I conclude my empirical analysis with an investigation of the incentive compatibility of EXAM. I repeat the following procedure many times: As before, I simulate $$ w_{i{t}_1}$$ and run EXAM to get treatment-assignment probabilities $$ p_{it}^{*}\left(ϵ\right)$$. I then randomly pick one subject $$ j$$ as a WTP manipulator and one potential WTP manipulation $$ w_{j{t}_1}^{\prime }$$ by $$ j$$. I choose the manipulator $$ j$$ uniformly randomly. The manipulation $$ w_{j{t}_1}^{\prime }$$ is from $$ N\left(w_{j{t}_1},100\right)$$, where $$ w_{j{t}_1}$$ is $$ j$$’s true WTP. SI Appendix reports similar results under different scenarios, covering different types of misreporting—that is, both overreporting and underreporting with different magnitudes. I run EXAM on the simulated data but with the WTP manipulation $$ w_{j{t}_1}^{\prime }$$ to get treatment-assignment probabilities $$ p_{it}^{\prime }\left(ϵ\right)$$. I finally compute the gain from the manipulation $$ w_{j{t}_1}^{\prime }$$:

Fig. 3.

EXAM vs RCT: Incentive (WTP manipulation $$ \sim $$ true WTP$$ +N\left(\mathrm{0,100}\right)$$). This figure shows the histogram of true WTP gains from potential WTP misreporting to EXAM, quantifying the incentive compatibility of EXAM. Each solid vertical line represents the mean WTP gain from potential WTP misreporting to EXAM. The dashed vertical line is for RCT, where the true WTP gain from any WTP misreport is zero.

If the experimenter was only interested in the most precise estimation of treatment effects, a possible experimental design would be a stratified RCT that stratifies on the same covariates as EXAM. Another possible alternative is the stratified RCT of Hahn et al. (11).

Such designs may dominate EXAM in terms of the statistical efficiency in ATE estimation, though are inferior to EXAM in terms of welfare and incentive properties. In this sense, there is a tradeoff between information and welfare/incentive.

Conclusion

Motivated by the high-stakes nature of RCTs, I propose a data-driven experiment dubbed EXAM. EXAM is a particular stratified experiment derived from a hybrid experimental-design-as-market-design problem of maximizing participants’ welfare subject to the constraint that the experimenter must produce as much information and incentives as in RCTs (Propositions 2–4). These properties are verified and quantified in an empirical application where I simulate my design on a water-source-protection experiment. The body of evidence suggests that EXAM improves subject well-being with little information and incentive costs.