10/9/2009

Review of randomized evaluation
We'll study the math behind the technique.

=Rubin Causal Model=
outcome y that we measure (childhood IQ)
treatment w\in{0,1} that we provide (receiving iodine within first 6 months of birth)

so for person i, y_i(0) and y_i(1) is their outcome if they received or didn't
  receive the treatment y_i = (w_i)(y_i(1)) + (1-w_i)(y_i(0))
put another way,
  y_i =   |y_i(1) if w_i = 1
          |y_i(0) if w_i = 0

So we'd ideally like to create two universes, one where you get the treatment,
  and one where you don't.  Ideally, we'd like to measure the outcome for each 
  person with both treatments, but we can't in reality.

Average treatment effect:
  ATE = E[y_i(1)-y_i(0)]
  In truth, for any person, we will only measure y_i(1) or y_i(0).  In fact,
    it's worse than that.  It's not like everyone will even apply for the
    treatment or be able to get it, so we can only measure the treatment
    for a population that choose to get it.

Treatment on the treated population:
  TT = E[y_i(1)-y_i(0)|w_i=1]

In the real world, we only measure
  y_i(1) for w_i = 1
  and y_i(0) for w_i = 0

So we actually are measuring:
  E[y_i(1)|w_i = 1] - E[y_i(0)|w_i = 0]
    add and subtract a term to do:
    = E[y_i(1)|w_i = 1] - E[y_i(0)|w_i = 0] + E[y_i(0)|w_i=1] - E[y_i(0)|w_i=1]
              1                   2                   3                 4
    = (1 - 4) + (3 - 2)
    =   TT    + (E[y_i(0)|w_i=1] - E[y_i(0)|w_i = 0])
        TT    +    selection bias

Selection Bias---the difference between the people who were selected into the
  study and those that weren't, had they not taken the treatment.

So what we measure is both the effect on the treated _AND_ the selection bias
  of those that entered the study.  That kind of sucks.  It's like saying
  Kaplan courses help you raise your SATs because averages are higher, but in
  reality, the kaplan courses are taken by people who are fundamentally different
  than those that don't take them, and are biases to doing better!

So how do we deal with selection bias?  Possibilities:
  - w_i is orthogonal to y_i(1) or y_i(0) -> getting treatment has
    no relation to what your bias is!
    Randomization does this, since you randomize the likelihood of getting treated
    So E[y_i(0)|w_i=1] = E[y_i(0)|w_i = 0], and selection bias = 0!
  - w_i is orthogonal to y_i(1) or y_i(0) given some external variable x_i (e.g., wealth)
    so then E[y_i(0)|w_i=1, x_i] = E[y_i(0)|w_i = 0, x_i], meaning we've controlled
    for the variable which leads to a difference.  Think of it as putting the rich into
    a bin and the poor in a different bin, and comparing only within bins.  This reduces
    needless variance.

How to perform a randomized study
1) Start with extraneous variables (age, religion, education)
   randomize the trial within the bins that the variables form
   that's called stratified randomization.  That's the J-PAL school of thought.
   You can also just randomize across entire population, and control for
     the variables later (not recommended by J-PAL)
2) Do a power calculation---for a given effect size, how many people would
   it take to see a significant difference?  This helps match the budget of the
   study.
3) Attrition---measure how many people left the study
4) Non-compliance---what happens if people don't follow the protocol
5) Contagion---???