10/7/2009
Ideas to improve enrollment?
  - reduce costs (fees, uniform, books)
  - conditional cash transfers (as long as student is enrolled, varies by grade
    student is in)
  - provide meal
  - reduce transportation costs (contruction)
  - health introduction (deworming, vaccines)
  - improve teacher attendance
  - employ better teachers
  - employ women teachers
  - remedial education for students that are lagging behind
  - inform parents/children of the benefits of education
  - make curriculum relevant
  - create opportunities for people that graduate (labor market intervension)

So there are lots of ideas, but how to pick a policy that works on a
  limited budget?
What questions do you want answered?
  - was the change correctly implemented?
  - did the policy have impact on some metric?

Running example: school meals
  - did school meals increase attendance?
  - did the children eat the meals?
  - did meals lead to higher test scores?
  - did the meals result in increased nutrition?
  - who benefitted from the meals (poor vs. not)?
  - did it reduce absence?
  - did it reduce dropouts?
  - did it increase enrollment?
  - did it increase length of schooling?
  - is nourishment improved, or are meals being counteracted by less eating at home?
  - effect on boys?
  - effect on girls?
  - how easy was it to implement?
  - what was the cost of the program?

Needs Evaluation (to see if you can even start the program)
  - Who is the targetted population?
      all children? poor ones?
  - What's the nature of the problem being solved?
      how will school meals solve it?
  - How does the service fit the environment?
      do teachers feel comfortable cooking?

Process Evaluation (while you're running the program, to test reliability)
  - Are the services being delivered?
      money is being spent
      are school meals delivered and reaching children?
      well-studied case study: 13% of ugandan food for children got to children
  - Are there was of improving cost effectiveness?
      substitute expensive inputs/labor with cheaper ones
      are children spending all day at school eating and not studying?
  - Are the services reaching the right population?
      schools with a large absence problem
  - Are the clients satisfied with the service?
      teachers/students response to meals

Impact Evaluation
  - Key question: did school meals cause students to attend school more?
  - A lot of the questions listed in the running example above

Process evaluation usually happens, since paying organization audits the process
Impact evaluation happens surprisingly infrequently, especially not based on
  data analysis.  Need more evidence on what actually works.

When answering process questions, need to describe what happened
  - can be done from reading documents, interviewing people
  - verify that something actually happened, and see if people are happy with it
  - we're not checking if it's effective, only verifying that _something_ happened

To answer impact questions, need to consider counterfactuals---what would have
  happened in the absence of this program?
  - but that's impossible to recreate!  the program _did_ happen!
  - so the best thing you can do is to construct or mimic the counterfactual
    as best as possible---study enough control groups
  - so impact is difference in outcome between observed program 
    beneficiaries and observed control group.

Still a problem: selection bias
  - if only poor schools get aid, then you may see rich schools still do better
    in the end due to other factors
  - if teachers have to apply for aid, then they may be more inspired to work,
    and might bias results

So to get counterfactual, pick twice as many eligible villages, and randomly
  pick half of them to implement change in initially (to benefit from law of
  large #'s).
  - after splitting groups randomly, check that baseline metrics are equal before
    implementing change
  - with successful randomization, individuals assigned to get aid will differ
    from control only in receiving the aid (ideally)
Now, and difference between both groups can be attributed to the aid

=Examples of Randomized evaluations=
Reducing the cost of education:
   - Conditional Cash Transfers: PROGRESA in Mexico
      - 3.4% increase in enrollment on average. Larger impact at the
        secondary school levels.
   - School Uniforms in Kenya
      - School Uniforms distributed to 10,000 students in grade 6, and
        then 7 in 163 randomly selected schools
      - Drop out fell from 18% to 12% for girls, 13% to 9% for boys
- School meals
   - Evaluation for Pre-schools in Kenya: participation was 30%
     higher in schools were free breakfast was given
- School health
   - Deworming in Kenya: 0.15 years of extra education
     (25% increase in presence) (only costs $.50 to deworm a child)
   - Replicated in India (pre-school).
- Incentives for Students
   - Girls scholarship program based on good performance
     on tests scores in Kenya
- Informing parents about the returns to education
   - Madagascar: increase participation

Need to test drop-out rate and not enrollment rate between treatment and control
  to avoid students from moving from control school to treatment school.

Evaluation---cost/benefit analysis
  - Needs assessment tells you what the metric for cost/benefit ratio is
  - Process evaluation gives you cost of treatment
  - Impact evaluation gives you quantified benefits

Often, NGOs only tell you the cost of treatment per year, but don't tell you the
  measured benefit---years of increased schooling, as there is no impact evaluation

Deworming example:
  - cost per child: .5 dollars/year
  - benefit: increase of .15 years of education per child
  - deworming costs .5/.15 = 3.333 dollars per extra year of education induced

Cost per year of increased enrollment:
 informing parents of the benefit of education <
 deworming <
 uniforms <
 conditional cash transfers

This helps policymakers/NGOs prioritize change