Matching and weighting for quasi-experimental policy evaluation - a primer (online)

Policymakers need to know whether social programmes and policies make a difference, and often when randomised experiments are unethical or unfeasible. Causal matching and weighting, types of quasi-experiments, can satisfy this need. Using a range of examples, we will begin by exploring intuitive understandings of how matching and weighting methods work and progress towards a gentle introduction to key concepts that underpin them. A central message will be that a robust theory from the substantive topic area is an essential ingredient for quasi-experiments to deliver the conclusions that policymakers demand. By the end of the session, participants will be empowered to tackle the technical literature and learn more.

Course Description

 

This half-day (09:30 - 12:30) online course covers:

• The case for programme and policy evaluation – why evaluation matters and what it can offer policymakers.
• A running example: estimating the average causal effect of an intervention when there is one comparison group and data from two time points: baseline and endpoint.
• What a quasi-experiment is trying to estimate, couched in terms of the potential outcomes framework, and what you lose when you can’t run an RCT.
• Exact matching, why it usually fails, and how to fix it through coarsening.
• The idea of a multivariate distance and how distance metrics (Euclidean and Mahalanobis) can be used for matching.
• The idea of a propensity score and how it can be used for matching and weighting.
• Common themes when assessing matching/weighting analyses, regardless of the approach used.

 

By the end of the course participants will be able to:

•  Understand where quasi-experiments fit into the broader landscape of evaluations.
• Make a compelling case for running quasi-experiments and how they differ from RCTs.
• Explain the potential outcomes framework and how it applies to quasi-experiments.
• Understand what a causal estimand is and be able to choose an appropriate estimand for a given evaluation question.
• Understand the steps of estimating propensity scores and common ways to use them for matching and weighting.
• Understand a selection of approaches of matching directly on covariates (coarsened exact matching and distance-based matching).
• Appraise the quality of key elements of a quasi-experimental analysis that each of the methods introduced have in common.
• Engage with literature in the field with more confidence.

 

It is assumed that participants have a firm grasp of the foundation of quantitative methods used in social science, particularly linear and logistic regression and confidence intervals. It will also facilitate the course if participants have a specific idea for a quasi-experiment they would like to run.

 

Dr Andi Fugard (they/them) is Deputy Director of NatCen’s Centre for Evaluation. They have experience designing, project managing, and analysing data from quasi-experimental and randomised controlled trials across a range of policy areas, including mental health and education. Before joining NatCen, Andi was a Senior Lecturer at Birkbeck, University of London and Lecturer at University College London, in both roles teaching statistics for social science. They are an Associate Fellow of the Higher Education Academy and a member of the UK government Evaluation and Trial Advice Panel, which advises civil servants and What Works Centres.