Lecture 2: Correlation

Three questions drive this lecture: what is a correlation, what is it good for, and how do you measure one (means, variances, covariances, slope). Exam questions test one of these three.

What a correlation is

A statement about how two variables move together, across many units (people, countries, neighborhoods). Needs variation in both. If one variable is constant in your data, there's nothing to correlate.

• Positive: one feature shows up, the other tends to as well.
• Negative: one feature shows up, the other tends not to.
• Uncorrelated: one tells you nothing about the other.

Fact or correlation? The core skill

Statement about one group: descriptive fact. Statement comparing groups: correlation. From the opening quiz:

• "The fastest growing economies are democracies." One group only. Descriptive fact.
• "Incumbents who campaign more get fewer votes." Campaigning vs. votes, both vary. Correlation.
• "Coffee drinkers are more likely to get cancer." "More likely" = comparison to non-drinkers. Correlation.
• "Countries that signed the Paris Agreement have reduced emissions." Signatories only. Descriptive fact.
• "No student who attended office hours has failed." Attendees only. Descriptive fact.

Scandal example: Pr(Reelected | Scandal) = 62/(62+8) ≈ .886. Sounds harmless in isolation. But Pr(Reelected | No scandal) = 1192/(1192+101) ≈ .922. Since .886 < .922, scandal and reelection are negatively correlated. Always ask: compared to what?

Reading a 2x2 table

Two binary variables: compare conditional probabilities. Oil and democracy table:

Pr(Democracy | Not oil) = 118/(118+29) ≈ .802 vs. Pr(Democracy | Oil) = 9/(9+11) = .45. Oil producers are less democratic: negative correlation. Flip the conditioning, same answer: Pr(Oil | Democracy) = 9/(9+118) ≈ .071 vs. Pr(Oil | Autocracy) = 11/(11+29) = .275. Correlation is symmetric: sign never changes when you flip axes.

Rule: divide the cell count by the total of the group you condition on. Distractors almost always use the wrong total (grand total or wrong margin).

The three uses of correlation

• Description. E.g., are young people underrepresented among voters? Just needs good data.
• Prediction/forecasting. E.g., negative reviews correlate with food-borne illness, so a health department targets inspections. Requires a representative sample, attention to linearity, no extrapolating past your data. No causation needed: reviews don't have to cause illness to be useful.
• Causal inference. E.g., does calculus make students more successful? Requires assuming calculus-takers are otherwise identical to non-takers. Hard to defend outside special cases. Part III builds better tools.

Assumptions get stronger down the list. Favorite exam angle.

Every formula, and what each one means

Single variable (course uses N throughout, never N - 1):

• Mean: μ_x = (Σ x_i) / N
• Variance: σ²_x = (Σ (x_i - μ_x)²) / N
• Standard deviation: σ_x = √(σ²_x)

Two variables:

• Covariance: cov(x, y) = (Σ (x_i - μ_x)(y_i - μ_y)) / N. Sign = direction. Size is hard to read, depends on units.
• Correlation coefficient: corr(x, y) = cov(x, y) / (σ_x σ_y). Unitless, -1 to 1. Measures tightness: how closely points hug the line.
• Regression slope: β = cov(x, y) / σ²_x. Measures magnitude: one-unit increase in x → y higher by β on average. Carries units (y per x).

Tightness is different from magnitude

Slides: two scatterplots, similar tightness, slopes of 5.59 vs. 0.10. Correlation near 1 means consistent, not big. A tight relationship can be tiny; a loose one can be huge. "How much does y change" → always the slope, never the correlation coefficient. Also: corr(x, y) is symmetric, but the slope changes if you swap x and y (denominator becomes σ²_y).

Linearity, and what to do about it

Covariance, correlation, and slope all measure linear relationships. A strong U-shape can show correlation near zero. Two fixes:

• Split into subsets where the relationship is roughly linear.
• Transform variables (logs are the classic case) to make it linear.

Worth memorizing: with two binary variables, the slope equals the difference in conditional probabilities: β = Pr(y = 1 | x = 1) - Pr(y = 1 | x = 0). This is how a 2x2 table becomes a regression coefficient on the midterm.

Classic traps and misconceptions

• One-group trap. "Most X are Y" / "few X do Y" is one group. Descriptive fact until compared.
• No-variation trap (selecting on the dependent variable). "Every celebrity with great skin uses this product" only looks at people with great skin. No variation, no correlation to compute. More data of the same kind doesn't fix it.
• Wrong denominator. Grand total or wrong margin gives a plausible wrong answer. Match denominator to the group you condition on.
• N vs. N - 1. Course always divides by N. Distractors use N - 1, or skip the square root (reporting variance as SD).
• Correlation coefficient vs. slope. Coefficient = tightness, slope = magnitude. Swapping them is a standard distractor.
• Causal language. Slope means "associated with, on average." "Causes" or "increases" from observational data is wrong, even with the right number attached.
• Zero correlation ≠ no relationship. Just no linear one. A strong nonlinear one can hide underneath.

Worked example

Four stores, advertising spend x ($1000s) and weekly sales y ($1000s):

Means: μ_x = 10/4 = 2.5, μ_y = 16/4 = 4. Covariance: cov(x, y) = (3 + 0 + 0 + 3)/4 = 1.5. Variance of x: σ²_x = (2.25 + 0.25 + 0.25 + 2.25)/4 = 1.25, so σ_x ≈ 1.12. Variance of y: σ²_y = (4 + 0 + 0 + 4)/4 = 2, so σ_y ≈ 1.41.

Correlation: corr = 1.5/(1.12 × 1.41) ≈ 0.95: tight positive relationship. Slope: β = 1.5/1.25 = 1.2: each extra $1000 of advertising is associated with $1200 higher weekly sales. Two numbers, two questions. Regressing x on y instead gives 1.5/2 = 0.75, a different slope, while the correlation coefficient stays 0.95.

How the exam asks this

Exercise deck and both midterms use one format: a short scenario (policy office, influencer health claim, business dataset), then 4-8 interpretation MCQs. Expect these patterns:

• "What are correlations good for?" Options: description, prediction, causal starting point. Answer: all of the above. Cousin version: match a research question to descriptive/predictive/causal.
• "Which of these are not correlations?" Giveaways: one-group statements ("most neighbourhoods that get opportunity zones are less well off") and no-variation claims. Real correlations use "tend to" or "more likely."
• Small-table computations. Compute a subgroup mean or SD, divide by N. Distractors: variance without the square root, N - 1 version, wrong subgroup.
• 2x2 table pairs. Compute a conditional proportion (watch the denominator), then judge the sign by comparing two conditional probabilities. Trap: restating the descriptive fact when the comparison flips the sign.
• Interpretation wording. Correct answer says "on average" and "associated with," right units (percentage points for binary outcomes). Distractors use causal verbs, percent instead of percentage points, or swap coefficient for slope.
• Influencer-claim autopsies. Spot the mistake: causal claim from correlation, selecting on the dependent variable, correlation claimed with no variation. Near-verbatim on Midterm 1.
• R code completion. Line of best fit: geom_smooth(method = "lm") on top of geom_point(). Slope: lm(y ~ x, data = df).

Lecture 3: Causation, Counterfactuals, and Potential Outcomes

What a causal effect actually is

A causal effect is what would happen under a change, compared to what would happen without it. Two identical worlds, one difference: that's the whole definition.

Example: Goop's "Body Vibes" stickers claim to reduce anxiety. That claim means: compare a world where you wear the sticker to an identical world where you wear a 10-cent Walmart sticker. Lower anxiety in the first world = causal effect. Every marketing claim ("reduces stress", "boosts sales", "prevents cavities") is a hidden comparison between two worlds we never see side by side.

Potential outcomes notation

Treatment T, outcome Y. Each unit i has two potential outcomes:

• Y1i = outcome for unit i if T = 1 (treated world)
• Y0i = outcome for unit i if T = 0 (untreated world)

Individual causal effect:

Effect of T on Y for unit i = Y1i - Y0i

Average treatment effect (over N units):

ATE = (1/N) * sum over i of (Y1i - Y0i)

• Order matters: treated minus untreated.
• Y0i - Y1i flips the sign. Y1i + Y0i and Y1i / Y0i are meaningless here. Exam distractors love these.
• On the formula sheet, this chapter owns Y1i - Y0i. Covariance and regression formulas belong to other lectures.

The fundamental problem of causal inference

You never observe a causal effect directly. Treated unit → you see Y1i, Y0i is unobservable. Untreated unit → reverse. One outcome per unit, ever. So Y1i - Y0i can't be computed from data for any single unit.

• Individual effects exist and are well defined, just unobservable. "Cannot be known" ≠ "does not exist."
• We can often credibly estimate the average effect across a group. This is what randomization and controlling for confounders (later lectures) are for.

Sometimes you want the average (FDA approving a drug for the population). Sometimes you want the individual effect (a jury deciding if one action harmed one person). Exams ask which one a given decision-maker needs.

There is no such thing as "the" cause

What caused WWI? The assassination? The alliances? The July Ultimatum? A fish that swam right instead of left millions of years ago? Infinitely many things, if different, would have prevented the war. No single "the cause" exists.

Causal questions must be specific counterfactual contrasts ("what is the effect of this treatment"), never a hunt for the one true cause.

Causes do not guarantee their effects

Uncle got a flu shot, still got the flu, says shots don't work. Wrong: causal effects are about probabilities, not guarantees.

• The shot can lower flu risk or severity without eliminating it.
• Right comparison: his outcome with the shot vs. his counterfactual outcome without it (maybe worse flu, or earlier flu).
• One vaccinated person catching the flu proves nothing.

Time order does not establish causation

Tempting shortcut: A happens before B, so A causes B. (Granger won a Nobel building tests on this idea.) Counterexamples:

• Christmas cards precede Christmas. Cards don't cause Christmas.
• Anxiety precedes exams. Anxiety doesn't cause the exam.
• Beer and face paint precede the World Junior Hockey Championship.

Common thread: anticipation. People act ahead of an expected event, so the "cause" appears in the data before an "effect" it never produced.

Causation does not require physical connection

Causation isn't billiard balls colliding. A threat, a reminder, an ad, a notification: no physical contact, real behavior change, real causal effect. What matters is the counterfactual comparison, not a physical mechanism.

Causation need not imply correlation

Correlation doesn't imply causation, you know that. Reverse also holds: a real causal effect can show zero correlation, or the wrong-sign correlation.

Classic case: firefighters reduce fire damage, but houses firefighters visit show more damage. Why: firefighters get sent to the worst fires. Treatment targeted at the worst cases flips the raw correlation positive even though the true effect is protective.

Same pattern recurs: nurses to outbreak zones, roundabouts to dangerous intersections, tutoring to struggling students.

Classic traps and misconceptions

• Before/after testimonials. "I used it, then I improved" ignores what would've happened anyway (time passing, a bad patch ending).
• Unobservable vs. undefined. Individual effects exist; we just can't observe them.
• Demanding guarantees. A cause can shift probabilities without working every time.
• Sign flips. Y0i - Y1i is a favorite wrong answer. Effect = treated minus untreated, always.
• Comparing across people. One person's outcome minus a different person's outcome is not an individual causal effect.
• Reading the wrong columns. In a godlike table, real data only reveals Y1i for treated rows and Y0i for untreated rows. Check whether the question means the full table or just what's observable.
• Wrong-sign correlations. Positive correlation between a protective treatment and a bad outcome usually means targeting, not harm.

Worked example: godlike table for a tutoring program

Four students, T = 1 means tutored. Suppose we magically know both potential grades:

Full table: ATE = (5 + 0 + 2 + 1) / 4 = 2. Effect on the treated (A, B): (5 + 0) / 2 = 2.5.

Now hide the counterfactual columns, like real data does. Tutored group observed: 80, 70 (mean 75). Untutored group observed: 88, 59 (mean 73.5). Observed gap: 75 - 73.5 = 1.5. Matches neither the ATE (2) nor the effect on the treated (2.5).

Why: without tutoring, the tutored students would've averaged (75 + 70) / 2 = 72.5, vs. 73.5 for the untutored group. That baseline gap of -1 drags the observed number down: 1.5 = 2.5 + (-1). This is the seed of the ATT plus Bias decomposition, formalized in later lectures.

How the exam asks this

Recurring patterns across the Chapter 3 deck and both midterms:

• One scenario, many questions. A claim (mindfulness app, red light therapy influencer, fluoride debate) followed by 4-8 MCQs, each testing a different concept. Read the scenario carefully once.
• "Which expression is the causal effect?" Algebra lookalikes: Y1i - Y0i (correct), Y0i - Y1i, Y1i + Y0i, Y1i / Y0i. Free marks if you know the convention.
• Agree/disagree with reasoning. Two options can share a verdict; only the reasoning differs. Correct reasoning is almost always: "we don't know what would've happened without the treatment."
• "Why can't this be causal evidence?" Correct answer names the unobserved counterfactual. Distractors blame sample size, measurement error, missing regressions: plausible-sounding, all wrong here.
• Godlike Y1i/Y0i tables. Both potential outcomes shown for 6-10 units. Asked: which group looks better in observed data, what's the overall effect from the full table, do the two answers disagree. Compute row by row, then average. Watch whether the question means the whole table or only what's observable.
• The trap gallery. Expect one question each on: causes without guarantees (flu shot uncle), time order (Christmas cards, before/after stories), no physical connection needed (an app "can't" cause anything), and causation without correlation (firefighters).
• Bridge to later chapters. Midterms dress this up as SDM vs. ATT numbers (roundabout study, godlike table) and claims like "this observed difference is the treatment effect." The skill being tested: naive comparison ≠ causal effect.

Lecture 4: Correlation Requires Variation (Selecting on the Dependent Variable)

Core rule: a correlation is a comparison, and a comparison requires variation in both variables. Same outcome for everyone in your sample? That's a descriptive fact, not a correlation. It tells you nothing about how two things move together.

Correlation vs. descriptive fact

• Descriptive fact: summarizes one group. "Most politicians facing a scandal win reelection."
• Correlation: compares across groups. "Politicians facing a scandal are reelected at a lower rate than politicians without a scandal."

The descriptive fact sounds like evidence, but it's only half a comparison. You need the reelection rate for scandal and no-scandal politicians, then compare.

Table from lecture:

Conditional probabilities:

• Pr(Reelected | No scandal) = 1,192 / (1,192 + 101) ≈ .922
• Pr(Reelected | Scandal) = 62 / (62 + 8) ≈ .886

"Most scandal politicians win reelection" is true (.886 is most). But scandal is negatively correlated with reelection: .886 < .922. Headline and correlation point opposite ways.

Flip the conditioning, same story:

• Pr(Scandal | Not reelected) = 8 / 109 ≈ .073
• Pr(Scandal | Reelected) = 62 / 1,254 ≈ .049

Losers were more likely to have had a scandal. Either direction of conditioning works, as long as you compare two groups.

Why variation is mathematically required

• cov(x, y) = Σ (x_i − μ_x)(y_i − μ_y) / N
• corr(x, y) = cov(x, y) / (σ_x σ_y)

No variation means every x_i = μ_x. Every deviation is zero, covariance is zero, and σ_x = 0 sits in the denominator. Correlation is undefined: zero divided by zero. That's the formal reason you can't compute a correlation from one column of a table.

Selecting on the dependent variable

Building your sample from the outcome you care about, then hunting for common features. Lecture examples:

• 10,000 hour rule. Gladwell studied successful people (Gates, the Beatles) and found 10,000+ practice hours. He never checked non-great achievers, who may have practiced just as much. No comparison row, no correlation.
• Mysterious illness parable. Many sick people drank the beverage. Meaningless until you check how many healthy people also drank it (if it's water, everyone did).
• AC/DC and teen suicide. 1985 Senate testimony cited suicidal teens who listened to AC/DC. Millions of non-suicidal teens listened too. Only the "Suicidal" column was examined.
• Suicide terrorism. Pape found foreign occupation present in most attacks since 1980. Sampled only on attacks, not on occupied places without attacks.

Selecting on the independent variable is the same error (only sampling people who got the treatment): a missing row or column, no comparison possible.

Why smart people keep doing this

• Feels intuitive: "look at the successes, find what they share."
• The world hands us selected samples by default: doctors mostly see sick people (herniated discs correlate with back pain? plenty of pain-free people have them too). We ask successful survivors for mentorship advice. Disaster commissions (Challenger, 2008 crisis) study disasters without a matched study of non-disasters.

Traps and misconceptions

• "Most X are Y" is not a correlation. Descriptive fact about one group. Need the rate of Y among not-X too.
• A high percentage proves nothing alone. Pr(Reelected | Scandal) ≈ .886 sounds harmless until compared to .922, which flips the story.
• Selecting on the DV can flip the sign. Reduced-charge developments can look like winners among completed projects while the full table shows the opposite.
• "We surveyed people who did X and most succeeded" has the same flaw for any X: policy, habit, feature. Ask: which cells of the 2x2 table were actually collected?
• Direction comes from comparing conditional probabilities, never cell counts alone. 62 reelected scandal politicians vs. 8 defeated ones means nothing without the no-scandal column.
• This is a step before causality. A properly computed correlation is still just a correlation. Lecture 4 stops earlier: many famous claims fail before causation is even on the table, because no correlation was established.

Worked example

A coding bootcamp claims: "90% of our graduates work in tech." Correlated with working in tech? Can't tell yet: descriptive fact about one group. Full city table:

Conditional probabilities:

• Pr(Tech | Bootcamp) = 90 / 100 = .90
• Pr(Tech | No bootcamp) = 400 / 500 = .80

.90 > .80: bootcamp is positively correlated with tech employment (10 point gap). Two takeaways:

• The ad's "90%" meant nothing until you saw the .80 baseline.
• If the no-bootcamp row were 450/50, the baseline would also be .90 and the correlation would be exactly zero, with the identical headline. The correlation lives in the comparison, not the headline.

How the exam asks this

The exercise deck and midterms recycle a small set of patterns:

• "Which statement is actually a correlation?" Four statements (housing policy, influencers, business data). One compares an outcome across levels of another variable ("areas with low charges tend to have more permits"). Distractors are single-group descriptive facts ("most new units are condos") or superlatives about one case.
• "Do you agree?" claims. Someone surveys only people who did the thing (developers who built, celebrities with great skin, failed initiatives) and announces a correlation or recommendation. Correct answer names the missing group ("the sample only includes those who built, leaving out those who didn't"). Watch for overcorrecting distractors like "correlation can never be identified in surveys": too strong, wrong.
• "Which cells of the 2x2 table were actually collected?" Rows = outcome, columns = treatment, labeled A/B/C/D. Selecting on the DV fills one row; selecting on the IV fills one column.
• Compute a conditional probability from a full table. E.g., "probability of tooth decay for municipal water children." Divide the cell by its own group total (row or column). Distractors use the wrong denominator: other group, grand total (a joint probability), or flipped conditioning.
• Interpret the comparison. After computing both probabilities, pick the answer that states an association ("less likely to experience tooth decay"). Reject causal wording and reject "no relationship can be described without a regression."
• Name the mistake. "Lots of celebrities with great skin use red light therapy, so you should too": selecting on the dependent variable. Why not a correlation: no variation in one variable.
• Failure post-mortems. "All our failed projects had feature X, so avoid X." Correct answer: successful projects might have had X too; without them, no comparison.

Speed tips:

• Identify the dependent variable first. Ask whether the sample contains both of its values.
• Table appears: write both conditional probabilities in the margin before reading the options.
• Claim starts with "most" or "all of the [successes/failures]": flag it as a descriptive-fact or DV-selection candidate.

Lecture 5: Regression for Description and Forecasting

Regression, upgraded from correlation. Used here for two jobs only: description and forecasting. Not causal inference (that's later). Every interpretation stays associational, never "effect" language.

The regression equation, piece by piece

Running example: daily Chicago crime on temperature.

Crime_i = α + β * Temperature_i + ε_i

• Crime_i: dependent variable (outcome, Y).
• Temperature_i: independent variable (predictor, X).
• α: intercept, predicted Y when X = 0.
• β: slope, change in predicted Y per one-unit increase in X.
• ε_i: error term. Everything the model omits, not just "noise": day of week, policing, events, and more.

Move ε to the other side to get the fitted value:

Crime_i − ε_i = α + β * Temperature_i

PredictedCrime_i = α + β * Temperature_i, also written Ĉrime_i = α + β * Temperature_i

Every observation = prediction (line) + error (miss). A hat means "predicted by the model."

Which line? The one that fits best

OLS minimizes the sum of squared errors. Closed-form slope (formula sheet):

β = cov(x, y) / σ²_x

Compare to correlation: corr(x, y) = cov(x, y) / (σ_x * σ_y).

• Same covariance underneath both.
• Correlation: unit-free, bounded −1 to 1.
• Slope: units of Y per unit of X, useful for prediction.

Reading a regression table

Example: turnout on age, 70 observations.

• Parentheses = standard errors (precision of the estimate).
• Some tables use Estimate / Std. Error / t value columns instead, where t value = Estimate / Std. Error.
• Quick 95% CI: CI95 = [β̂ − 2 * SE(β̂), β̂ + 2 * SE(β̂)]
• R2 = share of variation in Y the model explains, in-sample only. No causal meaning, no guarantee on new data.

The R workflow you must recognize

# load tidyverse
library(tidyverse)

# read in data
chi = read.csv("ChicagoCrimeTemperature2018.csv")

# new temp variable in celsius
chi = chi |> mutate(tempc = (5/9)*(temp-32))

# make a scatter plot with a fitted line
ggplot(aes(x = tempc, y = crimes), data = chi) +
  geom_point() +
  geom_smooth(method = "lm", se = FALSE)

# run a regression
model = lm(crimes ~ tempc, data = chi)

# make a table
library(modelsummary)
modelsummary(model,
             vcov = "HC1",
             statistic = "({std.error})",
             gof_map = c("nobs", "r.squared"))

• lm(y ~ x, data = df): dependent variable goes before the tilde.
• geom_point(): draws the scatter.
• geom_smooth(method = "lm"): adds the regression line.
• "Could not find function ggplot" error: missing library(tidyverse) or library(ggplot2).

When a line is the wrong shape: polynomial regression

Turnout rises with age, then flattens or falls. A straight line fits poorly at the extremes. Two fixes:

• Run separate linear regressions on subsets of the data.
• Fit a polynomial by adding powers of X: Turnout_i = α + β1 * Age_i + β2 * Age_i^2 + ... + βN * Age_i^N + ε_i

Warning: higher-order polynomials always improve in-sample fit as you add terms. That's the danger, not a virtue.

Overfitting and out-of-sample testing

A wiggly high-degree polynomial chases sample noise. Looks great in-sample, forecasts badly on fresh data.

Test: out-of-sample testing. Fit on one portion, check predictions on data the model never saw. Lecture demo: flexible model wins in-sample, simpler model wins out-of-sample.

Tradeoff to memorize: complex models fit better but risk overfitting; simple models may underfit but are easier to interpret and communicate.

Classic traps and misconceptions

• Causal language. "β = −0.68, so opportunity zones decrease business" is wrong. Descriptive regression shows groups differ on average, not that treatment caused it. A large t value doesn't fix this.
• Swapping the variables. Dependent variable goes left of the equation, before the tilde in lm(). Distractors reverse X and Y, or call "t value" a model variable.
• Misreading the table. Estimate = coefficient. Parentheses (or Std. Error column) = precision. Wrong-column arithmetic produces distractor answers.
• Intercept overreach. Intercept = predicted Y at X = 0. For binary X, that's the group-0 average: meaningful. For X = 0 outside the data range (like age 0), it's a line-anchor, interpret with care. But "meaningless, ignore it" is also wrong.
• Deterministic readings. "All neighborhoods have exactly 1.43 businesses" is wrong. Regression describes averages; individual points scatter around the line via ε.
• "Error term is just noise." Wrong. ε holds every omitted factor: income, zoning, demand, etc. This sets up omitted variable bias later.
• Extrapolation. A line fit on ages 20–40 gives shaky predictions for age 78 if the true relationship bends.
• Chasing R2. Adding polynomial terms to push R2 toward 1 is a classic overfitting trap. In-sample fit is the wrong forecasting criterion.

Worked example: predict turnout

Using the table above: T̂urnout_i = 0.145 + 0.004 * Age_i.

• Age 31: 0.145 + 0.004 * 31 = 0.145 + 0.124 = 0.269 → predicted turnout about 26.9%.
• Age 78: 0.145 + 0.004 * 78 = 0.145 + 0.312 = 0.457 → about 45.7%. Treat with suspicion: if turnout flattens or declines at older ages, this extrapolates past the data. Exactly why the lecture turns to polynomials and out-of-sample checks.
• Slope: each extra year of age is associated with a 0.4 percentage point higher predicted turnout, on average.
• Intercept: predicted turnout at age 0 is 14.5%, an anchor for the line, not a claim about newborn voters.

How the exam asks this

Recurring patterns from the chapter 5 exercise deck and both midterm practice sets. Drill until automatic.

• Scenario plus regression table. Realistic setup (opportunity zones, fluoride and tooth decay, transit/retail) with an Estimate / Std. Error / t value table. 4–8 questions: identify dependent/independent variables, match the equation to the table, compute a predicted value, interpret intercept, interpret slope.
• Prediction arithmetic with binary X. "Predicted outcome for a treated unit" = intercept + slope. Distractors: intercept alone, slope alone, wrong-column numbers.
• "How do you respond?" dialogue questions. Someone overclaims causality. Correct answer: regression shows association, but is still informative. Wrong answers: endorse the causal claim, cite a big t value as causal proof, or over-correct and deny regression shows anything.
• Error term questions. Colleague calls ε "just random noise." Right answer: lists concrete omitted factors.
• Model complexity questions. Someone proposes stacking polynomial terms for a "perfect" model. Credited answer: states the overfitting tradeoff, points to out-of-sample performance.
• R code completion and debugging. Fill in geom_smooth(method = "lm"), write lm(y ~ x, data = df) with correct variable order, know "could not find function" means a missing library() call, recognize filter() and mutate().
• Precision and inference add-ons. Smaller sample → wider standard errors, estimates stay unbiased. 95% CI meaning: a procedure that captures the truth in 95% of repeated samples, never "95% probability the truth is in this interval." P-value: probability of a result this extreme if the null were true, never the probability the null is true.

Lecture 6: Samples, Uncertainty, and Statistical Inference

Core question: when you compute a number from a sample, how much can you trust it? SEs, CIs, hypothesis tests, p-values all exist to separate real relationships from luck.

The favourite equation

Estimate = Estimand + Bias + Noise

• Estimate: what you got from your data.
• Estimand: the true population value you wanted.
• Bias: systematic, same-direction error from a flawed design.
• Noise: random error. Zero on average, swings above or below truth in any one sample.

Polling version: true support is q (estimand). Sample share is q̂ (estimate). q̂ = q + bias + noise.

Estimand, estimator, estimate

• Estimand: the target (true share q).
• Estimator: the recipe (sample N voters, ask, average).
• Estimate: the number that recipe produced this time (q̂ = 0.532, say).

Good estimator = unbiased (right on average) + precise (low noise). Dartboard analogy: bias = off-center, precision = tight clustering. Unbiased-but-imprecise and precise-but-biased are different problems, different fixes.

Standard error: quantifying precision

SE = standard deviation of estimates across many hypothetical samples.

SE ≈ sqrt(q(1 - q) / N)

• Bigger N shrinks SE, but with diminishing returns: SE falls with sqrt(N), so 4x the sample only halves the SE. By N ≈ 10,000, sampling variability is nearly gone, regardless of population size. Population size is not in the formula.
• q(1 - q) peaks at q = 0.5: a 50/50 split is the noisiest case.
• Catch: the formula needs q, the thing you don't know. Plug in q̂, or use 0.5 as the conservative worst case.

Confidence intervals and the margin of error

CLT: the distribution of estimates is approximately normal. 95% of draws fall within ~2 SDs of the mean.

margin of error = 2 * SE

CI95 = [estimate - 2*SE, estimate + 2*SE]

Regression version (formula sheet): CI95 = [β̂ - 2*SE(β̂), β̂ + 2*SE(β̂)]

What 95% CI means: run the same unbiased poll repeatedly, and the truth lands inside the interval 95% of the time. Coverage is a property of the procedure, across samples, not a 95% probability for this one interval. Says nothing about bias: a biased poll with a tiny margin of error is just precisely wrong.

Hypothesis testing and p-values

Logic runs in reverse: assume a boring world (null hypothesis), ask how surprising your estimate would be there.

Example: poll of 1,000, 532 support Republican, q̂ = 0.532. Null: tie, q = 0.5. Under null, SE = sqrt(0.5(1 - 0.5)/1000) ≈ 0.016.

(0.532 - 0.5) / 0.016 = 2 SEs above the null.

A result that favourable happens ~2.5% of the time by chance (one tail of the 95% band). p-value = 0.025. Small, so reject the null: statistically significant evidence the Republican leads.

General recipe: (estimate - null value) / SE. More than ~2 SEs out means p < 0.05.

Same machinery for regression: population relationship Income_i = α + β * YearsOfEducation_i + ε_i, β is the estimand, β̂ is your sample estimate. Software gives SE(β̂), build CIs and tests the same way.

What if you have the entire population?

Example: data on every student, correlate GPA with varsity sports. No sampling variability, so is inference pointless? No: noise is broader than sampling error. What you observe is still one draw from many ways the world could unfold (shocks, injuries, schedules). SEs and tests still tell you whether a pattern is a stable relationship or a fluke of this realization.

Substantive vs statistical significance

Statistical: is this attributable to chance? Substantive: is it big enough to matter? Independent questions. Two failure modes:

• Precise estimate of a tiny effect, oversold as important (social media voting nudges: significant, trivial).
• Imprecise estimate of a large effect, wrongly read as null (Second Reform Act: wide CI is weak evidence, not evidence of no effect).

Classic traps

• p-value trap: p = 0.038 is the chance of an estimate this extreme if the null is true. Never the probability the null is true. Never the probability your result is wrong. Read the last five words of every exam option.
• CI trap: correct reading = repeated sampling, requires an unbiased poll. "Truth is in this interval" or "95% probability for this interval" = wrong.
• Big-N trap: bigger N shrinks noise, does nothing to bias. Rerun a flawed study (health-conscious volunteers only) at 10x N: same biased answer, tighter error bars.
• "Significant means important": p < 0.05 guarantees nothing about magnitude. Compare effect size to a real benchmark.
• Extrapolation: a 1200mg turmeric study says nothing about a teaspoon. Estimates apply only to conditions actually studied.
• SE arithmetic slips: forgetting the square root (that's variance), or misplacing sqrt(N). Write out sqrt(q(1 - q)/N) and substitute step by step.

Worked example

City polls 2,500 voters on a ballot measure; 1,325 say yes.

• Estimate: q̂ = 1325 / 2500 = 0.53.
• SE: SE = sqrt(0.53 * 0.47 / 2500) = sqrt(0.0000996) ≈ 0.01.
• Margin of error: 2 * 0.01 = 0.02, so CI95 = [0.51, 0.55].
• Test null of a tie (q = 0.5): (0.53 - 0.50) / 0.01 = 3 SEs above null. That extreme, <1% by chance, p far below 0.05: reject null. Statistically significant evidence yes leads.
• Interpretation check: unbiased design means intervals like this capture the true share 95% of the time across repeated polls. With 400 voters instead: SE = sqrt(0.25/400) = 0.025, estimate only 1.2 SEs above tie, can't reject. Same point estimate, different conclusion, purely from precision.

How the exam asks this

Chapter 6 deck and both midterms recycle these patterns. Expect a scenario block (TikTok influencer citing a study, a poll, a policy dataset) with a design, effect size, 95% CI, p-value, then 4-8 interpretation MCQs:

• "What is the null hypothesis?": options like β = 1, β > 0, β = 0, β = estimate. Almost always no effect, β = 0 (or tied election, q = 0.5).
• p-value definition: four near-identical sentences, only one conditions correctly on the null being true. Distractors: "chance the null is true," missing conditioning clause, conditions on null being false.
• CI definition: repeated-sampling wording is correct. Distractors: truth is inside this interval, or CI proves practical importance.
• Bias vs noise from a design flaw: unrepresentative recruitment = bias. Follow-up: bigger sample does what? (Equally biased, more precise. "Larger samples always remove bias" = stock wrong answer.)
• Substantive vs statistical significance: significant-but-tiny effect oversold (turmeric -0.5%), or imprecise large estimate wrongly called null.
• Regression table arithmetic (Estimate, Std. Error, t value): build CI as estimate ± 2 SEs, check if it excludes zero, predict that shrinking N widens SEs while point estimates stay unbiased.
• Extrapolation traps: what the study does/doesn't support for untested doses, groups, settings.
• Vocabulary: match estimand, estimator, estimate to definitions inside the scenario.

Lecture 7: Over-Comparing and Under-Reporting (P-Hacking and Publication Bias)

Core line from the slides:

multiple testing + selective reporting = unreliable results

A hypothesis test controls false positives for one test. Run many tests and only show the interesting ones: that control breaks. A "significant" result may just be the lucky survivor of tests you never saw.

Where this sits in the course

Lecture 6: the machinery (p-values, hypothesis tests, CIs). Lecture 7: why that machinery still misleads.

• Tests produce false positives and false negatives.
• A p-value is not "probability the null is true." It's how likely data this extreme would be if the null were true.
• Statistical significance ≠ substantive significance. Tiny effects can be significant in huge samples.
• p < .05 is arbitrary. Nothing special happens at .05.
• The big one: hidden tests inflate the false positive rate of the ones you do see.

Paul the Octopus: one example, three p-values

2008-2010: Paul the octopus called 12 of 14 matches correctly. Testing the null that Paul is a coin flipper. Tool: binomial probability.

Pr(k successes in n trials) = p^k * (1 - p)^(n - k) * n! / (k! * (n - k)!)

Step 1: naive test (p = 1/2, n = 14).

• Pr(12) = (1/2)^12 * (1/2)^(14-12) * 14! / (12!(14-12)!) ≈ .0056
• Pr(13) = (1/2)^13 * (1/2)^(14-13) * 14! / (13!(14-13)!) ≈ .0009
• Pr(14) = (1/2)^14 * (1/2)^(14-14) * 14! / (14!(14-14)!) ≈ .00006
• p ≈ .0065, about 1 in 155. Looks like clairvoyance.

Step 2: fix the null. Coin-flip null was wrong. Paul mostly picked Germany games (13 of 14), favored Germany (11 of 13), and Germany usually wins (9 of 13). Mindless-Paul success probability:

(11/13)(9/13) + (2/13)(4/13) ≈ .633

Retest with p = .633 over 13 Germany games:

• Pr(11) = .633^11 * .367^2 * 13! / (11! 2!) ≈ .069
• Pr(12) = .633^12 * .367^1 * 13! / (12! 1!) ≈ .020
• Pr(13) = .633^13 * .367^0 * 13! / (13! 0!) ≈ .002
• p ≈ .091, about 1 in 11. (A weighted-coin alternative gives p ≈ .030.) Either way, way less impressive than 1 in 155.

Step 3: count the other octopuses. Even p ≈ .03 assumes Paul was the only animal tested. With 10 guessing octopuses, chance at least one matches Paul under the null:

Pr(at least one as good as Paul | null) = 1 - (1 - .03)^10 ≈ .26

Leon the porcupine, Petty the hippo, Anton the tamarin, Mani the parakeet: all also predicting, plus countless animals nobody covered because they guessed wrong. We heard about Paul because he got lucky. That selection kills the meaning of his p-value.

The general at-least-one formula

Memorize this, it shows up constantly:

Pr(at least one significant result | all nulls true) = 1 - (1 - p)^k

p = per-test false positive rate (often .05), k = number of independent tests. k = 20 at .05: 1 - .95^20 ≈ .64. A false positive is more likely than not.

Two ways publication bias arises

• P-hacking: analyst tries different samples, specifications, outcomes until something clears p < .05, reports only that. Dishonest or self-deceiving.
• P-screening: researcher runs one honest planned test. Journals and journalists just prefer publishing significant results. Filter is downstream of the researcher.

Both produce the same damage: false positives, overestimated effects. No fraud required.

Real-world evidence

• Get-out-the-vote experiments: published studies alone average ~3.3 points effect (Enos, Fowler, Vavreck 2014). Pooling 200+ published and unpublished studies: ~0.5 points (Green, McGrath, Aronow 2013). Sixfold overstatement from screening alone.
• Bill Miller: Legg Mason Value Trust beat the market 15 years straight. Under EMH, each year is a coin flip: Pr(15 heads in a row | null) = (1/2)^15 ≈ 1 in 30,000. But ~24,000 funds trade yearly, so chance some fund hits that streak: 1 - (1 - (1/2)^15)^24,000 ≈ .52. A coin-flip world produces a Bill Miller about half the time, and the press guarantees you hear about him.

Potential solutions (the slide list)

• Default to skepticism.
• Test important, plausible hypotheses, not cute ones.
• Adjust p-values for multiple testing.
• Replication: a true effect reappears in fresh data.
• Further investigation motivated by theory.
• Preregistration: commit to hypotheses, outcomes, analysis before seeing data. Removes room to hack.
• Stricter thresholds (p < .005 proposed), or de-emphasize significance altogether.

Classic traps and misconceptions

• "p < .05 means less than 5% chance the null is true." Wrong, most-tested misconception in the course. The p-value conditions on the null being true; says nothing direct about the null's probability.
• Judging a result without asking how many tests were run. p = .01 from one preregistered test vs. p = .01 surviving from 25 outcomes: very different evidence.
• Per-test rate ≠ overall error rate. Each test: 5% false positive risk. Across 20 tests: ~64% chance of at least one, from 1 - .95^20.
• Publication bias needs cheating. False. P-screening distorts the literature even with fully honest researchers.
• Wrong null model. Paul at 1 in 155 assumed a fair coin. Building in Germany's win rate and Paul's preference moved p to .03-.09. Always check whether the null actually describes "no skill" in context.
• Survivorship. You see winners (Paul, Bill Miller, the viral study) because they won. Losers are invisible: the sample you see is selected on success.
• Preregistration as a guarantee. It removes analyst flexibility, which matters a lot, but a preregistered study can still false-positive by chance. A mid-study outcome switch (the professor's exercise scenario) reintroduces p-hacking worry.
• Concluding too fast. One significant result among 25 is weak evidence, but doesn't prove false positive or fraud. Right response: caution plus replication.

Worked example: the sports pundit

A TV pundit picks 9 of 10 playoff winners correctly. Network claims genuine expertise.

Step 1: p-value under pure guessing (p = 1/2, n = 10).

• Pr(9) = (1/2)^9 * (1/2)^1 * 10! / (9! 1!) = 10/1024 ≈ .0098
• Pr(10) = (1/2)^10 = 1/1024 ≈ .0010
• p = Pr(9) + Pr(10) ≈ .011

Alone, this rejects the null at 5%. Looks like real skill.

Step 2: count the pundits. ~50 pundits across TV, radio, podcasts picked the same games. If all 50 are pure guessers, chance at least one matches her record:

1 - (1 - .011)^50 ≈ .42

A field of clueless pundits produces a "9 out of 10 genius" ~42% of the time. The network gives the victory lap to whoever got lucky. Her individual p = .011 is real arithmetic on a selected survivor: nearly worthless as evidence of skill. Right response: ask how many pundits were picking, adjust for multiple comparisons, check replication next season.

How the exam asks this

The professor's exercise deck runs one scenario (a Niagara Region randomized pilot of a Rapid Housing Support program) through a chain of interpretation MCQs. Midterms use the same template. Expect:

• "Would you like to know more?" questions: given an impressive result, what extra info matters? Correct: how many outcomes examined, whether preregistered, whether one primary outcome was named in advance.
• Multiple-outcome confidence questions: 25 outcomes measured, exactly one significant. Right answer: confidence should fall, many tests inflate false positive risk. Wrong: "more comprehensive," "significance is significance," "other 24 proven zero."
• Preregistration chains: preregistered primary outcome being the significant one raises confidence, then a twist: audit shows the primary outcome was switched mid-study. Right answer: looks like p-hacking even if the team claims innocence. Wrong: "researchers can update plans freely" or "any change is fraud."
• Advice-to-decision-maker finales: council/executive asks whether to scale a program. Credited answer: measured middle, promising evidence, replicate before scaling. Wrong: overconfident ("p < .05, it worked"), dismissive ("certainly a false positive"), accusatory ("any revision means fraud").
• Calculation items: binomial streak probabilities like (1/2)^10, at-least-one computations like 1 - (1 - p)^k for octopuses, mutual funds, 20-outcome tests. Know these cold; identify which wrong option matches which error (per-test rate, forgotten complement, wrong exponent).
• P-value meaning questions: someone claims "p < .001 means 0.1% chance the effect is zero." Correct response states the conditional: if the null were true, data this extreme would be very unlikely. Appears on both practice midterms.
• Publication bias magnitude questions: published studies average one effect size, full registry (with unpublished) averages much smaller (3.3 vs. 0.5 get-out-the-vote contrast). Identify selective publication as the explanation.
• Recycled mechanics inside the scenario: regression tables (Estimate, Std. Error, t value) to interpret, short R snippets (a filter(p_value < 0.05) pipeline dresses up selective reporting as a code question).

Lecture 8: Reversion to the Mean

First, a quick upgrade to confidence intervals

Tighter CI formula:

[β̂₁ - 2 × SE(β̂₁), β̂₁ + 2 × SE(β̂₁)]

2 is a rough guide. The real value is 1.96 (large-sample t-critical). Fully general two-tailed version:

[β̂₁ - t(1-α/2, df) × SE(β̂₁), β̂₁ + t(1-α/2, df) × SE(β̂₁)]

One-tailed critical value: t(1-α, df). Know cold:

• α: significance level (0.05 for a 95 percent CI).
• df: observations minus number of coefficients (intercept included). Simple regression: df = n - 2.
• Why 1 - α/2 for two tails: reject in the most extreme 5 percent, split 2.5 percent per side. Look up the 97.5th percentile: t(.975, df).

Course default: assume large sample, 2 is fine. Exam can still ask what 2 approximates and how df is counted.

The big idea: every outcome is signal plus noise

• Signal: stable, systematic part (true quality, true height, true customer base).
• Noise: transient luck (flu on test day, good sleep, one big tour bus).

An extreme outcome usually reflects both: strong signal + a lucky/unlucky noise draw. Signal persists. Noise doesn't (next draw is random again). So the next measurement is expected to land closer to the mean. That's reversion to the mean.

Same idea, different names: reversion/regression to the mean, Galton's regression to mediocrity (tall parents, somewhat shorter kids), cosmic habituation (striking findings shrink on replication because the original was partly a noise draw).

The one required condition

Reversion happens if and only if outcomes contain both signal and noise.

• Pure signal, no noise: no reversion. Height measured twice, three weeks apart: same result.
• Pure noise, no signal: complete reversion. Random number 1-100, two waves: best guess for wave 2 is just the mean.
• A mix: partial reversion. Noisier outcome → stronger reversion.

Chicago panel example (two survey waves, three weeks apart):

It is a statistical pattern, never a force

Nothing pulls values toward the mean. No gravity, no thermostat. The only mechanism: the luck behind an extreme first observation is unlikely to repeat. Two exam favorites:

• Runs in both directions in time. If John Junior is exceptionally tall, expect his son shorter AND his father shorter. A causal force only works forward; shared signal + independent noise is symmetric.
• Population spread doesn't shrink. Extreme individuals revert, but others get fresh extreme draws. Overall spread stays constant across generations. "Everyone becomes mediocre" is the classic Galton misread.

Why it fools people: interventions are triggered by extremes

We act exactly when things are unusually bad (or pick winners when things are unusually good). Noise fades, outcomes drift back to normal, the action gets the credit. Lecture's examples, all at least partly mean reversion:

• COVID-19 cases fell after shelter-in-place orders (orders came at the peak).
• Less pain reported after knee surgery (surgery happens at peak pain).
• Headache gone after aspirin (taken at peak pain).
• Teams win more after firing the coach (fired during bad streaks).
• Crime fell in NYC precincts targeted for extra policing (targets were spike precincts).
• Less depression reported after starting medication (started at a low point).
• Placebo "healing" (people enroll in trials at peak symptoms).

This doesn't prove the interventions do nothing. It means naive before-and-after is contaminated. In Ȳ₁ᵀ - Ȳ₀ᵁ = ATT + Bias + Noise, selecting units for extreme outcomes plants reversion inside Bias. Observed improvement = true effect + recovery that would've happened anyway.

Related trap: "cracking under pressure," Sports Illustrated cover jinx. Athlete studied/cover-featured right after a hot streak, next stretch looks worse. Probably reversion, not choking.

Spotting reversion in a regression or a scatterplot

Plot wave-2 outcomes against wave-1, fit a line. Slope formula:

β = cov(x, y) / σ²ₓ

Slope below 1 (flatter than the 45-degree line) = signature of reversion. With equal variance across waves and independent noise:

slope = σ²(signal) / (σ²(signal) + σ²(noise))

All signal → slope 1 (no reversion). All noise → slope 0 (full reversion, flat line at the mean). Flatter line = noisier variable. Comparing two groups: flatter fitted line = more reversion. Careful: reversion test compares slope to 1; default R t-statistic tests against 0.

Worked example

Student happiness (1-10), two waves, three weeks apart. Disposition variance = 4, day-to-day noise variance = 4.

slope = 4 / (4 + 4) = 0.5

Class mean = 6. Student who reported 9 in wave 1 (3 above mean):

6 + 0.5 × (9 - 6) = 7.5

Expected drop of 1.5 points, no intervention needed. If the university enrolled every student scoring ≤3 in a wellness program, the same arithmetic predicts sizable "improvement" by wave 2 even if the program does nothing.

Contrast: height (noise variance ~0, slope ~1): a student 3 inches above mean stays ~3 inches above mean. Arbitrary number (signal variance 0, slope 0): best wave-2 prediction for everyone is the mean (~50.5).

Classic traps and misconceptions

• Treating reversion as a force. "Prices pulled down like gravity" is wrong. Outcomes move toward the mean only because noise fades.
• Crediting a policy for improvement in units selected on extreme outcomes. If treated zones/schools/patients were chosen for being extreme, some (or all) improvement is reversion.
• Expecting reversion everywhere. Near-pure-signal variables (height, store location) barely revert.
• Thinking reversion shrinks the population over time. Spread stays put; only conditional expectations for extreme units move inward.
• Forgetting time symmetry. Reversion predicts backward too (exceptional son → less-extreme father expected). This is how you distinguish it from a causal story.
• Testing the wrong null. Slope significantly different from 0 = persistence exists. Reversion = slope below 1.
• Confusing reversion with reverse causation. "Dangerous intersections got roundabouts": two issues, reverse causation in assignment (past outcomes drove treatment) and reversion (extremes would've moderated anyway). Exam wants both named correctly.
• CI sloppiness. 2 approximates 1.96 = t(.975, df); df = observations minus all coefficients (including intercept).

How the exam asks this

Chapter 8 deck wraps everything in one running policy scenario (Niagara development-charge relief for neighbourhoods with the biggest price jumps), then fires 4-8 interpretation MCQs. Recurring patterns:

• Straight definition: "What does reversion to the mean describe?" Keyed answer: random variation, extreme outcomes followed by less extreme ones because part of the extremity was noise. Wrong distractors: "government intervened," "outcomes move to equilibrium," "the mean itself shrinks."
• Required condition: needs both signal and noise. Follow-up: pure signal → reversion disappears (identical conditions, identical results).
• "Best response?" Someone claims a program worked because extreme units improved. Right answer: flag reversion as an alternative explanation, without claiming the program did nothing.
• "Why is this incorrect?" Someone calls reversion gravity or a pull. Keyed answer: it's a pattern from fading noise, no force involved.
• Selection on extremes: treated zones were chosen for extreme pre-period values. Why does that raise the reversion worry? Extreme performers tend to moderate later even without intervention.
• Figure reading: scatter of this year vs. last year, fitted lines by treatment group. Flatter line = more reversion. Wrong distractors: distance from the 45-degree line "proves" the policy worked, or tighter clustering means equalized values.
• Before-and-after with a bad baseline (midterm 2 style): low sales, then treatment, then recovery. Keyed answer: a temporary negative shock faded on its own, recovery looks like a treatment effect.
• Cross-topic connections: naming reversion inside ATT + Bias + Noise, distinguishing it from reverse causation, and the fix (randomize among equally extreme units, or compare treated extremes to untreated extremes).
• CI note: why 2 approximates 1.96, what t(1-α/2, df) means, why two-tailed tests use α/2 (2.5 percent per tail at the 5 percent level), computing df as observations minus coefficients including the intercept.

Lecture 9: Why Correlation Doesn't Imply Causation

Part II: is the correlation real? Part III: even if real, why is it weak evidence of causation?

One framework (potential outcomes), one equation:

• Estimate = Estimand + Bias + Noise

Master this decomposition and you can answer most of the chapter.

Potential outcomes and counterfactuals

For unit i and binary treatment T, two parallel worlds:

• Y1i = outcome for unit i if T = 1 (treated world)
• Y0i = outcome for unit i if T = 0 (untreated world)
• Causal effect for unit i: Y1i - Y0i

You only ever observe one of the two. The other is a counterfactual you never see. This is the fundamental problem of causal inference.

Example: a charter student shows you her Y1i. Her Y0i is unobservable. Charter kids differ from public-school kids before any teaching happens, so a naive comparison is apples to oranges.

Three estimands: ATE, ATT, ATU

An estimand is the quantity you want. Notation: Ȳ1T = average Y1 among treated, Ȳ0U = average Y0 among untreated, etc.

• ATE (everyone): ATE = Ȳ1 - Ȳ0
• ATT (treated): ATT = Ȳ1T - Ȳ0T
• ATU (untreated): ATU = Ȳ1U - Ȳ0U
• Weighted link: ATE = ATT × Pr(T) + ATU × Pr(U)

ATT needs Ȳ0T: unobservable. That's why naive comparisons go wrong.

The master decomposition

Data gives you the simple difference in means (SDM): treated average minus untreated average, observed cells only.

• Population difference in means = Ȳ1T - Ȳ0U
• Sample difference in means = Ȳ1T - Ȳ0U + Noise
• Estimate = Estimand + Bias + Noise
• Targeting ATT: Sample difference in means = ATT + Bias + Noise, where Bias = (Ȳ1T - Ȳ0U) - (Ȳ1T - Ȳ0T) = Ȳ0T - Ȳ0U
• Targeting ATU: Sample difference in means = ATU + Bias + Noise, where Bias = (Ȳ1T - Ȳ0U) - (Ȳ1U - Ȳ0U) = Ȳ1T - Ȳ1U
• Targeting ATE: Bias = (Ȳ0T - Ȳ0U) × Pr(T) + (Ȳ1T - Ȳ1U) × Pr(U)

ATT bias term = baseline difference: how treated vs. untreated would compare with no treatment. If charter applicants would've scored higher anyway, Ȳ0T - Ȳ0U > 0, and SDM overstates the effect.

• Bias: about group comparability. Unaffected by sample size.
• Noise: sampling error. Shrinks as N grows.

Where does bias come from?

Two sources. Know both.

• Confounders: a third factor that affects treatment status AND (independently) the outcome. Both links required. Traffic volume confounds crosswalk-accidents only if busy intersections get more crosswalks AND have more accidents.
• Reverse causation: outcome affects treatment. "We installed crosswalks where accidents were already frequent" = outcome drove treatment. Arrow runs backwards.

A single scenario can contain both. Different question stems may target either.

Signing the bias

You can sign the bias even without computing it. For confounder Z: sign of corr(Z, T) times sign of corr(Z, Y).

• Same signs (both +/both −): biased upward (too positive).
• Opposite signs: biased downward (too negative).

Example: traffic volume correlates positively with crosswalk installation and with accidents. Positive × positive = upward bias. The naive regression makes crosswalks look worse at preventing accidents than they really are.

Confounders vs. mechanisms

Mechanism: on the causal pathway (treatment changes it, it changes the outcome).

• Control for a confounder: removes bias.
• Control for a mechanism: removes part of the real effect. New problem.

Quick test: does the third variable cause the treatment (confounder, control for it) or does the treatment cause the third variable (mechanism, leave it out)?

Traps and misconceptions

• "SDM is the causal effect." Only if bias is zero: Ȳ0T = Ȳ0U (for ATT). Randomization delivers this; self-selection almost never does.
• "A bigger sample fixes it." More data shrinks Noise only. Bias unaffected. 5,000 people can still give a precise wrong answer.
• "Statistically significant, so causal." Significance answers the noise question. Not the bias question. Order: significant? genuine? causal?
• Calling something a confounder with only one link. Both arrows required.
• Mixing up ATT, ATU, ATE. ATT = effect column, treated rows only. ATU = untreated rows only. ATE = all rows. SDM uses only observed cells: Y1 for treated, Y0 for untreated.
• Wrong bias identity. ATT bias: Ȳ0T - Ȳ0U. ATU bias: Ȳ1T - Ȳ1U. Don't confuse with SDM (Ȳ1T - Ȳ0U) or ATT itself (Ȳ1T - Ȳ0T).
• Controlling for mechanisms. Throwing every variable into a regression is wrong if some sit on the causal path.

Worked example

A tutoring centre lets students opt into a free exam-prep program (T). Suppose we magically observe both potential test scores for four students:

• SDM (observed cells only): Ȳ1T - Ȳ0U = (80+76)/2 - (60+58)/2 = 78 - 59 = 19
• ATT (treated rows, both columns): Ȳ1T - Ȳ0T = 78 - 69 = 9
• ATU (untreated rows): Ȳ1U - Ȳ0U = 64 - 59 = 5
• ATE (average the effect column): (10+8+6+4)/4 = 7, check weighting: 9 × 0.5 + 5 × 0.5 = 7
• Bias for the ATT: Ȳ0T - Ȳ0U = 69 - 59 = 10, confirms SDM = ATT + Bias = 9 + 10 = 19

Story: stronger students opted in. Even without tutoring, they'd have scored 10 points higher. The naive comparison (19) more than doubles the true effect on the treated (9). More students wouldn't fix this.

How the exam asks this

Recurring pattern: a policy/business scenario (crosswalks, roundabouts, gym programs, product placement) with a full potential-outcomes table, then interpretation MCQs.

• Notation ID: "Which expression is the SDM?" among Ȳ1T - Ȳ0U, Ȳ1T - Ȳ0T, Ȳ1 - Ȳ0. Know SDM vs. ATT vs. ATE vs. bias.
• The identity: pick SDM = ATT + Bias + Noise from rearranged impostors; state when bias is zero (Ȳ0T = Ȳ0U).
• Table computations: compute SDM (observed cells, watch the sign), ATT, ATE. Distractors are always other estimands, so label your work.
• "A colleague says..." questions: someone claims SDM is causal, or a large sample / small p-value proves causation. Correct answers: SDM = causal effect only if bias is zero; sample size shrinks noise only; p-values speak to chance, not bias.
• Confounder logic: correct answer always states Z correlates with BOTH treatment and outcome; distractors give only one link.
• Classification: "we treated units that already had bad outcomes" = reverse causation. A variable the treatment itself changes = mechanism (don't control for it).
• Signing the bias: two setup questions give sign of corr(Z, T) and corr(Z, Y), third asks direction of omitted-variable bias. Multiply the signs.
• Regression versions (heavy on Midterm 2): a table where the treatment coefficient shrinks as controls are added, explain as confounding. Plus R completion: lm(y ~ treatment + confounder, data = df), filter(), geom_smooth(method = "lm").

Lecture 10: Controlling for Confounders

A confounder affects both treatment and outcome. It's why a simple difference in means is biased: treated and untreated groups would differ even with no treatment. This lecture covers the fix, controlling, and its limits.

The core idea

Controlling: compare treated vs. untreated units with the same confounder value, then average those comparisons.

Running example: are Congress members extreme because party leaders whip them into line?

• Republicans vote more conservative than Democrats.
• Confounder: personal ideology (conservative people join the Republican party to begin with).
• Fix: compare legislators with similar personal ideology (from candidate surveys) across parties.
• Whatever gap survives is a cleaner estimate of party pressure.

Multiple regression automates this. In Y = α + β*T + γ*X + ε, β is the T-Y association holding X constant. If X captures all the confounding, β is an unbiased causal estimate.

Omitted variable bias: the formulas

Long regression (what you want):

Y = α + β*T + γ*X + ε

β is unbiased. If X is unobserved, you're stuck with the short regression:

Y = αS + βS*T + εS

Gap between what you get and what you want, the omitted variable bias:

βS - β = γ * Π

where Π is the slope from regressing omitted X on treatment T:

Π = cov(T, X) / var(T)

Professor's exercise deck writes the same result as:

Bias(β̂1) = β2 * cov(T, X) / var(T)

Π is just the bivariate slope formula, β = cov(x, y) / σx², with the omitted variable as outcome and treatment as regressor.

Two conditions both required for bias:

• Omitted variable matters for the outcome (γ ≠ 0)
• Omitted variable correlates with treatment (Π ≠ 0)

Either one zero, no bias.

Signing the bias

Exams love this. Sign of the bias = sign of γ * Π:

• γ > 0 and Π > 0: upward bias, βS too positive.
• γ < 0 and Π < 0: upward bias again, negatives cancel.
• Opposite signs: downward bias, βS too negative.

Upward means "too positive." If the true effect is negative, upward bias makes it look less negative or even positive. This is why treatment coefficients shrink toward zero as controls enter: the control removes bias and moves the estimate toward the truth.

Worked example

True model: score = 40 + 5*tutoring + 2*motivation (motivation on 0-10 scale).

Tutored students are more motivated: regressing motivation on tutoring gives Π = 3.

Omit motivation, short regression gives:

βS = β + γ*Π = 5 + 2*3 = 11

Naive comparison more than doubles the true effect of 5.

Sign the bias without data: γ positive (motivation raises scores), Π positive (tutored students more motivated) → upward bias, +6.

What controlling cannot fix

• Unobserved confounders. You can only control for what you measure. Missing ambition, enthusiasm, family background stays in the error term and keeps biasing the estimate. No number of observed controls fixes this.
• Reverse causation. If the outcome influences the treatment (dangerous intersections get safety upgrades, successful neighborhoods get tax designations), controlling for other variables does nothing.

Lecture's illustration: is social media bad for you? Researchers ran a paid-quit experiment, and separately analyzed observational data with controls. A controlled observational estimate only earns the same trust as the experiment when the no-unobserved-confounders assumption is credible, and that assumption is untestable.

Confounders vs. mechanisms

A mechanism sits on the causal path: T causes M, M causes Y.

Don't control for mechanisms. The mechanism is part of the effect you're estimating; controlling for it blocks the pathway and understates the total impact.

Exercise deck version: Opportunity Zones work by cutting property taxes to zero, so controlling for property tax rate erases the program's main channel.

Nasty case: some variables are both confounder and mechanism. Effect of economic prosperity on civil war, control for democracy? Democracy plausibly causes prosperity (confounder) and is plausibly caused by prosperity on the way to affecting conflict (mechanism). No statistical fix here. Reason about the substantive question and defend your choice.

Heterogeneous effects: you estimate a LATE

If treatment effects differ across units, controlling weights units whose treatment varies more within control-variable groups (in the whipping example: the middle ideology bin, where both parties are represented).

That weighted answer is a local average treatment effect (LATE) for those heavily-weighted units, generally different from the ATE for everyone.

General issue across causal inference methods: always ask whose effect you're estimating and whether it generalizes. Slide's consolation: better LATE than nothing.

Flavours of controlling

• Regression: include confounders as right-hand-side variables.
• Matching: match each treated observation to untreated observations with similar control values, then compare.
• Entropy balancing: reweight untreated units so treated and untreated groups have similar control-variable distributions.

All three balance observed variables. None touches unobserved confounders or reverse causation. No magic.

Classic traps and misconceptions

• The R² trap. Higher R² means better prediction of Y. It says nothing about bias or causality. Almost any variable correlated with Y raises R², including bad controls that worsen the causal estimate.
• The control-coefficient trap. Control coefficients (income, crime, traffic) exist to clean up the treatment estimate. Each is an association holding other regressors fixed; the design says nothing about its causal status, since nobody controlled for the confounders of the controls.
• The "control for everything" trap. Rule: control for variables that affect both treatment and outcome and are not mechanisms. Throwing in every variable risks controlling for mechanisms and other post-treatment variables.
• The mechanism trap. Giveaway on multiple choice: a variable the treatment itself changes. Controlling for it understates the total effect.
• "With controls, it's as good as an experiment." A coefficient that survives controls is causal only if no unobserved confounders remain, unverifiable from the data. Randomized experiments don't need that assumption, which is why they're the benchmark.
• "The coefficient stopped moving, so we're done." Stability across specifications is comforting but not proof bias is gone: every specification can share the same blind spot.

How the exam asks this

Chapter 10 exercise deck and both midterm practice exams use a consistent recipe:

• One scenario, many questions. A realistic setting (Opportunity Zone tax program, shelf-placement dataset, roundabout safety study, influencer claim) followed by 4-10 MCQs: correlation vs. causation, short vs. long regressions, bias direction, caveats.
• Formula recognition. Pick the correct bias expression β2 * cov(T, X) / var(T) out of distractors that swap in β1, drop the variance denominator, or pair the wrong variables in the covariance.
• Sign-the-bias questions. Given how the confounder relates to treatment and outcome, say whether the naive estimate is too positive or too negative.
• Regression tables. Columns (1), (2), (3), standard errors in parentheses, treatment coefficient shrinking as controls enter. Asked: what shrinkage implies, whether the final column is now causal (no: unobserved confounders may remain), how to interpret the coefficient with controls ("holding income, unemployment, and crime constant"). Keep CI95 = [β̂ - 2*SE(β̂), β̂ + 2*SE(β̂)] ready for any distinguishable-from-zero follow-up.
• Trap statements to rebut. "R² rose, so the model is causal." "Crime's coefficient proves crime causes fewer businesses." "With these controls the regression must match an experiment." Correct answer is always the calm one that names the specific limit.
• Which variable should NOT be controlled for? The mechanism, the channel through which the policy works (professor's version: PropertyTaxRate in the Opportunity Zone scenario).
• Reverse causality spotting. One sentence reveals the outcome influenced treatment assignment ("neighborhoods with more businesses were more likely to be designated"). Name the problem.
• R code. Complete lm(y ~ t + x, data = df) specifications (+ adds controls, * adds interactions), filter() for subsetting rows, geom_smooth(method = "lm") for fitted lines by color group.

Lecture 11: Randomized Experiments

Observational comparisons can't guarantee you controlled for everything. Randomization fixes this by design: the bias term vanishes, and a simple difference in means becomes an unbiased causal estimate.

Why randomization kills bias

Core decomposition of what you compute from data:

• Population difference in means = Ȳ1T − Ȳ0U
• Sample difference in means = Ȳ1T − Ȳ0U + Noise
• Average treatment effect (ATE) = Ȳ1 − Ȳ0
• Bias = (Ȳ0T − Ȳ0U) × Pr(T) + (Ȳ1T − Ȳ1U) × Pr(U)

Bias appears when:

• Treated and untreated would differ even with no treatment (different Ȳ0), or
• The groups would respond differently to treatment (different Ȳ1)

Randomization makes treatment status independent of potential outcomes:

Ȳ0T = Ȳ0U and Ȳ1T = Ȳ1U → Bias = 0

Difference in means = ATT + Bias + Noise still holds. Randomization zeroes the bias term only. Noise remains: unbiased ≠ precise.

Motivating puzzle: breastfeeding correlates positively with infant health in rich countries, negatively in poor ones. Confounding: who breastfeeds and what the alternative is (clean vs. contaminated water) differs by setting. Belarus researchers randomized encouragement to breastfeed to resolve this.

Ways to randomize

• Independent randomization: coin flip per unit. Simple, but group sizes can end up lopsided.
• Constrained group sizes: randomly sort units, assign the first k (or first x%) to treatment.
• Blocked (stratified) randomization: group by covariates correlated with Y0/Y1 (school, baseline score), then randomize within each block. Guarantees balance on those variables, improves precision.

Estimation and inference

Estimate ATE via sample difference in means, or regress outcome on treatment dummy: lm(outcome ~ treated) gives the same number as the slope. (Favorite exam fact.)

With m treated units out of N total:

SE = √( Var(Y0)/(N − m) + Var(Y1)/m )

Hypothesis testing as usual: posit a null (usually zero effect), ask how likely your estimate is by chance. Two tools:

• t-test
• Randomization inference: reshuffle treatment labels many times, see where your actual estimate falls

All earlier caveats about p-values and CIs still apply.

CI95 = [β̂ − 2 × SE(β̂), β̂ + 2 × SE(β̂)]

Noncompliance: ITT, compliers, and the Wald estimator

Noncompliance: assigned-to-treatment people skip it, or control-group people get treatment anyway. Two wrong fixes:

• Comparing takers vs. non-takers: take-up is self-selected, reintroduces selection bias.
• Dropping noncompliers: remaining groups are no longer randomly comparable.

Intent-to-treat (ITT): effect of assignment, not usage. Difference in mean outcomes by assignment group. Unbiased. Often the more policy-relevant number, since governments can offer a program more easily than force usage.

Four subject types by response to assignment:

Assume no defiers (reasonable in most settings). Then estimate the complier share, the first stage:

Pr(treated | assigned to treatment) − Pr(treated | not assigned to treatment) = Pr(Complier) + noise

Exclusion restriction: assignment affects the outcome only through treatment. Under that:

ITT = CATE × Pr(Complier) + noise

Divide to get the Wald estimator:

Wald estimator = ITT / First stage = ITT / Pr(Complier)

This unbiasedly estimates CATE (complier average treatment effect): the causal effect for compliers only. ITT spreads the effect across everyone assigned (including unaffected never-takers), so dividing by complier share scales it back up.

Instrumental variables

The Wald estimator is a special case of instrumental variables (IV). Need: an outcome, a treatment, an instrument that affects the treatment (in an experiment: random assignment). Assumptions:

• Exogeneity: instrument's effect on treatment and outcome is credibly estimable (randomization delivers this).
• Exclusion restriction: instrument moves the outcome only through the treatment.
• Compliers exist: instrument actually shifts some people's treatment status.
• No defiers.

Common implementation: two-stage least squares (2SLS). Key point: IV addresses noncompliance. It's a fix inside a design, not a research design itself.

When experiments go wrong

Chance imbalance. Bad luck can make treatment and control differ on pre-treatment covariates. Options: discard the experiment, proceed as normal (randomization only promises balance on average), or control for the unbalanced covariates. A small imbalance in one covariate is expected, not evidence of failure.

Imprecise estimates. Unbiased ≠ precise. Fixes: bigger sample (more even treatment/control split), blocked/stratified design, control for pre-treatment covariates to soak up variance.

Attrition. Subjects exit before outcome measurement. Three cases, worst last:

• Random attrition: just less precision
• Nonrandom but unrelated to assignment: still workable
• Attrition caused by treatment itself: biased. Example: exercise-treatment subjects who lost no weight skip the follow-up out of embarrassment, so the observed treated sample is selected on success.

Responses: design to mitigate attrition, test whether attrition relates to treatment or pre-treatment covariates, estimate bounds.

Interference (spillovers). One unit's treatment affects another's outcome: friends pooling resources, a New Jersey minimum wage hike affecting Pennsylvania employment via border crossers, one voter mobilizing the household. Sometimes spillovers are the object of study; otherwise they contaminate the control group. Fix: randomize at a higher level (classroom, store, village) so treated and control units don't interact.

Natural experiments

Sometimes the world randomizes for its own reasons: draft lotteries, charter school admission lotteries, gambling winnings, randomized ballot order. Treat like experiments with messy compliance:

• Scrutinize the exclusion restriction (did draft assignment affect earnings only via military service?)
• Remember compliers may be unusual people, limiting how far the CATE generalizes

Classic traps and misconceptions

• "Randomization guarantees balanced groups." Guarantees balance on average, in expectation. Single samples can show chance imbalance; that alone doesn't prove failure or confounding.
• Comparing takers to non-takers. Once people self-select into taking treatment, that comparison is observational again. Same problem with dropping noncompliers.
• Confusing assignment with usage. ITT is the effect of being assigned (offered). "Usage changed outcomes by the ITT amount for everyone" is wrong on both quantity and population.
• Multiplying instead of dividing. CATE = ITT / Pr(Complier). With partial compliance, CATE is larger in magnitude than ITT. If your "CATE" is smaller than the ITT, you multiplied by mistake.
• Generalizing the CATE. It's the effect for compliers only. Never-takers and always-takers may respond differently.
• Picking the wrong estimand for policy. "Should we offer or restrict access" → ITT. "What does actually using the treatment do" → CATE.
• "Insignificant means zero effect." Small studies have low power; a real effect can produce an insignificant estimate. Hiding null results causes publication bias; publishing them keeps meta-analyses honest.
• p-value and CI misreadings. p-value = probability of an estimate this large by chance, if the null were true. Never the probability the null is true. A 95% CI describes long-run coverage across repeated samples.
• Mixing up exogeneity and the exclusion restriction. Exogeneity: the instrument is as good as random. Exclusion restriction: assignment touches the outcome only through take-up.

Worked example

A utility company randomly selects 150 of 300 households to receive a free smart thermostat installation offer. Among the 150 offered, 90 install one; no control household installs. Average daily electricity use afterward: 32.4 kWh for the offered group, 34.2 kWh for the control group.

• ITT: 32.4 − 34.2 = −1.8 kWh. Being offered the thermostat reduced average use by 1.8 kWh per day.
• First stage: 90/150 − 0/150 = 0.60. An estimated 60 percent of households are compliers.
• Wald / CATE: −1.8 / 0.60 = −3.0 kWh. Among compliers, installing the thermostat reduced use by about 3 kWh per day.

Relies on no defiers and the exclusion restriction (the offer letter changed usage only by getting thermostats installed). If merely receiving the letter made households more energy conscious, the exclusion restriction fails and the Wald number overstates the effect.

How the exam asks this

The exercise deck runs one scenario (a Ministry of Education team reviewing studies on social media and youth mental health) through a chain of interpretation MCQs. Expect these moves on the final:

• "Why does random assignment matter?" Credited answer: assignment becomes independent of potential outcomes, eliminating observed and unobserved confounding. Distractors: equalizes sample sizes, equalizes outcomes, raises R².
• Chance imbalance vignettes. Baseline anxiety (or steps, or sales) differs slightly across arms. Credited answer: balance holds on average, small imbalances happen. Follow-up: why "just control for it" or "scrap the study" may be hasty.
• Low power and the file drawer. A 60-person pilot finds nothing: credited answer is low power, never "the effect is zero." Follow-up: don't hide the null study, publishing nulls reduces publication bias.
• Compute the ITT from a table of means. Watch the sign and wording direction (an outcome 0.82 lower for the assigned group means ITT = −0.82).
• Compute the compliance rate (takers among assigned minus takers among controls), then CATE = ITT / compliance. Distractors: the multiplied version, the raw ITT, a ratio with the wrong denominator.
• Name the missing assumption. Given randomization, compliers, and no defiers: the answer is the exclusion restriction. Also watch for "which best distinguishes ITT from CATE" and "which effect should guide the Minister's decision" (ITT for access policy, CATE for effect of use).
• Spot the pathology from a footnote. Treated students convince classmates to quit social media: interference. Redesign answer: randomize at the classroom (cluster) level.
• Revise the overclaiming summary. Junior analyst writes "use decreases scores by 1.37 for everyone." Credited revision hedges to compliers and to access, with the right number on the right estimand.

Also on midterms: regression output tables (Estimate, Std. Error, t value) where the treatment-dummy slope is the difference in means; CI/p-value meaning questions; ATT + Bias + Noise decompositions from potential-outcomes tables; R completion items using lm(), filter(), ggplot(), geom_smooth(method = "lm"). Test-taking pattern: distractors use absolutes (guarantees, proves, always, exactly); credited answers are specific and hedged.

Lecture 12: Regression Discontinuity Designs

Refresher, not a rebuild. RD is the fallback when you can't run an experiment but treatment is assigned by a cutoff rule. Examples: scholarship at 80 average, extra prenatal monitoring at age 35, remedial course at 9 demerit points.

The setup: running variable, cutoff, treatment

Three ingredients. Exam loves testing these:

• Running variable R: the continuous score driving assignment (admission average, age, demerit points).
• Cutoff c: threshold where treatment status flips.
• Treatment D: what happens once you cross it. Sharp design: D = 1 if R >= c, D = 0 otherwise.

Logic: 79.9 and 80.1 are basically the same person. Only treatment jumps at the cutoff. So an outcome jump there = treatment effect. Comparing just-below to just-above approximates a local experiment.

Cost: RD gives a local average treatment effect (LATE), only valid near the cutoff. Says nothing about people far from it.

The identifying assumption: continuity of potential outcomes

Y(1) and Y(0) must vary smoothly through the cutoff. Only treatment probability jumps at c. If that holds, any jump in observed outcomes = the treatment effect there.

What breaks continuity:

• Manipulation/sorting of the running variable. Timing a delivery to dodge "advanced maternal age," a teacher bumping 79.6 to 80, retaking a test until you clear the bar. Now people just above/below differ systematically. Comparison is contaminated.
• Other things changing at the same cutoff. Turning 65 triggers both insurance and pension eligibility: a health jump at 65 can't be pinned on insurance alone.

Diagnostic: histogram of the running variable. A spike on one side, dip on the other = manipulation fingerprint. Exercise deck ends on this.

Note: continuity can hold even with heterogeneous treatment effects away from the cutoff. Design still valid, but estimate only speaks for units near the cutoff. Don't extrapolate.

Three ways to implement an RD

• 1. Naive: small window around cutoff, mean outcome just above minus just below.
• 2. Local linear: larger window, separate linear regressions each side, difference the fitted values at the cutoff. One regression can do this (see formula below).
• 3. Polynomial: even larger window, regress outcome on treatment controlling for polynomials of the running variable.

Bandwidth tradeoff (shows up every exam): narrower window = less bias (line approximates well over a short range) but fewer observations = more variance/noise. Wider window = more precision, more bias risk. Best practice: report estimates across several bandwidths, show robustness.

Key formulas

• Sharp RD assignment rule: D_i = 1 if R_i >= c, 0 otherwise
• RD estimand (difference in limits at the cutoff): tau_RD = lim(R -> c from above) E[Y | R] − lim(R -> c from below) E[Y | R]
• Local linear model with the running variable centered at the cutoff (R_i = score − c): Y_i = alpha + tau*D_i + beta1*R_i + beta2*(D_i × R_i) + e_i. tau = jump at R = 0. beta1 = slope below cutoff. beta1 + beta2 = slope above. alpha = predicted outcome just below the cutoff.
• Fuzzy RD estimator (a Wald / IV ratio): tau_fuzzy = (jump in E[Y] at cutoff) / (jump in Pr(D = 1) at cutoff). Same logic as Wald = ITT / Pr(Complier): outcome jump = ITT, jump in treatment probability = complier share.
• Confidence interval (judges significance of tau): CI95 = [tau_hat − 2*SE(tau_hat), tau_hat + 2*SE(tau_hat)]

Sharp versus fuzzy RD

Sharp: threshold fully determines treatment. Above = treated, below = untreated, no exceptions.

Fuzzy: threshold influences but doesn't fully determine treatment. Noncompliance exists: some below-cutoff units get treated anyway, some above go untreated. Treatment probability jumps at the cutoff, but not 0 to 100 percent.

Fix: combine RD with IV. Crossing the cutoff is the instrument. Needs:

• Exclusion restriction: crossing the cutoff affects the outcome only through treatment.
• Monotonicity (no defiers): crossing the cutoff never reduces anyone's treatment chance.

Resulting LATE: effect for compliers close to the cutoff, units whose treatment status moves because they crossed the threshold.

Two case studies from the lecture

• Are extremists more electable? Can't just regress vote share on extremism: candidate selection is endogenous, biased estimate. Instead: close primaries, extremist vs. moderate. Running variable = extremist's primary win margin, cutoff = zero. Barely winning vs. barely losing is close to a coin flip. Comparing general-election outcomes across zero identifies the effect of nominating an extremist. Finding: extremist nominees do worse.
• Bombing in Vietnam. Officials scored villages on a continuous security scale, converted to letter grades A-E via cutoffs. Officials saw only grades, not raw scores. Villages just on either side of a grade cutoff: nearly identical scores, discontinuously different bombing probability (fuzzy RD). Finding: bombing was counterproductive, pushed villages toward the insurgency.

Classic traps and misconceptions

• Reading tau as a global effect. RD identifies the effect only near the cutoff. "Applies to everyone, everywhere" answer choices are wrong.
• Wide-band naive comparisons. Comparing all 30-34 vs. all 35-39 year olds mixes the smooth age trend with the jump at 35. RD's whole point is isolating the discontinuity from the trend.
• Confusing the first stage with the effect. A jump in treatment probability at the cutoff is the first stage: evidence the design has bite, and the denominator of the fuzzy estimator. Never the treatment effect itself.
• Forgetting to scale in a fuzzy RD. Raw outcome jump = effect of crossing the threshold (like an ITT). Divide by the jump in treatment probability to get the complier effect.
• Believing a narrow bandwidth is free. Less functional-form bias, more variance. Nothing cuts both except more data.
• Ignoring the density check. A spike just on the favorable side of the cutoff signals sorting, breaks continuity. A threat to identification, not just standard errors.
• Misreading the regression table. In Y = alpha + tau*D + beta1*R + beta2*(D × R) + e, only tau is the RD effect. beta1 = below-cutoff slope, beta2 = change in slope. A significant slope is not a treatment effect.
• Assuming near-cutoff units are identical. Similar on average only; treatment probability is what changes discontinuously. "Guarantees identical risk factors" or "reduces measurement error to zero" options are wrong on purpose.

Worked example

A university awards a bursary to entrants with an admission average of 85 or above. Using students within 4 points of the cutoff, with dist = average − 85 and above = 1 if average >= 85, you estimate:

gpa = 2.40 + 0.15*above + 0.06*dist + 0.02*(above × dist), with SE(tau_hat) = 0.05.

• Predicted first-year GPA just below the cutoff: 2.40. Just above: 2.40 + 0.15 = 2.55. RD estimate of the bursary effect at the cutoff: tau_hat = 0.15 GPA points.
• Confidence interval: CI95 = [0.15 − 2(0.05), 0.15 + 2(0.05)] = [0.05, 0.25]. Zero is outside the interval: statistically distinguishable from no effect.
• Slopes: below the cutoff, GPA rises 0.06 per point of average; above, 0.06 + 0.02 = 0.08.
• Now suppose take-up is imperfect: share receiving the bursary jumps from 0.10 just below to 0.60 just above (first stage jump = 0.50). Fuzzy RD effect for compliers at the cutoff: 0.15 / 0.50 = 0.30 GPA points. Same numerator, scaled by who actually changed treatment status.

How the exam asks this

Exercise deck: one realistic scenario ("advanced maternal age" cutoff at 35 triggering extra prenatal monitoring) plus interpretation MCQs. Midterms use the same template: policy story, figure or regression table, 4-8 questions. Expect:

• Figure reading: x-axis and dashed vertical line = running variable and cutoff. Vertical gap in a treatment-probability plot = the first stage jump. Describe an outcome jump cautiously: "local decrease near the cutoff," "appears associated."
• Design logic: why compare just-below vs. just-above instead of everyone (near-cutoff units are similar except in treatment probability, approximating an experiment). Which behavior violates continuity (timing/manipulating the running variable to dodge the cutoff, never something innocuous like different ultrasound machines).
• Sharp vs. fuzzy classification: treatment probability jumps without going 0 to 100? Pick "fuzzy," with the correct reason. Distractors: "fuzzy because the cutoff is unknown," "sharp because the rule is exact." Both wrong.
• Fuzzy assumptions: pick "exclusion restriction and monotonicity (no defiers)" over decoys like equal sample sizes or identical covariates at all ages.
• Model interpretation: given Y_i = alpha + tau*D_i + beta1*R_i + beta2*(D_i × R_i) + e_i, identify tau (the local effect at the cutoff). Distractor: reading beta2 as the effect.
• Bandwidth: tradeoff question, nearly verbatim: narrower reduces functional-form bias near the cutoff but raises variance (fewer observations).
• LATE scope: "for whom is this identified": units near the cutoff whose treatment status is influenced by crossing it.
• Naive comparison bias: wide bins (30-34 vs. 35-39) are biased vs. RD because they mix broad running-variable trends with the cutoff jump.
• Histogram question: a spike on one side of the cutoff suggests manipulation, threatens identification.
• Midterm-style extras attached to any topic: regression output tables (Estimate, Std. Error, t value); R code completion with lm(), filter(), ggplot() + geom_smooth(method = "lm"); CI/p-value interpretation where the right answer is the careful frequentist statement, wrong answers claim certainty, probability of the null, or guarantees.
• Test-taking pattern: options with "always," "proves," "guarantees," "eliminates all bias," "reduces error to zero" are almost always wrong. Correct options say "local," "on average," "near the cutoff," "under the continuity assumption."

Lecture 13: Difference-in-Differences Designs

The big idea. Treatment turns on at different times for different units (one province hikes minimum wage, some cities get TV early). DiD compares the change in treated units to the change in control units. Differencing wipes out two confounder families:

• Anything constant across units over time (California is just different from Alabama, always).
• Anything that hits all units equally at once (a national recession, a universal policy).

What's left: the differential change in the treated group, your treatment effect, if parallel trends holds.

Parallel trends (the key assumption). Absent treatment, the treated group's outcome would have moved by the same amount as the control group's. It's a claim about trends, not levels: groups can start at completely different levels; that gap is constant and differences away. DiD breaks when the treated group is on a different trajectory for reasons unrelated to treatment.

Basic calculation. Two groups, two periods:

DiD = (Ybar_T,post - Ybar_T,pre) - (Ybar_C,post - Ybar_C,pre)

Same result differencing the other order (post gap minus pre gap):

DiD = (Ybar_T,post - Ybar_C,post) - (Ybar_T,pre - Ybar_C,pre)

Regression version. One regression with an interaction reproduces the 2x2 DiD exactly:

Y = b0 + b1*Treated + b2*Post + b3*(Treated x Post) + e

Know each coefficient cold, exams test this directly:

• b0 = control group mean, pre period.
• b1 = pre-period gap, treated vs. control (level difference).
• b2 = change over time in control group (common time shock).
• b3 = the DiD estimate. How much more treated changed vs. control. The only causal number, and only under parallel trends.

R: lm(y ~ treated * post, data = df). The * gives both main effects plus interaction; row treated:post is your estimate. Significance: CI95 = [b3_hat - 2*SE(b3_hat), b3_hat + 2*SE(b3_hat)]. T value beyond ~2 in absolute value means the interval excludes zero.

Three implementations (match method to data format):

• By hand: plug the four group means into the formula above.
• First differences: wide format, one row per unit. Compute each unit's change in outcome and treatment, regress one on the other: dY_i = a + b*dT_i + e_i. R: lm(d_outcome ~ d_treatment, data = wide_df).
• Fixed effects: long format, one row per unit-period. Regress outcome on treatment plus unit FE and time FE: Y_it = b*T_it + unit FE + time FE + e_it. R: lm(y ~ treatment + factor(unit) + factor(period), data = long_df). Unit FE absorb constant unit differences; time FE absorb common shocks. Generalizes to many units, many periods.

Diagnostics. Parallel trends is untestable (it's about an unobserved counterfactual), so build a circumstantial case:

• Pre-trends. Plot both groups over several pre-treatment periods. Moving together before treatment makes parallel trends credible. One pre-period alone tells you almost nothing.
• Lead-treatment test. In the FE setup, add a variable for the treatment's next-period value alongside actual treatment. Nonzero lead coefficient means outcomes were already diverging before treatment. Bad news for the design.
• Unit-specific linear trends. Add a per-unit time trend. If parallel trends holds, your estimate barely moves. If it jumps, worry.

Running examples.

• Minimum wage and employment: comparing California to Alabama in levels is hopeless. Compare employment changes when one state raises wages against changes elsewhere.
• Television and kids: TV arrived in some 1940s-50s US cities years before others. Compare cognitive outcomes across early vs. late TV cities.
• Obesity contagion: Christakis and Fowler: friends who became obese predicted your own obesity, a DiD-flavored claim. Same method also "finds" height and acne are contagious: a placebo test showing the assumption fails (friends share environments and self-select, so trends were never parallel).

Traps and misconceptions.

• "Groups start at different levels, so DiD is invalid." Wrong. Level differences are exactly what DiD removes. Parallel trends is about slopes, never starting points.
• "Treated differs from control in X, so it's biased." Only if X changes differentially over time during the study window. Time-invariant differences (more bars, more rural roads, bigger population) get absorbed by differencing or unit FE.
• Reading the wrong number. Within-treated before/after change mixes treatment with common time shocks. Post-period treated/control gap mixes treatment with constant group differences. Only the interaction coefficient is the DiD estimate.
• Genuine threats are time-varying and group-specific: another shock hitting only the treated group at the same time as treatment. That's the shape of every "legitimate threat" answer.
• Overclaiming precision. DiD gives an average differential change (the ATT under parallel trends), never an exact per-unit effect, never "proof" of mechanism.
• Misreading the lead test. A significant lead coefficient is never good news. It signals pre-existing trends and undermines the design.

Worked example. Province A raises minimum wage in January 2024; Province B does not. Teen employment rates:

Treated change: 50 - 52 = -2. Control change: 45 - 48 = -3. DiD = -2 - (-3) = +1.

Naive before/after: wage hike "cost" 2 points. But employment was falling everywhere by ~3 points, so relative to that counterfactual the treated province did 1 point better. Interaction regression: b0 = 48, b1 = 4 (pre gap), b2 = -3 (common decline), b3 = +1 (the DiD). This sign flip between naive change and DiD is a favorite exam moment.

How the exam asks this

Stable template across the Chapter 13 deck and midterm style: a realistic scenario (police enforcement pilot, transit program, policy rollout across divisions or municipalities), 4-8 MCQs on a small 2x2 table, a plot, and a regression table.

• "What does this number represent?" You're shown the within-treated change (e.g. 4.0 to 2.8). Recognize it as the before/after change, mixing treatment with other time-varying factors. Not the causal effect, not the ATT, not pure noise.
• Arithmetic on the table. Compute post-period treated/control gap, then the full DiD. Distractors: treated change alone, control change alone, cross-sectional gap, sign errors.
• Name the assumption. Correct: groups would have followed the same trend absent treatment. Distractors: equal pre-period levels, identical covariates, identical post-period shocks.
• Regression tables, models (1)-(3): adding Treated, then Post, then Treated x Post, with Estimate and Std. Error columns. Identify the interaction as the DiD, interpret sign and size, check significance via t value or the two-SE CI.
• Colleague-correction questions. E.g. "traffic volume differs across divisions but is stable over time, so it can't bias the DiD" (true: time-invariant factors are differenced out). Or "the DiD shows the unit installed caused exactly 0.9 fewer fatalities" (overclaim: DiD is an average difference in changes). Pick the fix that matches the actual error.
• Spot the legitimate threat. Among constant differences (more young drivers, more rural roads, bigger area), the correct threat is the time-varying, treated-only shock: e.g. a new 2023 policy affecting only treated divisions.
• Event-study plots. Similar pre-slopes support parallel trends; opposite pre-treatment slopes violate it; a level gap doesn't matter. The pre-to-post slope difference is the DiD estimate itself.
• R code completion, midterm style: complete lm() calls (interaction with treated * post, first differences, fixed effects with factor()), occasionally ggplot with geom_smooth(method = "lm") for trend plots.

Lecture 14: Assessing Mechanisms

Earlier lectures asked: does treatment change the outcome, and by how much? This lecture asks why and how. Mechanism matters for two reasons:

• Substantive understanding: "it worked" tells you little on its own.
• Portability: knowing why it worked here tells you whether it'll work elsewhere.

Headline: there is no algorithm for finding mechanisms. No regression outputs "the mechanism." It takes theory, design, and independent tests. The exam rewards guarded, humble conclusions and punishes claims that a mechanism is "proven."

Causal mediation analysis (and why to distrust it)

Example: do charter schools raise college attendance via AP classes? Mediation analysis runs two regressions:

• Short: college attendance = α1 + β1 * charter school + ε1
• Long: college attendance = α2 + β2 * charter school + γ2 * AP classes + ε2

Claimed interpretation:

• β1 = total effect of charter schools.
• β2 = "direct" effect, net of AP classes.
• (β1 - β2) / β1 = share of the effect running through AP classes.

Grain of salt. AP classes is a post-treatment variable. Charter schools may be randomly assigned, but who takes AP classes isn't. Those students differ in unobserved ways (motivation, family support, ability) that also drive college attendance. So γ2 is confounded, β2 is contaminated, and the decomposition breaks.

The slides make it concrete: you can construct a case where AP classes have zero true causal effect on attendance, yet the mediation regression still reports a healthy "share" flowing through them. Controlling for a variable the treatment itself created is the classic bad control mistake, and mediation analysis does it on purpose.

So what actually works? Four strategies.

Strategy 1: Measure intermediate outcomes

Regress candidate mediators on treatment. This part is legitimate: if treatment is randomized, its effect on the intermediate outcome is well identified. Logic is asymmetric:

• Treatment does not move the mediator → mechanism hard to defend, can be ruled out (or demoted).
• Treatment does move the mediator → mechanism is plausible, nothing more. You still don't know the mediator's effect on the final outcome, so the channel isn't confirmed.

Example: CBT for at-risk youth in Liberia. Improved economic outcomes, reduced crime. No effect on self-control (weak candidate mechanism). Big effects on social networks and attitudes toward violence (plausible channels).

Strategy 2: Test independent theoretical predictions

If Mechanism A is real, what else should be true in the data? Test that.

Example: discrimination against women in elections. Women who run win about as often as men. Proof of no discrimination? No. If discrimination exists and women know it, only exceptional women run. That predicts: marginally elected women should outperform marginally elected men (a woman who barely won cleared a higher bar). Anzia and Berry confirm it: districts electing a woman get a more productive legislator.

Extra checks sharpen the case:

• Widows who inherit a seat (no hurdle cleared) do not outperform.
• The female performance premium is largest in conservative districts, where the hurdle is highest.

Each confirmed prediction cuts out rival explanations.

Strategy 3: Test mechanisms through design (multiple treatment arms)

Build the mechanism test into the experiment. Example: Gerber, Green, and Larimer's voter turnout postcards.

• Postcard with household + neighbor voting records (threat to publicize): +8 points turnout.
• Milder postcard, no neighbor comparison: +5 points turnout.

With multiple arms plus a control, you can compare arms to each other: the gap isolates the incremental effect of social pressure. Each arm vs. control gives the total effect. One arm alone couldn't separate "any postcard works" from "social pressure works."

Strategy 4: Disentangle competing mechanisms

Sometimes two mechanisms push in opposite directions. Example: does prosperity reduce violent conflict? Commodity price shock studies often find null or mixed results.

• Higher wages raise the opportunity cost of fighting (less conflict).
• More resources mean more to fight over (more conflict).

A null total effect can hide two large, offsetting channels. Fix: find variation that moves one mechanism at a time. Labor-intensive commodity shocks mainly move wages; capital-intensive shocks mainly change the size of the prize.

Classic traps and misconceptions

• Controlling for a post-treatment variable. Sounds rigorous, is wrong. The mediator was created by treatment and is selected, so controlling for it distorts the treatment coefficient. Most-tested trap in this chapter.
• Trusting the mediation share. (β1 - β2) / β1 only measures the true mechanism share if the mediator is as good as randomly assigned, which it almost never is. The share can come out negative or above 1.
• "Treatment moves the mediator, therefore that's the mechanism." That only makes it plausible. Confirming it needs the mediator's effect on the outcome too, which you usually don't have.
• "The biggest arm proves the only mechanism." If the social-comparison arm beats the information arm, social comparison plays a role. Smaller effects from other channels can still be real.
• "A null effect means nothing is going on." Women win as often as men, yet discrimination can still exist. Competing mechanisms or selection can mask real forces.
• Coefficients on mediators/controls are associations. γ2 is confounded. Never read it as the mediator's causal effect.

Worked example

A province randomizes a job-training program, studies annual earnings. Candidate mechanism: industry certification.

• Short: earnings = α1 + β1 * training gives β1 = 4000.
• Long: earnings = α2 + β2 * training + γ2 * certification gives β2 = 2500 and γ2 = 3000.

Mediation arithmetic: (β1 - β2) / β1 = (4000 - 2500) / 4000 = 0.375. Claim: 37.5 percent of the effect flows through certification, 2500 is the "direct" effect.

Grain of salt: suppose certification has zero causal effect on earnings, and trainees who complete it are just the most motivated (would've earned more anyway). Then γ2 = 3000 reflects motivation, the drop from 4000 to 2500 reflects the same selection, and 37.5 percent is an artifact. Randomizing training doesn't fix this, since certification itself was never randomized. Better moves:

• Test whether training raises certification rates (intermediate outcome).
• Derive an independent prediction (gains should concentrate in occupations requiring the certificate).
• Redesign with arms (training with exam access vs. without).

How the exam asks this

Chapter 14 exercise deck: one long policy scenario (household water-use reports, information mechanism vs. social-comparison mechanism), 4-8 interpretation MCQs. Recurring patterns:

• Read the arms, pick the guarded conclusion. Results by arm (comparison version cuts usage a lot, information-only a little). Correct answer: one mechanism "appears stronger, though the other may also operate." Anything claiming a mechanism is "proven," "the only mechanism," or "definitively" shown is wrong.
• Spot the post-treatment control. Analyst proposes outcome = α + β*T + γ*Awareness + ε, awareness surveyed after treatment. Correct: mediator is post-treatment and possibly confounded, so controlling for it distorts the treatment estimate. Wrong distractors: "not numeric," "too subjective," "cannot vary."
• Counterfactual phrasing. Right form: "would the outcome still change if households got treatment while the mediator (beliefs) stayed unchanged?" Distractors swap in irrelevant counterfactuals (richer households, different pricing).
• Pick the informative intermediate-outcome test. Right test: does treatment move the mediator, do outcome changes track mediator changes. Distractors test irrelevant correlates (graphic design of the report).
• Pick the design. Clean answer: randomize households into one arm per mechanism plus control, compare arms. Distractors randomize something irrelevant (billing cycles) or sacrifice design for power (treat only top quartile).
• Why mediation decomposition fails. Right answer: mediators are post-treatment and possibly confounded, so the required assumptions are unrealistic. Distractors invent fake rules (mediators must be continuous, one mediator can never matter).
• Midterm furniture carries over. Regression tables (Estimate, Std. Error, t value), lm() and ggplot completions (geom_smooth(method = "lm")), short-vs-long regression language, and the rule that control coefficients are associations, never causal effects.

Lecture 15: Turning Statistics into Substance

Final part of the course: what to do with quantitative evidence. Core message: a statistic is only useful when it maps onto what you actually care about. Evidence plus objective, priors, and values equals a decision. Evidence alone isn't enough.

Match the statistic to the objective

Flagship example: fuel efficiency. EPA improves small sedans by 2 MPG or large SUVs by 2 MPG (equal numbers, equal miles driven). Looks equally effective. It isn't, because the objective is gallons burned, not MPG.

gallons per year = miles driven / MPG

• Sedan: 30 MPG, 10,000 miles = 10,000 / 30 = 333 gallons/year. At 32 MPG: 10,000 / 32 = 313 gallons. Saves ~20 gallons/car.
• SUV: 10 MPG = 10,000 / 10 = 1,000 gallons/year. At 12 MPG: 10,000 / 12 = 833 gallons. Saves ~167 gallons/car.

Same 2 MPG gain, ~8x more fuel saved on SUVs (1/MPG is nonlinear). Rule: find the quantity the decision actually turns on, then convert to it.

Percents and percentage points

percent change = (new - old) / old * 100

Example: fatalities 200 → 300 is (300 - 200) / 200 * 100 = 50, a 50 percent increase.

Traps:

• Small base: 1 shark attack → 3 is a 200 percent increase. Also just 2 extra attacks. Big percent, tiny substance.
• Zero base: 0 attacks last year makes the percent change undefined.
• Negative values: percent change on outcomes that can go negative produces nonsense.
• Percent vs. percentage point: drug "reduces cardiac problems by 44 percent." Control group rate: 2.75 percent. Absolute drop: ~1.2 percentage points (slides round to ~1). The percentage-point number is what matters for weighing benefit against cost.
• Nonsensical shares of a net change: Governor Walker: "over 50 percent of U.S. job growth in June came from Wisconsin." US net growth ~19,000, Wisconsin ~9,500, math checks out. But 5 states added more jobs than Wisconsin, and by the same logic Minnesota was 70 percent and California 150 percent. When gains and losses offset in a net total, shares of that net can exceed 100 percent and mean nothing.

Reading figures without being fooled

Polished graphics aren't proof of good thinking. Ask of any figure:

• What data and analysis produced it?
• Are the assumptions sound?
• Would another statistic or visualization be more informative?
• Does it address the actual question?
• Is the scale appropriate?
• Anything distracting or misleading?

Truncated axes: the classic trick, makes small differences look huge. Good figures: keep it simple, focus on substance, use a table when it's just as clear (89 vs 90 is fine as a table), show the data, convey uncertainty.

Bayes' Rule: updating beliefs with evidence

Evidence alone doesn't tell you what to believe. Bayes' Rule tells you how to update:

Pr(A|B) = Pr(A) * Pr(B|A) / Pr(B)

Pr(B) = Pr(A) * Pr(B|A) + Pr(not A) * Pr(B|not A)

In claim/evidence form:

Pr(Claim|Evidence) = Pr(Claim) * Pr(Evidence|Claim) / Pr(Evidence)

People v Collins (LA, 1964): couple matched an eyewitness description (blonde ponytail, yellow car, Black man with beard and mustache). A mathematician multiplied population probabilities of each feature, testified 1-in-12-million chance the couple was innocent. Two errors:

• Features are dependent, can't multiply: Pr(beard and mustache) != Pr(beard) * Pr(mustache).
• Wrong question answered. He gave Pr(Match|Innocent). Court needed Pr(Innocent|Match). Swapping the two is the prosecutor's fallacy.

Bayes' Rule converts one into the other:

Pr(Innocent|Match) = Pr(Innocent) * Pr(Match|Innocent) / [Pr(Innocent) * Pr(Match|Innocent) + Pr(Guilty) * Pr(Match|Guilty)]

Plug in: 2,000,000 innocent couples in LA, 1 guilty couple, Pr(Match|Innocent) = 1 in 1,000,000, Pr(Match|Guilty) = 1. Expected matches: 2 innocent, 1 guilty. Pr(Innocent|Match) = 2/3. Even at one-in-a-million match odds, innocent is more likely than guilty, because the prior (Pr(Innocent)) starts out huge. Shortcut: count expected couples in each scenario, then condition on what you observed.

Bayes' Rule and statistical significance

Same logic: how confident should you be that an effect is real, given a significant result? Claim = "relationship is real." Evidence = "study found significance."

Pr(real | significant) = (Power * Prior) / (Power * Prior + Significance * (1 - Prior))

• Prior = Pr(relationship real): belief before the study.
• Power = Pr(significant result | relationship real): chance the study catches a real effect.
• Significance level = Pr(significant result | relationship not real): false positive rate, usually 0.05.

Denominator = law of total probability: every significant result comes from a real-effect world (Power × Prior) or a no-effect world (Significance × (1 - Prior)).

Costs and benefits

A real, well-estimated benefit still doesn't settle a decision. Quantitative analysis usually captures only one cost or benefit among many. A program with real benefits can still fail a cost test, have side effects, or carry costs that are hard to quantify and depend on values. (Chapter 17: whether costs and benefits should just be summed.)

Classic traps and misconceptions

• Percent change confused with percentage point change (44 percent reduction of a 2.75 percent rate is ~1 percentage point).
• Large relative change on a tiny base treated as substantively large (shark attack trap).
• Share of a net change quoted as meaningful when gains/losses offset (can exceed 100 percent).
• Optimizing the headline statistic instead of the objective (MPG instead of gallons).
• Prosecutor's fallacy: reporting Pr(evidence|innocent) for Pr(innocent|evidence).
• Multiplying probabilities of dependent events as if independent.
• Assuming 5 percent significance means 95 percent chance the effect is real. Posterior depends on prior and power, can be well under 50 percent.
• Treating power as the posterior, or ignoring the prior once evidence arrives.
• Jumping from "posterior above 50 percent" straight to "act." Still need to weigh benefits against all costs.
• Trusting a figure because it looks polished. Check the scale first.

Worked example: is the significant result real?

Supplement study: significant at 5 percent, 80 percent power. Your prior that any given supplement works: 4 percent. New confidence?

• Numerator: Power * Prior = 0.80 * 0.04 = 0.032
• Denominator: 0.032 + Significance * (1 - Prior) = 0.032 + 0.05 * 0.96 = 0.032 + 0.048 = 0.080
• Posterior: 0.032 / 0.080 = 0.40

Check with counts, People v Collins style. 10,000 supplement ideas, 400 truly work. Power catches 0.80 * 400 = 320 real ones. Of 9,600 duds, 5 percent significance lets through 0.05 * 9,600 = 480 false positives. Among 800 significant results, 320 / 800 = 40 percent are genuine. Belief moved from 4 percent to 40 percent (huge update), yet the supplement is still more likely useless than useful. Both true at once.

How the exam asks this

Exercise deck runs one consumer scenario (Omega-X memory supplement: skeptical you, enthusiastic friend, a study headline), then MCQs that walk Bayes' Rule piece by piece. Expect the same on the final:

• Headline interpretation: "memory improved 7 percent", where the correct answer separates percentage point change (70 to 77) from relative percent change. Distractors: the relative reading, an invented ratio, "cannot tell without significance."
• Define the ingredients: prior (belief before the study), power (Pr(result | relationship real)), significance level (Pr(result | relationship not real)). Distractors swap the conditioning, the People v Collins error.
• Map to the formula: numerator = Power × Prior. Denominator = Power × Prior + Significance × (1 - Prior). Total probability decomposition holds because every result comes from a real-effect world or a no-effect world.
• Plug and chug: compute posterior with clean numbers (deck uses prior 0.20, power 0.80, significance 0.05: 0.16 / 0.20 = 0.80), then recompute for a friend with prior 0.50 (0.40 / 0.425 ≈ 0.94). Point being tested: same evidence, different priors, different posteriors. Distractors: the power alone (0.80), the unchanged prior, 1 minus significance (0.95), 0.50.
• Judgment call: "best response" question, friend wants to buy now. Correct: weigh expected benefits against all costs (price, side effects, opportunity cost, risk the research is wrong). Wrong: "posterior above 50 percent means always buy," "significance means ignore the price."
• Midterm-style seasoning: influencer or policy scenario, a question on what a p-value/significant result does and doesn't mean, maybe a figure or R snippet with a truncated axis or a statistic that fails to match the objective (MPG-to-gallons conversion is a likely two-step arithmetic question).

Memorize Posterior = Power * Prior / (Power * Prior + Significance * (1 - Prior)). Practice the arithmetic as fractions. When a conditional probability shows up, say out loud which way the conditioning runs before answering.

Lecture 16: Measure Your Mission

Core idea: a flawless analysis can still answer the wrong question. Before trusting a number, run four checks:

• Outcome: does it match your actual goal?
• Treatment: does it match the action you'll take?
• Sample: does it generalize to who/where you care about?
• Strategic adaptation: will people react once you act on this, breaking the relationship you estimated?

1. Outcomes: the danger of partial measures

A partial measure only tracks the outcome where you're looking. The problem relocates to the blind spot.

• Airports: weapons caught at detectors looks great, smuggling just shifts to uncovered routes.
• Speed cameras (professor's example): average speed at camera sites falls 18%, but drivers reroute and collisions can rise elsewhere. The metric improves; the mission (fewer serious collisions region-wide) may not.
• Ask: where am I not measuring? Could the problem just move there (spatial spillovers)?
• A falling metric at monitored sites can even overstate success: risk shifted, not removed.

2. Outcomes: intermediate outcomes as proxies

True outcome too slow/hard to measure, so we use a proxy: polls for elections, blood pressure for heart attacks. Two failure modes:

• Correlation isn't causation for the proxy either. Basketball correlates with height, but making someone play more basketball won't make them taller. A proxy that correlates with the goal observationally may not move the goal when you intervene.
• Heterogeneity in who responds. A campaign message might only shift the poll answers of people who'll never vote. A drug might only lower blood pressure in patients who were never at risk. Proxy moves, mission doesn't.

3. Outcomes: does the target match the mission?

Slide example: 3 of the 8 richest people dropped out of college, so a friend plans to do the same to get rich. Two errors stacked:

• Familiar ones (Ch. 4, 7, 9): selecting on the dependent variable, no comparison group, tiny unrepresentative sample.
• New (Ch. 16): even if dropping out maximized odds of becoming a top-8 billionaire, is that the goal? A sane goal is maximizing expected wealth or minimizing poverty risk, and dropping out likely does badly on both. Pick the objective first, then measure it.

4. Treatments: short run and long run are different treatments

Hot days correlate with lower economic growth. Researchers projected more hot days under climate change and multiplied through to a 23% global income loss by 2100.

The catch: a single hot Tuesday and decades of climate change are different treatments. No time to adapt to a random hot day; over decades, people adopt AC, change crops, migrate, redesign cities. Effect of the treatment studied (weather shocks) ≠ effect of the treatment that matters (climate change).

5. Samples: external validity

World Bank nutrition program in southern India: trained mothers to convert budgets into better nutrition, RCT showed it cut malnourishment. Same program in Bangladesh failed.

Likely reason: in Bangladesh, the father and mother-in-law control food purchasing, not the mother, so training mothers changed nothing. India's RCT was internally valid; it just didn't transport, because the mechanism depended on local context. Always check: does the treatment, sample, and setting you studied match where you'll act?

6. Samples: selection into your sample

Think about how observations got into your data.

• SAT example: Dean finds SAT weakly predicts GPA among enrolled students, proposes dropping the SAT. But admissions needs to know if SAT predicts performance among applicants. If admission depends on SAT plus other qualities (essays, grit), then among admits, a mediocre SAT means the student must have been strong elsewhere to get in. That induces a negative relationship between SAT and other qualities within the selected sample, masking SAT's real predictive power in the applicant pool.
• Baseball example: 2017 MLB pitchers batted .125 vs .259 for other positions. 2018 Chicago-area high school pitchers hit .322 vs .317 for everyone else. High school pitchers are just the best athletes overall. MLB selects pitchers purely on pitching, ignoring hitting at the selection step, so the selected sample of MLB pitchers ends up bad at hitting. The gap reflects selection, not "pitching skill causes bad hitting."

7. Strategic adaptation

Once a measure drives policy, people respond to the policy, and the old relationship breaks.

• England's window tax: window count was a fine wealth proxy, until taxed, then people bricked up windows.
• Baseball infield shift: positioning based on past spray charts changed batters' incentives.
• War on drugs: crack down on one drug/route, activity shifts to substitutes.

Always ask: if I act on this relationship, who has an incentive to change behavior, and does my analysis account for that?

Formulas

No new formulas in Lecture 16, it's a judgment chapter. Two course formulas still apply in the background:

• Selection into a sample or treatment shows up as the bias term in Difference in means = ATT + Bias + Noise.
• The selected-sample examples are about how selection changes corr(x, y) = cov(x, y) / (sd(x) * sd(y)) within the sample you observe.

Worked example: how selection weakens a predictor

Every applicant has an SAT score and a grit score, 1 to 4 each, unrelated in the applicant pool. True performance: performance = SAT + grit. College admits anyone with SAT + grit >= 6. Admitted students:

Pattern: the SAT-2 admit has grit 4; SAT-4 admits have grit as low as 2. Among admits, cov(SAT, grit) is about -0.28, versus zero in the applicant pool.

Result: corr(SAT, performance) is about 0.71 in the applicant pool, but falls to 0.50 among admits (shrinks further with a stricter cutoff). The Dean studying only enrolled students concludes the SAT is weak, when it's doing exactly its job in the pool where the decision is made.

Classic traps and misconceptions

• "The metric improved, so the mission is succeeding." Partial measures can improve while the real problem relocates or worsens.
• "The proxy correlates with the goal, so moving the proxy moves the goal." Basketball and height. Intervening on a correlated proxy needs its own evidence.
• Ignoring heterogeneity. A treatment that moves the proxy only for people irrelevant to the mission (non-voters, healthy patients) looks like a win and does nothing.
• Selecting on the dependent variable, wrong objective. Studying only billionaires, never asking if "chance of being top 8" is even the right thing to maximize.
• Extrapolating short-run estimates to long-run policy. Weather effects assume no adaptation; climate change comes with decades of it.
• Assuming what worked there works here. Internally valid experiments can fail to generalize when the mechanism depends on context (India vs. Bangladesh).
• Evaluating a predictor inside a selected sample. Selection on multiple criteria induces negative correlation among them within the sample, weakening each predictor's apparent power (SAT dean, MLB pitchers).
• Forgetting strategic adaptation. Once a measure becomes a policy target, people game it, and pre-policy relationships stop holding (window tax, the shift, war on drugs).

How the exam asks this

Chapter 16 exercises run one long scenario (automated speed cameras in Niagara Region) with interpretation MCQs. Midterms reuse this format with influencer claims and business datasets. Expect:

• "Which critique is most appropriate?" A metric improves, official declares success. Correct answer: mission misalignment. Distractors (measurement error, small sample, seasonality) sound plausible but miss the point.
• "Which risk does this pattern illustrate?" Behavior shifts around the measurement (rerouting, relocated activity). Answer: strategic adaptation or spatial spillovers. Distractors: attrition bias, reverse causality, regression to the mean.
• "Which outcome is most aligned with the mission?" Pick the region-wide, representative measure of the real goal over site-specific metrics, activity counts, or opinion surveys.
• "Why might the estimate overstate the true impact?" Look for displaced, not reduced, risk. Mirror case: general deterrence at unmonitored sites would make the site-specific estimate understate impact. Read the direction carefully.
• Selected-sample questions. A dean/employer/league studies a predictor inside an already-selected group and calls it useless. Name the applicants-vs-admits problem and the induced negative correlation among selection criteria.
• Recommendation questions. "To better evaluate progress, the Region should..." Correct answer widens measurement to the mission (region-wide collisions, representative road sample). Reject options that just intensify the narrow metric (more cameras, more tickets).
• Regression questions. Small, insignificant coefficient on a selected sample. Trap: treating a large p-value as proof of no effect. Right answer flags the selection problem and what a p-value can/can't say.

Exam strategy: state the mission in one sentence. Then check outcome, treatment, sample, adaptation, in that order. The right answer almost always connects the measured quantity back to the stated mission.

Lecture 17: The Limits of Quantification

Final chapter. Question flips: not "what does the data show" but "what can't data tell us." Two limits:

• Evidence is sometimes too weak to settle a question.
• Even perfect evidence can't tell you what your values should be.

Exam tests one skill: spot where the numbers stop and the value judgments start.

Making decisions with limited evidence

Analysts "look where the light is": they study what's easy to measure. Many important questions stay unanswered. You still have to decide.

Key testable idea: "there is no evidence for X" means one of two very different things. Diagnose which:

• Situation 1: No compelling study exists. Little evidence either way. X could be true or false.
• Situation 2: Rigorous, high-powered studies exist and show X is wrong.

Headlines collapse these into one phrase. They demand opposite responses.

Link to earlier chapters: failure to reject the null is never proof of the null. A wide, insignificant confidence interval is consistent with zero effect and with a large effect. Absence of evidence is weak evidence of absence, at best.

Flossing case study. 2016 NYT: only "very unreliable" evidence that flossing reduces plaque. Reasons you might floss anyway, each a diagnostic tool for any "no evidence" claim:

• Small samples: imprecise estimates, low power.
• Low compliance: people assigned to floss often didn't, diluting the effect and hurting power more.
• Short-term outcomes only: flossing's benefits (avoiding decay, gum disease) build over years.
• Good biological theory supports flossing.
• Evidence was more mixed than the headline: studies did find significant reductions in gingivitis, and never even measured decay or gum separation.

Lesson: when evidence is inconclusive, combine it with theory and substantive knowledge, then decide. Doing nothing is also a decision. Inconclusive evidence doesn't automatically support the status quo.

OIRA example. U.S. Office of Information and Regulatory Affairs requires cost-benefit analysis before approving regulations. No convincing quantification of a pollutant's harm means OIRA ignores it. That gives the unmeasured benefit an implicit weight of zero: a substantive, probably wrong, assumption smuggled in as procedure. Quantify when you can, but don't let inability to measure precisely block acting on benefits you reasonably believe exist.

Quantification and values

Quantitative evidence should serve our values. Careless quantification lets the desire to measure start steering the values instead. Two failure modes:

1. Quantification smuggles in values you don't share: algorithmic bias. Health-care algorithms predict future costs from claims, diagnoses, procedures, and medications to enroll high-need patients in care programs. Race is excluded deliberately. Bias happens anyway: predicted cost proxies for access, and access differs by race. Patients with historically worse access spend less for the same sickness level. The algorithm reads low past spending as low need, so equally sick patients score below the cutoff. Removing race didn't remove the bias, it hid the bias inside the outcome variable. This is Obermeyer et al. (2019), from the slides.

2. Quantification shapes your values in bad ways: cost-benefit analysis. Tempting move: sum all costs and benefits, let the total decide, no values needed. Rebuttal: maximizing the simple sum is itself a value choice (the book calls it "crass utilitarianism"). Summing ignores distribution: who pays, who benefits, when. Two sharpening examples from the slides:

• Removing children from abusive homes: the child's welfare isn't just a line item to net against costs.
• Rich countries dumping toxic waste on poor countries: a raw dollar-sum CBA can say yes. Most people's values say no.

Quantifying is fine. Pretending the aggregation rule is value-free is the error.

Closing charge of the course. These skills can exploit people who can't think clearly with data. Don't. Be transparent about your evidence's strengths and weaknesses, be clear on what the data can and can't show, and be honest about which parts of your conclusion are values versus evidence. Final-exam checklist from the last slide: philosophy of causation, selecting on the dependent variable, substantive versus statistical significance, reversion to the mean, publication bias, correlation versus causation, and how to get credible causal estimates.

Formulas

No new formulas this lecture. Two earlier tools get reused:

• 95% confidence interval, for judging whether "no evidence" means imprecision: CI95 = [beta_hat - 2*SE(beta_hat), beta_hat + 2*SE(beta_hat)]
• The p-value logic: probability of an estimate this extreme if the null were true. Why an insignificant result never proves the null.

The exercise deck also uses two invented decision-rule formulas. Both embed value judgments:

• Fiscal ranking ratio: Score = ProjectedTaxRevenueAfterYear5 / TaxRevenueWaived (counting only fiscal quantities is itself a choice)
• Weighted scoring algorithm: Score = 0.6 * PredictedJobs + 0.4 * ProjectedLongTermTaxRevenue (0.6 and 0.4 are a political statement about jobs versus revenue)

Classic traps and misconceptions

• "No evidence for X, therefore X is false." Diagnose first: strong-evidence-against, or weak-evidence-either-way? Small samples, low compliance, short follow-up point to the second.
• "Insignificant estimate means zero effect." Failure to reject the null is never proof of the null. Check the CI: if it contains both zero and large effects, the study is uninformative.
• "Evidence is inconclusive, so do nothing." Status quo is also a choice. Combine weak evidence with theory and decide.
• "We removed the sensitive variable, so no bias." Bias enters through proxies. Past health spending proxies for access, so predicting cost reproduces racial disparities with no race variable in sight.
• "Algorithms and formulas are value-neutral." Someone chose the objective, inputs, and weights. Each choice encodes what (and who) counts.
• "If we can't quantify it, leave it out." That assigns an implicit weight of zero, biasing the evaluation toward whatever's easy to measure (the OIRA problem).
• "Add up costs and benefits, pick the max." That rule is crass utilitarianism: a value system chosen silently. Ignores distribution, fairness, rights.
• Exercise-deck distractor pattern: wrong options are absolutist ("a single metric is never useful", "algorithms are objective", "future residents cannot be considered"). Right option is the nuanced one: keep the quantitative tool, name the hidden value choice.

Worked example: weights are values

Niagara Regional Council must give a tax waiver to one of two firms, ranked by Score = w1 * PredictedJobs + w2 * ProjectedRevenue. A mayor claims this "removes politics from the decision." Applicants (inputs on comparable scales):

Proposed weights w1 = 0.6, w2 = 0.4:

• Firm A: 0.6*90 + 0.4*20 = 54 + 8 = 62
• Firm B: 0.6*40 + 0.4*85 = 24 + 34 = 58

Firm A wins. Now Council values long-run revenue more: w1 = 0.3, w2 = 0.7:

• Firm A: 0.3*90 + 0.7*20 = 27 + 14 = 41
• Firm B: 0.3*40 + 0.7*85 = 12 + 59.5 = 71.5

Firm B wins. Data didn't change, only the weights did. The winner is decided by how much the Region values jobs relative to revenue: the exact political question the algorithm claimed to eliminate. The algorithm didn't remove the value judgment, it froze one particular judgment into arithmetic and hid it from debate. Fix: transparency. State the weights, state what's left out (job quality, who benefits, neighbourhood effects), let Council argue values directly.

How the exam asks this

The exercise deck: one policy scenario (Niagara property tax waiver), then 8-10 interpretation questions like "Which critique is strongest?", "Which reply is most accurate?", "Which answer best reflects the Chapter 17 framework?" Expect the final to reuse this format. Recurring patterns:

• Single-metric critiques: a policy judged by one number (net jobs in 3 years). Right answer: the metric undervalues other outcomes the decision-maker cares about. Wrong answers: single metrics are "never" useful, or the metric is "unrelated" to anything.
• "Value-neutral" formula/algorithm claims: a ratio or weighted score offered as objective. Right answer: names the embedded value choice (what's counted, how it's weighted, whose welfare matters). Wrong answers: declare algorithms objective, or demand weights be "estimated statistically."
• Distributional reasoning: who benefits (large external firms vs. small local owners, current vs. future residents, vulnerable neighbourhoods). Right answer addresses distribution of benefits and costs, distinct from totals or averages.
• Hard-to-quantify effects: revitalized corridors, neighbourhood identity, displacement of existing firms. Right answer includes them and names the bias from omitting them. Wrong answers exclude them or bury them in a fake score.
• Synthesis question: how to use quantitative tools without letting them override values. Answer is always transparency: document what's included, excluded, and whose welfare is weighted.
• Midterm-style crossovers: "no evidence" headlines paired with a regression table (Estimate, Std. Error, t value). Compute the wide CI, state correctly what a p-value does and doesn't mean, conclude the study is uninformative rather than proof of no effect. Flossing rebuttals (small n, low compliance, short horizon, supportive theory, unmeasured outcomes) are a ready-made checklist.

Tiebreaker: correct options keep quantification while making the hidden value judgment explicit. Options with "never", "always", "cannot", "guaranteed", or "purely objective" are almost always distractors.