Lecture 2: Correlation

This lecture answers three questions. What is a correlation? What are correlations good for? And how do we actually measure one, using means, variances, covariances, and the slope of the line of best fit. Almost every exam question on this material tests one of those three things.

What a correlation is

A correlation describes the extent to which two features of the world tend to occur together. It is always a statement about two variables measured across many units (people, countries, neighborhoods), and it requires variation in both variables. If either variable takes only one value in your data, no correlation exists to be measured.

• Positive correlation: when one feature occurs, the other tends to occur too.
• Negative correlation: when one feature occurs, the other tends not to occur.
• Uncorrelated: the occurrence of one feature tells you nothing about the other.

Fact or correlation? The core skill

The lecture drills one distinction hard: a statement about a single group is a descriptive fact, and a correlation requires a comparison across groups. The opening quiz gives five statements:

• "The fastest growing economies are democracies." Describes only the fastest growing economies. Descriptive fact.
• "Incumbents who campaign more get fewer votes." Relates campaigning (varies) to votes (varies). Correlation.
• "Coffee drinkers are more likely to get cancer." "More likely" implies a comparison with non-drinkers. Correlation.
• "Countries that signed the Paris Agreement have reduced their emissions." Only describes signatories. Descriptive fact.
• "No student that regularly attended office hours has failed the course." Only describes attendees. Descriptive fact.

The scandal example shows why this matters. It is true that most politicians facing a scandal win reelection: Pr(Reelected | Scandal) = 62/(62+8) ≈ .886. That sounds like scandals are harmless. But the comparison group wins even more often: Pr(Reelected | No scandal) = 1192/(1192+101) ≈ .922. Since .886 < .922, scandal and reelection are negatively correlated, even though the one-group descriptive fact sounded positive. Always ask: compared to what?

Reading a 2x2 table

With two binary variables, you establish a correlation by comparing conditional probabilities. From the lecture's oil and democracy table:

Pr(Democracy | Not oil) = 118/(118+29) ≈ .802 versus Pr(Democracy | Oil) = 9/(9+11) = .45. Democracy is less likely among oil producers, so oil and democracy are negatively correlated. You get the same conclusion conditioning the other way: Pr(Oil | Democracy) = 9/(9+118) ≈ .071 versus Pr(Oil | Autocracy) = 11/(11+29) = .275. Correlation is symmetric: flipping the axes (or the conditioning) never changes the sign.

The mechanical rule: divide the cell count by the total of the group you are conditioning on. Exam distractors are almost always the same cell divided by the wrong total (the grand total, or the other margin).

The three uses of correlation

• Description. Are young people underrepresented among voters? The correlation between age and turnout answers this directly. No assumptions needed beyond having good data.
• Prediction and forecasting. Negative online reviews correlate with food-borne illness, so a health department can use reviews to target inspections. This requires assuming the sample you have is representative of the population you want to predict for, and you should think about linearity and avoid extrapolating beyond your data. Prediction does not require causation: reviews do not have to cause illness to be useful flags.
• Causal inference. Does taking calculus make students more successful? The correlation is a starting point, and treating it as causal requires assuming that students who take calculus are otherwise identical to those who do not in their underlying chances of success. Outside special circumstances this assumption is very hard to defend. Part III of the course builds better tools.

The assumptions escalate as you move down the list. That escalation is itself a favorite exam question.

Every formula, and what each one means

Building blocks for a single variable (note: this course divides by N, the population formula, never N - 1):

• Mean: μ_x = (Σ x_i) / N
• Variance: σ²_x = (Σ (x_i - μ_x)²) / N
• Standard deviation: σ_x = √(σ²_x)

Measures for two variables:

• Covariance: cov(x, y) = (Σ (x_i - μ_x)(y_i - μ_y)) / N. Its sign gives the direction of the correlation. Its size is hard to interpret because it depends on the units of both variables.
• Correlation coefficient: corr(x, y) = cov(x, y) / (σ_x σ_y). Unitless, always between -1 and 1. Measures the tightness of the linear relationship: how closely the points hug the line.
• Slope of the regression line: β = cov(x, y) / σ²_x. Measures the substantive magnitude: on average, for every one-unit increase in x, y is higher by β. It carries units (units of y per unit of x).

Tightness is different from magnitude

The slides show two scatterplots with similar tightness but slopes of 5.59 and 0.10. A correlation coefficient near 1 tells you the relationship is very consistent. It tells you nothing about whether the relationship is big enough to matter. A tight relationship can be substantively tiny, and a loose relationship can be substantively huge. When a question asks "how much does y change as x changes," the answer comes from the slope, never from the correlation coefficient. Also note: corr(x, y) is symmetric, while the slope changes if you swap x and y (the denominator becomes σ²_y).

Linearity, and what to do about it

Covariance, correlation, and the regression slope all measure linear relationships. Interesting relationships need not be linear: a strong U-shaped relationship can produce a correlation near zero. Two fixes from the lecture:

• Separately analyze subsets of the data where the relationship is roughly linear.
• Rescale or transform variables (logs are the classic example) so the relationship becomes approximately linear.

One more connection worth memorizing: when both variables are binary, the slope of the line of best fit equals the difference in conditional probabilities, β = Pr(y = 1 | x = 1) - Pr(y = 1 | x = 0). This is exactly how a 2x2 table turns into a regression coefficient on the midterm.

Classic traps and misconceptions

• The one-group trap. "Most X are Y" or "few X do Y" describes a single group. It is a descriptive fact until you compare against the other group.
• The no-variation trap (selecting on the dependent variable). "Every celebrity with great skin uses this product" examines only people with great skin. With no variation in one variable, a correlation cannot even be computed. More data of the same kind does not fix it.
• Wrong denominator. In a 2x2 table, dividing by the grand total or the wrong margin gives a plausible-looking wrong answer. Match the denominator to the group you condition on.
• N versus N - 1. Course formulas divide by N. Distractors are built from the N - 1 version and from forgetting the square root (reporting a variance as an SD).
• Correlation coefficient versus slope. The coefficient measures tightness, the slope measures magnitude. Swapping them is a standard distractor.
• Causal language. The slope means "associated with, on average." Any option saying the treatment "causes" or "increases" the outcome, from observational data, is wrong even when the number attached to it is right.
• Zero correlation means no linear relationship. It does not rule out a strong nonlinear one.

Worked example

Four stores, with advertising spend x (in $1000s) and weekly sales y (in $1000s):

Means: μ_x = 10/4 = 2.5, μ_y = 16/4 = 4. Covariance: cov(x, y) = (3 + 0 + 0 + 3)/4 = 1.5. Variance of x: σ²_x = (2.25 + 0.25 + 0.25 + 2.25)/4 = 1.25, so σ_x ≈ 1.12. Variance of y: σ²_y = (4 + 0 + 0 + 4)/4 = 2, so σ_y ≈ 1.41.

Correlation coefficient: corr = 1.5/(1.12 × 1.41) ≈ 0.95, a very tight positive relationship. Slope: β = 1.5/1.25 = 1.2. Interpretation: on average, each additional $1000 of advertising is associated with $1200 higher weekly sales. Notice the two numbers answer different questions, and notice that regressing x on y instead would give 1.5/2 = 0.75, a different slope, while the correlation coefficient would stay 0.95.

How the exam asks this

The professor's exercise deck and both midterms use one consistent format for this topic: a short realistic scenario (a Niagara Region policy office, an influencer's health claim, a business dataset), followed by four to eight interpretation-heavy multiple choice questions. Expect these specific patterns:

• "What are correlations good for?" with description, prediction, and causal starting point as options, and "all of the above" as the answer. Also its cousin: match a research question to descriptive, predictive, or causal.
• "Which of these are not correlations?" The giveaways are one-group statements ("most neighbourhoods that get opportunity zones are less well off", "few employers who have low taxes hire more workers") and no-variation claims. The real correlations use comparative language like "tend to" or "more likely."
• Small-table computations. A five-row dataset where you compute a subgroup mean or a subgroup standard deviation, dividing by N. Distractors: the variance without the square root, the N - 1 version, and a mean from the wrong subgroup.
• 2x2 table questions in pairs. First compute a conditional proportion (watch the denominator), then judge the sign of the correlation by comparing the two conditional probabilities. The trap answer restates the descriptive fact ("most OZ neighborhoods...") when the comparison actually flips the sign.
• Interpretation wording. The correct slope interpretation says "on average" and "associated with," in the right units (percentage points for binary outcomes). Distractors use causal verbs, percent instead of percentage points, or swap the correlation coefficient for the slope.
• Influencer-style claim autopsies. Identify the mistake: making a causal claim from a correlation, selecting on the dependent variable, or asserting a correlation with no variation in one variable. These appear nearly verbatim on Midterm 1.
• R code completion. Later chapters and both midterms expect you to know that the line of best fit is added with geom_smooth(method = "lm") on top of geom_point(), and that a slope comes from lm(y ~ x, data = df).

Lecture 3: Causation, Counterfactuals, and Potential Outcomes

What a causal effect actually is

A causal effect is a change in some feature of the world that would result from a change to some other feature of the world. The key word is "would": causation is defined by comparing two counterfactual worlds that are identical in every way except one.

The lecture's running example is Goop's "Body Vibes" stickers, which supposedly reduce anxiety and clear skin. Saying "Body Vibes reduce anxiety" means: if we compared two otherwise identical worlds, one where you wear the Body Vibes sticker and one where you wear a 10-cent Walmart sticker, your anxiety would be lower in the first world. Every marketing claim like this ("reduces stress", "boosts sales", "prevents cavities") is secretly a claim about a comparison between worlds we can never see side by side.

Potential outcomes notation

For a treatment T and outcome Y, each unit i (a person, a store, an intersection) carries two potential outcomes:

• Y1i = outcome for unit i if T = 1 (the treated world)
• Y0i = outcome for unit i if T = 0 (the untreated world)

The individual causal effect is the difference between them:

Effect of T on Y for unit i = Y1i - Y0i

When we average those individual effects over a group of N units, we get the average treatment effect:

ATE = (1/N) * sum over i of (Y1i - Y0i)

Order matters: it is treated minus untreated. Y0i - Y1i flips the sign, and Y1i + Y0i or Y1i / Y0i mean nothing in this framework. The professor's exercise deck tests exactly this with lookalike expressions. Note that on the course formula sheet, this chapter contributes the Y1i - Y0i logic; the covariance and regression formulas belong to other lectures.

The fundamental problem of causal inference

We can never observe a causal effect directly. If unit i is treated, we see Y1i and the untreated outcome Y0i becomes an unobservable counterfactual. If i is untreated, the reverse. Each unit reveals exactly one of its two potential outcomes, so Y1i - Y0i can never be computed from data for any single unit.

Two consequences:

• Individual causal effects exist and are perfectly well defined. They are just unobservable. (Do not confuse "cannot be known" with "does not exist".)
• Although we cannot know the effect for one individual, we can often credibly estimate the average effect across a group, which is where the rest of the course (randomization, controlling for confounders) comes in.

Sometimes the average is what we care about (the FDA deciding whether a drug is safe and effective for the population). Sometimes the individual effect is what we care about (a jury deciding whether one person's action harmed another specific person). Exams like to ask which one a given decision maker needs.

There is no such thing as "the" cause

What caused World War I? The assassination of Archduke Ferdinand? The alliance network? The July Ultimatum? The Paleozoic fish that swam right instead of left, allowing humans to evolve? There are infinitely many things that, had they been different, would have prevented the war. So there is no single "the cause" of any event. Causal questions must be posed as specific counterfactual contrasts ("what is the effect of this treatment"), never as a hunt for the one true cause.

Causes do not guarantee their effects

Your uncle says flu shots do not work because he got the shot and still got the flu. He is wrong, because causal effects are about probabilities and counterfactuals. The shot can lower the chance of getting the flu, or reduce its severity, without eliminating it. The right comparison is your uncle's outcome with the shot versus his outcome in the counterfactual world without it, and that second outcome might have been a worse flu, or the flu arriving sooner. One vaccinated person catching the flu falsifies nothing.

Time order does not establish causation

A tempting empirical strategy: if A tends to happen before B, conclude A causes B. Clive Granger won a Nobel Prize for causality tests built on this idea. But the lecture's counterexamples show why precedence alone fails:

• Christmas cards precede Christmas (cards do not cause Christmas).
• Anxiety precedes exams (anxiety does not cause the exam to happen).
• Beer and face paint precede the World Junior Hockey Championship.

The common thread is anticipation: when people expect a future event, they act before it arrives, so the "cause" can show up in the data ahead of the "effect" it did nothing to produce.

Causation does not require physical connection

We tend to picture causation as billiard balls: one thing physically pushing another. But a threat, a reminder, an advertisement, or an app notification can change behavior and therefore have a real causal effect, with no physical contact at all. What makes something causal is the counterfactual comparison, never the presence of a physical mechanism.

Causation need not imply correlation

You already know correlation does not imply causation. The reverse is also true: a genuine causal relationship can produce no correlation, or a correlation of the wrong sign.

The classic case: firefighters reduce fire damage, yet houses recently visited by firefighters show more fire damage. Why? Firefighters are sent to the worst fires. When a protective treatment is targeted at the worst cases, the raw correlation between treatment and bad outcomes turns positive even though the true effect is protective. Remember this pattern: it reappears whenever treatment is assigned based on how bad things are (nurses to outbreak zones, roundabouts to dangerous intersections, tutoring to struggling students).

Classic traps and misconceptions

• Before/after testimonials. "I used it and then I improved" ignores what would have happened without the treatment. Improvement can come from time passing, a temporary bad patch ending, or anything else.
• Confusing unobservable with undefined. Individual effects exist; the fundamental problem is that we cannot observe them.
• Demanding guarantees. A cause can shift probabilities without working every time for every person.
• Sign flips. Y0i - Y1i is a favorite wrong option; the effect is always treated minus untreated.
• Comparing across people. Subtracting one person's outcome from a different person's outcome is a comparison of people, never an individual causal effect.
• Reading the wrong columns. In a godlike potential outcomes table, real data would only reveal Y1i for treated rows and Y0i for untreated rows. Questions often hinge on whether you are asked about the full table or only what would be observed.
• Wrong-sign correlations. A positive correlation between a protective treatment and a bad outcome usually signals targeting, never proof of harm.

Worked example: godlike table for a tutoring program

Four students; T = 1 means the student attended tutoring. Suppose we magically know both potential final grades:

From the full table: ATE = (5 + 0 + 2 + 1) / 4 = 2. The effect on the treated (students A and B) is (5 + 0) / 2 = 2.5.

Now hide the counterfactual columns, as real data would. We observe grades 80 and 70 for the tutored group (mean 75) and grades 88 and 59 for the untutored group (mean 73.5). The observed gap is 75 - 73.5 = 1.5, which matches neither the ATE (2) nor the effect on the treated (2.5).

Why? The tutored students would have averaged (75 + 70) / 2 = 72.5 without tutoring, while the untutored students averaged 73.5 without it. The tutored group started slightly weaker, and that baseline difference of -1 drags the observed gap down: 1.5 = 2.5 + (-1). This is the seed of the ATT plus Bias decomposition that later lectures formalize.

How the exam asks this

The professor's Chapter 3 deck and both midterms recycle a small set of patterns for this topic:

• One scenario, many questions. A realistic claim (a mindfulness app, red light therapy influencer, fluoride debate) is followed by 4 to 8 MCQs that each probe one concept. Read the scenario once, carefully, because every question leans on it.
• "Which expression is the causal effect?" Options are algebra lookalikes: Y1i - Y0i (correct), Y0i - Y1i, Y1i + Y0i, Y1i / Y0i. Free marks if you know the convention.
• Agree/disagree with reasoning. Two options can share the same verdict ("Disagree because...") and only the reasoning separates them. The correct reasoning is almost always the counterfactual one: "we do not know what would have happened without the treatment."
• "Why can't this be causal evidence?" The correct answer names the unobserved counterfactual. Distractors blame sample size, measurement error, or missing regressions; those are all defensible-sounding and all wrong for this question type.
• Godlike Y1i/Y0i tables. You are shown both potential outcomes for 6 to 10 units and asked: which group looks better in the observed data, what is the overall (average) effect from the full table, and do those two answers disagree. Compute individual effects row by row, then average. Watch whether the question uses the whole table or only what would be observed.
• The trap gallery. Expect one question each on: causes not guaranteeing effects (the flu shot uncle), time order (Christmas cards logic, before/after influencer stories), physical connection (an app or message "can't" cause anything), and causation without correlation (the firefighters pattern).
• Bridge to later chapters. Midterm questions dress this material in SDM versus ATT numbers (the roundabout study computes both from a godlike table) and in claims like "this observed difference is the treatment effect." The Chapter 3 skill being tested is recognizing that the naive comparison and the causal effect are different objects.

Lecture 4: Correlation Requires Variation (Selecting on the Dependent Variable)

This lecture is about a mistake so common that entire best-selling books are built on it: claiming a correlation when your data cannot possibly show one. The core rule is simple: a correlation is a comparison, and a comparison requires variation in both variables. If everyone in your sample has the same outcome (or the same treatment), you have a descriptive fact, and a descriptive fact tells you nothing about how two things move together.

Correlation vs. descriptive fact

A descriptive fact summarizes one group: "most politicians facing a scandal win reelection." A correlation compares across groups: "politicians facing a scandal are reelected at a lower rate than politicians without a scandal." The first sentence sounds like evidence about scandals and reelection. It is only half of a comparison. To know whether scandal and reelection are correlated, you need the reelection rate for scandal politicians and for no-scandal politicians, then you compare.

Here is the full table from lecture:

Now compute the conditional probabilities (these are the formulas from the slides):

• Pr(Reelected | No scandal) = 1,192 / (1,192 + 101) ≈ .922
• Pr(Reelected | Scandal) = 62 / (62 + 8) ≈ .886

So "most scandal politicians win reelection" is true (.886 is most), yet scandal is negatively correlated with reelection, because .886 < .922. The headline fact and the correlation point in different directions. You can also flip the conditioning and get the same story:

• Pr(Scandal | Not reelected) = 8 / 109 ≈ .073
• Pr(Scandal | Reelected) = 62 / 1,254 ≈ .049

Politicians who lost were more likely to have had a scandal. Either direction of conditioning works, as long as you compare two groups.

Why variation is mathematically required

Recall from the formula sheet:

• cov(x, y) = Σ (x_i − μ_x)(y_i − μ_y) / N
• corr(x, y) = cov(x, y) / (σ_x σ_y)

If a variable does not vary, every x_i = μ_x, so every deviation (x_i − μ_x) is zero, the covariance is zero, and σ_x = 0 sits in the denominator. The correlation is literally undefined: you would be dividing zero by zero. This is the formal version of "you cannot compute a correlation from one column of the table."

Selecting on the dependent variable

This is the habit of building your sample using the outcome you care about, then looking for common features. Classic lecture examples:

• The 10,000 hour rule. Gladwell studied wildly successful people (Bill Gates, the Beatles) and found they all practiced 10,000+ hours. But he never looked at non-great achievers. Maybe many of them also practiced 10,000 hours and never became great. Without that row of the table, no correlation exists.
• The mysterious illness parable. Many sick people drank the beverage. Sounds damning, until you ask how many healthy people also drank it (if the beverage is water, everyone did).
• AC/DC and teen suicide. The 1985 Senate testimony pointed to suicidal teens who listened to AC/DC. Millions of non-suicidal teens also listened. Only the "Suicidal" column was examined.
• Suicide terrorism. Pape studied suicide attacks since 1980 and found foreign occupation present in most cases. He sampled on the outcome (attacks happened) and did not systematically examine occupied places with no suicide terrorism.

You can also select on the independent variable (only sampling people who got the treatment). Same disease, same cure: you are missing a row or column of the 2x2 table, so no comparison is possible.

Why smart people keep doing this

• It feels intuitive: "look at the successes and find what they share."
• The world is organized to feed us selected samples: doctors mostly see sick people (are herniated discs correlated with back pain? Many pain-free people have them too). We ask mentors for advice, and mentors are the successful survivors. Commissions investigate disasters (Challenger, the 2008 financial crisis) without a matched study of non-disasters.

Traps and misconceptions

• "Most X are Y" is not a correlation. It is a descriptive fact about one group. A correlation needs the rate of Y among not-X too.
• A high percentage proves nothing by itself. Pr(Reelected | Scandal) ≈ .886 sounds like scandals are harmless. The comparison to .922 shows the opposite.
• Selecting on the DV can even produce the wrong sign. Reduced-charge developments could look like winners among completed projects, while the full table shows the opposite.
• "We surveyed people who did X and most succeeded" has the same flaw whether X is a policy, a habit, or a product feature. Ask immediately: which cells of the 2x2 table were actually collected?
• Correlation direction depends on comparing conditional probabilities, never on cell counts alone. 62 reelected scandal politicians vs 8 defeated ones means nothing without the no-scandal column.
• This is a step before causality. Even a properly computed correlation is still just a correlation. Lecture 4's point is more basic: many famous claims fail before we even reach the causation question, because no correlation was ever established.

Worked example

A coding bootcamp advertises: "90% of our graduates work in tech." Is bootcamp attendance correlated with working in tech? You cannot tell yet: that is a descriptive fact about one group. Suppose you collect the full table for a city:

Compute both conditional probabilities:

• Pr(Tech | Bootcamp) = 90 / 100 = .90
• Pr(Tech | No bootcamp) = 400 / 500 = .80

Since .90 > .80, bootcamp attendance is positively correlated with tech employment (a 10 percentage point gap). Notice two things. First, the ad's "90%" was uninformative until you saw the baseline of .80. Second, if the no-bootcamp row had been 450 and 50, the baseline would also be .90 and the correlation would be exactly zero, with the identical headline. The headline never changes; the correlation lives entirely in the comparison.

How the exam asks this

The professor's Chapter 4 exercise deck and the midterms recycle a small set of question patterns for this topic. Expect these:

• "Which of these statements is actually a correlation?" Four statements about a scenario (housing policy, influencers, business data). Exactly one compares an outcome across levels of another variable ("areas with low charges tend to have more permits"). The distractors are descriptive facts about a single group ("most new units are condos") or superlatives about one case.
• "Do you agree?" claims. Someone surveys only people who did the thing (developers who built, celebrities with great skin, failed initiatives) and announces a correlation or a recommendation. The correct answer identifies the missing group ("the sample only includes those who built, leaving out those who did not"). Watch for overcorrecting distractors like "correlation can never be identified in surveys": too strong, and wrong.
• "Which cells of the 2x2 table were actually collected?" Given rows (outcome) and columns (treatment) labeled A, B, C, D, identify which cells the flawed sample fills in. Selecting on the DV fills one row; selecting on the IV fills one column.
• Compute a conditional probability from a full table. Midterm 1 style: "What is the probability of tooth decay for municipal water children?" You must divide the cell by its own group total (row or column as appropriate). Distractors use the wrong denominator: the other group, the grand total (a joint probability), or the flipped conditioning.
• Interpret the comparison. After computing both probabilities, pick the statement that describes the relationship as an association ("less likely to experience tooth decay"), rejecting causal wording and rejecting "no relationship can be described without a regression."
• Name the mistake. "Lots of celebrities with great skin use red light therapy, so you should too" is selecting on the dependent variable, and the follow-up asks why it is not a correlation: because there is no variation in one of the variables.
• Failure post-mortems. "All our failed projects had feature X, so avoid X." Correct answer: the successful projects might have had X too; without them there is no comparison.

Speed tips: always identify the dependent variable first, then ask whether the sample contains both values of it. When a table appears, write the two conditional probabilities in the margin before reading the answer options. When a claim starts with "most" or "all of the [successes/failures]," flag it as a candidate descriptive fact or DV-selection problem.

Lecture 5: Regression for Description and Forecasting

This lecture takes the correlation tools from earlier chapters and upgrades them to the single most important tool in the course: linear regression. The framing matters. Correlations can be useful for three jobs (description, forecasting, and causal inference), and this lecture uses regression only for the first two. Causal inference comes later in the course. So every interpretation you write for this material should sound like an association, never like an effect.

The regression equation, piece by piece

The lecture's running example regresses daily reported crime in Chicago on temperature:

Crime_i = α + β * Temperature_i + ε_i

• Crime_i is the dependent variable (also called the outcome, the Y variable, the thing you want to describe or forecast).
• Temperature_i is the independent variable (the X variable, the predictor).
• α is the intercept or constant: the predicted value of Y when X equals zero.
• β is the slope: how much the predicted value of Y changes when X increases by one unit.
• ε_i is the error term: everything that affects Y for observation i that the model does not capture. It is emphatically more than "random noise." For crime, ε holds day of week, policing, events, and every other omitted influence.

Move the error term to the other side and you get the fitted (predicted) value, the point on the line:

Crime_i − ε_i = α + β * Temperature_i

PredictedCrime_i = α + β * Temperature_i, also written Ĉrime_i = α + β * Temperature_i

So each observation splits into two parts: the part the line explains (the prediction) and the part it misses (the error). A hat over a variable always means "predicted by the model."

Which line? The one that fits best

You could draw many candidate lines through a scatterplot. OLS picks the line that minimizes the sum of squared errors, and the resulting slope has a closed form that connects regression back to covariance (this is on the formula sheet):

β = cov(x, y) / σ²_x

Compare that with the correlation coefficient, corr(x, y) = cov(x, y) / (σ_x * σ_y). Both are built from the same covariance. The correlation is unit free and bounded between −1 and 1, while the regression slope is in the units of Y per unit of X, which is exactly what makes it useful for prediction.

Reading a regression table

The lecture estimates turnout on age with 70 observations and reports:

Numbers in parentheses are standard errors, a measure of how precisely each coefficient is estimated. Some tables instead report an Estimate column, a Std. Error column, and a t value column, where t value = Estimate / Std. Error. The formula sheet gives you the quick 95 percent confidence interval:

CI95 = [β̂ − 2 * SE(β̂), β̂ + 2 * SE(β̂)]

R2 tells you the share of variation in Y that the model accounts for in this sample. It says nothing about causality and nothing about how the model will do on new data.

The R workflow you must recognize

Exam questions quote this code and ask you to complete or debug it:

# load tidyverse
library(tidyverse)

# read in data
chi = read.csv("ChicagoCrimeTemperature2018.csv")

# new temp variable in celsius
chi = chi |> mutate(tempc = (5/9)*(temp-32))

# make a scatter plot with a fitted line
ggplot(aes(x = tempc, y = crimes), data = chi) +
  geom_point() +
  geom_smooth(method = "lm", se = FALSE)

# run a regression
model = lm(crimes ~ tempc, data = chi)

# make a table
library(modelsummary)
modelsummary(model,
             vcov = "HC1",
             statistic = "({std.error})",
             gof_map = c("nobs", "r.squared"))

Key patterns: lm(y ~ x, data = df) puts the dependent variable before the tilde. geom_point() draws the scatter, geom_smooth(method = "lm") adds the regression line. If R says it "could not find function ggplot," you forgot library(tidyverse) (or library(ggplot2)).

When a line is the wrong shape: polynomial regression

Turnout rises with age among younger voters and then flattens or falls at older ages, so one straight line fits poorly at the extremes. Two fixes from the lecture: run separate linear regressions on subsets of the data, or fit a polynomial by adding powers of X:

Turnout_i = α + β1 * Age_i + β2 * Age_i^2 + ... + βN * Age_i^N + ε_i

Higher-order polynomials always fit the sample at least as well (in-sample fit can only improve as you add terms). That is precisely the danger.

Overfitting and out-of-sample testing

A wiggly high-degree polynomial chases the noise in your particular sample. It looks brilliant in-sample and can forecast terribly on fresh data. The honest referee is out-of-sample testing: fit the model on one portion of the data, then check how well it predicts observations it never saw. The lecture's demonstration shows flexible models winning in-sample while a simpler model wins out-of-sample. The tradeoff to memorize: complex models may fit the sample better but can overfit and perform poorly on new data; simpler models may underfit but are easier to interpret and communicate.

Classic traps and misconceptions

• Causal language. "β = −0.68, so opportunity zones decrease business." Wrong. A descriptive regression shows that the groups differ on average; it cannot say the treatment produced the difference. A large t value does nothing to rescue a causal claim.
• Swapping the variables. The dependent variable is the outcome being predicted; it goes on the left of the equation and before the tilde in lm(). Distractors love to reverse X and Y or to claim the "t value" is a variable in the model.
• Misreading the table. The estimate is the coefficient. The number in parentheses (or the Std. Error column) measures precision. Do wrong-column arithmetic and you get distractor answers.
• Intercept overreach. The intercept is the predicted Y when X = 0. For a binary X, that is the average for the group coded 0, which is meaningful. When X = 0 sits outside the data (age 0), the intercept is a line-anchoring device and should be interpreted with care, though "it is meaningless and should be ignored" is also wrong.
• Deterministic readings. "All neighborhoods start with exactly 1.43 businesses" or "OZ neighborhoods always have exactly 0.68." Regression describes averages; individual observations scatter around the line by ε.
• The error term is "just noise." It contains every omitted factor that moves Y: income, zoning, demand, and so on. That is why omitting it from your thinking sets up the omitted variable bias material later.
• Extrapolation. A line fit mostly on 20-to-40-year-olds gives confident predictions for 78-year-olds that can be badly wrong if the true relationship bends.
• Chasing R2. Adding polynomial terms to push R2 toward 1 is the textbook path to overfitting. In-sample fit is the wrong criterion for forecasting.

Worked example: predict turnout

Use the table above: T̂urnout_i = 0.145 + 0.004 * Age_i.

• For a 31-year-old: 0.145 + 0.004 * 31 = 0.145 + 0.124 = 0.269, a predicted turnout of about 26.9 percent.
• For a 78-year-old: 0.145 + 0.004 * 78 = 0.145 + 0.312 = 0.457, about 45.7 percent. Treat this with suspicion: if turnout flattens or declines at older ages, the straight line extrapolates past what the data support, which is exactly why the lecture turns to polynomials and out-of-sample checks.
• Interpretation of the slope: each additional year of age is associated with a 0.4 percentage point higher predicted turnout, on average. Interpretation of the intercept: the predicted turnout at age 0 is 14.5 percent, an anchor for the line rather than a claim about newborn voters.

How the exam asks this

The professor's chapter 5 exercise deck and both midterm practice sets recycle a small number of question patterns. Drill these until they are automatic.

• Scenario plus regression table. You get a realistic setup (a municipal policy analyst studying opportunity zones, a fluoride and tooth decay dataset, a transit or retail question) and a table with Estimate, Std. Error, and t value columns. Then 4 to 8 questions walk the same ladder: identify the dependent and independent variables, pick the regression equation that matches the table, compute a predicted value by plugging in X, interpret the intercept, interpret the slope.
• Prediction arithmetic with binary X. "What is the predicted outcome for a treated unit?" means intercept plus slope. Distractors include the intercept alone, the slope alone, and numbers built from the wrong columns.
• "How do you respond?" dialogue questions. A manager or colleague overclaims ("this proves OZs reduce business") and the correct answer states that the regression shows an association while remaining open that regression is still informative. Wrong answers either endorse the causal claim, lean on a big t value as proof of causality, or swing too far and deny that regression shows anything.
• Error term questions. A colleague calls ε "just random noise" and the right answer lists concrete omitted factors that influence the outcome.
• Model complexity questions. Someone proposes stacking polynomial terms for a "perfect" model, and the credited answer states the overfitting tradeoff and points to out-of-sample performance.
• R code completion and debugging. Fill in geom_smooth(method = "lm"), write lm(y ~ x, data = df) with the correct variable order, know that a "could not find function" error means a missing library() call, and recognize one-liners like filter() and mutate().
• Precision and inference add-ons. Questions about what happens to standard errors when the sample shrinks (they widen, estimates stay unbiased), what a 95 percent CI does and does not mean (it is a procedure that captures the truth in 95 percent of repeated samples; it is never "a 95 percent probability the truth is in this interval"), and what a p-value is (the probability of a result this extreme if the null were true, never the probability the null is true).

Lecture 6: Samples, Uncertainty, and Statistical Inference

This lecture answers one question: when you compute a number from a sample, how much should you trust it? The whole toolkit (standard errors, confidence intervals, hypothesis tests, p-values) exists to separate genuine relationships from luck.

The favourite equation

Estimate = Estimand + Bias + Noise

• Estimate: the result you actually got from your data.
• Estimand: the thing you were hoping to measure (the true quantity in the population).
• Bias: systematic reasons your estimate is consistently off the mark (a flawed design pushes you in the same direction every time).
• Noise: idiosyncratic, random reasons your estimate is off. Noise is zero on average, and in any single sample it can push you above or below the truth.

Polling version: the true share of all voters supporting a candidate is q (the estimand). You poll a random sample and compute the sample share q̂ (the estimate). Then q̂ = q + bias + noise.

Estimand, estimator, estimate

Three different words, and the exam checks all three. The estimand is the target (true share q). The estimator is the procedure or recipe (collect a random sample of N voters, ask who they support, compute the average). The estimate is the number that recipe produced this time (q̂, say 0.532). A good estimator is unbiased (right on average) and precise (low noise). Think of a dartboard: bias is whether your throws center on the bullseye, precision is how tightly they cluster. You can be unbiased yet imprecise, or precise yet biased, and those are different problems with different cures.

Standard error: quantifying precision

The standard error is the standard deviation of the distribution of estimates your estimator could have produced across many hypothetical samples. For a polled proportion:

SE ≈ sqrt(q(1 - q) / N)

• Larger N shrinks the SE, with diminishing marginal returns: the SE falls with the square root of N, so quadrupling the sample only halves the SE. By around N = 10,000 sampling variability is nearly negligible, even for an infinitely large population. The size of the population itself does not appear in the formula.
• q(1 - q) is largest at q = 0.5, so a 50/50 split is the noisiest case.
• Fun catch from the slides: the formula needs q, which is exactly what you do not know (that is why you ran the survey). In practice you plug in q̂, or use 0.5 as the conservative worst case.

Confidence intervals and the margin of error

The Central Limit Theorem says the distribution of estimates from your estimator is approximately normal. In a normal distribution, 95% of draws fall within about two standard deviations of the mean. Pollsters exploit this:

margin of error = 2 * SE

CI95 = [estimate - 2*SE, estimate + 2*SE]

On the formula sheet, for a regression coefficient: CI95 = [β̂ - 2*SE(β̂), β̂ + 2*SE(β̂)]

What the 95% CI actually means: if you ran the same unbiased poll over and over, the truth would land inside the interval you build 95% of the time. Coverage is a property of the procedure, repeated across samples. It does not mean there is a 95% chance the truth sits inside the one interval you computed, and it says nothing at all about bias: a biased poll with a tiny margin of error is just precisely wrong.

Hypothesis testing and p-values

The logic runs in reverse: assume a boring world (the null hypothesis), then ask how surprising your estimate would be in that world. Lecture example: an unbiased poll of 1,000 voters finds 532 supporting the Republican, so q̂ = 0.532. Null: the election is a virtual tie, q = 0.5. Under the null, SE = sqrt(0.5(1 - 0.5)/1000) ≈ 0.016. The estimate sits

(0.532 - 0.5) / 0.016 = 2

standard errors above the null mean. A result at least that favourable to the Republican happens only 2.5% of the time by chance (95% of draws are within two SEs, and we are in one tail). So the p-value is 0.025. Since that is small, we reject the null and say we have statistically significant evidence the Republican leads. The general recipe: compute (estimate - null value) / SE, and if you are more than about 2 SEs out, p is below 0.05.

The same machinery works for regression. If the population relationship is Income_i = α + β * YearsOfEducation_i + ε_i, where α and β minimize the sum of squared errors in the whole population, then β is the estimand. Your sample regression gives β̂, which differs from β because of noise. Software estimates SE(β̂) for you, and from there you build CIs and run tests exactly as with the poll.

What if you have the entire population?

Suppose you have data on every student at the university and you correlate GPA with varsity sports. No sampling variability, so is inference pointless? The course's answer: noise is broader than sampling error. The data you observe are still one draw from many ways the world could have unfolded (idiosyncratic shocks to grades, injuries, schedules), so standard errors and tests still help you judge whether a pattern reflects a stable relationship or a fluke of this particular realization.

Substantive vs statistical significance

Statistical significance asks: is this estimate attributable to chance? Substantive significance asks: is the quantity big enough to matter? These are independent. The slides give both failure modes:

• A precise estimate of a tiny relationship gets misread as important (social media voting nudges: statistically significant, practically trivial).
• An imprecise estimate of a large relationship gets misread as a null result (did the Second Reform Act affect elections? A wide CI is weak evidence, and weak evidence of an effect differs from evidence of no effect).

Classic traps

• The p-value trap: a p-value of 0.038 is the probability of seeing an estimate at least that extreme by chance if the null hypothesis is true. It is never the probability that the null is true, and never the probability your result is wrong. Exam options differ only in the conditioning clause, so read the last five words of every option.
• The CI trap: the correct reading is about repeated sampling and requires the poll to be unbiased. Any option saying the truth "is in" this specific interval, or assigning a 95% probability to this specific interval, is wrong.
• The big-N trap: increasing the sample size shrinks noise and does nothing to bias. Rerun a flawed study (health-conscious volunteers only) with 10 times the sample and you get the same biased answer with tighter error bars.
• Significant means important: a p-value below 0.05 guarantees nothing about magnitude. Always compare the effect size to a meaningful benchmark.
• Extrapolation: a study of 1200mg of turmeric says nothing reliable about a teaspoon. Estimates apply to the conditions actually studied.
• SE arithmetic slips: forgetting the square root (that gives the variance), or dividing by sqrt(N) in the wrong place. Write out sqrt(q(1 - q)/N) and substitute step by step.

Worked example

A city polls a random sample of 2,500 voters about a ballot measure; 1,325 say yes.

• Estimate: q̂ = 1325 / 2500 = 0.53.
• Standard error: SE = sqrt(0.53 * 0.47 / 2500) = sqrt(0.0000996) ≈ 0.01.
• Margin of error: 2 * 0.01 = 0.02, so CI95 = [0.51, 0.55].
• Test the null of a tie (q = 0.5): the estimate is (0.53 - 0.50) / 0.01 = 3 standard errors above the null. Results that extreme arise well under 1% of the time by chance, so p is far below 0.05: reject the null. There is statistically significant evidence the yes side leads.
• Interpretation check: with an unbiased design, intervals built this way would capture the true share in 95% of repeated polls. If the sample had been 400 voters instead, the SE would have been sqrt(0.25/400) = 0.025, the estimate would sit only 1.2 SEs above a tie, and we could no longer reject it. Same point estimate, different conclusion, purely because of precision.

How the exam asks this

The professor's chapter 6 deck and both midterms recycle a small set of patterns. Expect a realistic scenario block (a TikTok influencer citing a study, a poll, a policy dataset) reporting a design, an effect size, a 95% CI, and a p-value, followed by 4 to 8 interpretation MCQs:

• "What is the null hypothesis?" with options like β = 1, β > 0, β = 0, and β equal to the estimate. The null is almost always no effect, β = 0 (or a tied election, q = 0.5).
• p-value definition: four near-identical sentences where only one conditions correctly on the null being true. Distractors say "chance the null is true", drop the conditioning clause, or condition on the null being false.
• CI definition: the repeated-sampling wording is correct; distractors assert the truth is inside this interval or that the CI proves practical importance.
• Bias vs noise from a design flaw: unrepresentative recruitment introduces bias; then a follow-up asks what a bigger sample does (equally biased, more precise; "larger samples always remove bias" is a stock wrong option).
• Substantive vs statistical significance: a significant but tiny effect (the -0.5% turmeric result) that an influencer oversells, or the reverse error of calling an imprecise large estimate a null result.
• Arithmetic on regression tables with Estimate, Std. Error, and t value columns: build the CI as estimate plus or minus 2 SEs, check whether it excludes zero, and predict that shrinking N widens the SEs while leaving the point estimates unbiased.
• Extrapolation traps: what the study does and does not support about doses, groups, or settings it never tested.
• Vocabulary: match estimand, estimator, and estimate to their definitions inside the scenario.

Lecture 7: Over-Comparing and Under-Reporting (P-Hacking and Publication Bias)

The whole lecture compresses into one line the professor puts on its own slide:

multiple testing + selective reporting = unreliable results

Hypothesis tests are designed to control the false positive rate for one test. The moment many tests get run and only the interesting ones get shown to you, that control breaks down. A "significant" result you are shown may just be the lucky survivor of dozens of tests you never saw.

Where this sits in the course

Lecture 6 gave you the machinery: p-values, hypothesis tests, confidence intervals. Lecture 7 lists why that machinery, even used correctly, can mislead:

• Tests produce false positives and false negatives.
• The p-value does not tell you the probability that the null is true. It tells you how likely data at least this extreme would be if the null were true.
• Statistical significance and substantive significance are different things. A tiny effect can be significant in a huge sample.
• The p < .05 threshold is arbitrary. Nothing magical happens at .05.
• And the big new problem: we do not get to see all the tests that were conducted. Hidden tests make the p-values we do see misleading and inflate false positives.

Paul the Octopus: one example, three p-values

In 2008 and 2010, Paul the octopus correctly predicted 12 of 14 national soccer matches. Test the null that Paul is a coin flipper. The tool is the binomial probability:

Pr(k successes in n trials) = p^k * (1 - p)^(n - k) * n! / (k! * (n - k)!)

Step 1, naive test (p = 1/2, n = 14). The p-value is the probability of 12 or more correct under the null:

• Pr(12) = (1/2)^12 * (1/2)^(14-12) * 14! / (12!(14-12)!) ≈ .0056
• Pr(13) = (1/2)^13 * (1/2)^(14-13) * 14! / (13!(14-13)!) ≈ .0009
• Pr(14) = (1/2)^14 * (1/2)^(14-14) * 14! / (14!(14-14)!) ≈ .00006
• p ≈ .0065, about 1 in 155. Looks like strong evidence of octopus clairvoyance.

Step 2, fix the null. The coin-flip null was wrong. Paul mostly predicted Germany games (13 of 14), he tended to pick Germany (11 of 13), and Germany usually wins (9 of 13). So a mindless Paul gets a game right with probability:

(11/13)(9/13) + (2/13)(4/13) ≈ .633

Redo the test with p = .633 over the 13 Germany games:

• Pr(11) = .633^11 * .367^2 * 13! / (11! 2!) ≈ .069
• Pr(12) = .633^12 * .367^1 * 13! / (12! 1!) ≈ .020
• Pr(13) = .633^13 * .367^0 * 13! / (13! 0!) ≈ .002
• p ≈ .091, about 1 in 11. An alternative version (games fixed, Paul flips a weighted coin picking Germany with probability 11/13) gives p ≈ .030. Either way, far less impressive than 1 in 155.

Step 3, count the other octopuses. Even p ≈ .03 assumes Paul was the only animal being tested. If 10 octopuses were all guessing, the chance that at least one does as well as Paul under the null is:

Pr(at least one as good as Paul | null) = 1 - (1 - .03)^10 ≈ .26

And we know Leon the porcupine, Petty the hippopotamus, Anton the tamarin, and Mani the parakeet were also predicting matches, plus countless animals we never heard about because they guessed wrong. We only heard about Paul because he was the lucky one. That selection destroys the meaning of his p-value.

The general at-least-one formula

Memorize this move, it appears constantly:

Pr(at least one significant result | all nulls true) = 1 - (1 - p)^k

where p is the per-test false positive rate (often .05) and k is the number of independent tests. With k = 20 tests at the .05 level, that is 1 - .95^20 ≈ .64: a false positive is more likely than not.

Two ways publication bias arises

• P-hacking: an analyst knows a significant result is more interesting, exciting, and publishable, so they play around with the sample, the specification, the outcome, until something crosses p < .05, and they report only that one. This is the dishonest (or self-deceiving) route.
• P-screening: every researcher is honest and runs exactly one planned test. But journals, editors, and journalists are more likely to publish and publicize the significant results. The filter sits downstream of the researcher.

Key point: both mechanisms produce the same damage, lots of false positives and overestimated effects. You do not need any fraud for the published record to be badly distorted.

Real-world evidence

• Get-out-the-vote experiments: looking only at published studies, the average estimated effect is about 3.3 percentage points (Enos, Fowler, and Vavreck 2014). Pooling over 200 published and unpublished studies, the average drops to about 0.5 percentage points (Green, McGrath, and Aronow 2013). Same literature, roughly a sixfold overstatement from screening alone.
• Bill Miller: Legg Mason Value Trust beat the market 15 years in a row. Under the efficient market hypothesis, each year is a fair coin flip, so Pr(15 heads in a row | null) = (1/2)^15 ≈ 1 in 30,000. Sounds superhuman. But about 24,000 funds trade in a given year, so the chance that some fund posts that streak is 1 - (1 - (1/2)^15)^24,000 ≈ .52. A coin-flip world produces a Bill Miller about half the time, and the financial press guarantees you will hear about him.

Potential solutions (the slide list)

• Skepticism as a default posture.
• Test important and plausible hypotheses rather than cute ones.
• Adjust p-values for multiple testing (raise the bar when many tests are run).
• Replication: a true effect should show up again in fresh data.
• Further investigation motivated by theory.
• Preregistration: publicly commit to the hypotheses, outcomes, and analysis before seeing the data, which removes the room to hack.
• Possibly stricter thresholds (p < .005 has been proposed), or de-emphasizing significance altogether.

Classic traps and misconceptions

• "p < .05 means there is less than a 5% chance the null is true." Wrong, and it is the single most-tested misconception in this course. The p-value conditions on the null being true and describes the data. It says nothing direct about the probability of the null.
• Judging a result without asking how many tests were run. A p-value of .01 from one preregistered test and a p-value of .01 that survived from 25 outcomes are wildly different pieces of evidence.
• Thinking the per-test error rate is the overall error rate. Each test has a 5% false positive risk, but across 20 tests the chance of at least one false positive is about 64%, from 1 - .95^20.
• Believing publication bias requires cheating. P-screening distorts the literature even when every individual researcher is honest.
• Wrong null model. Paul at 1 in 155 assumed a fair coin. Building in Germany's win rate and Paul's Germany preference moved the p-value to around .03 to .09. Always ask whether the null actually describes "no skill" in context.
• Survivorship: you observe the winners (Paul, Bill Miller, the viral study) precisely because they won. The losers are invisible, so the sample you see is selected on success.
• Treating preregistration as a guarantee. It removes analyst flexibility, which is huge, but a preregistered study can still produce a false positive by chance, and a mid-study change to the primary outcome (as in the professor's exercise scenario) reintroduces the p-hacking worry.
• Concluding the opposite too fast. One significant outcome among 25 is weak evidence, but it does not prove a false positive, and it does not prove fraud. The measured response is caution plus replication.

Worked example: the sports pundit

A TV pundit picks the winner in 10 playoff games and gets 9 right. Her network claims this proves genuine expertise, since a guesser would almost never do that.

Step 1: p-value under the null of pure guessing (p = 1/2, n = 10).

• Pr(9) = (1/2)^9 * (1/2)^1 * 10! / (9! 1!) = 10/1024 ≈ .0098
• Pr(10) = (1/2)^10 = 1/1024 ≈ .0010
• p = Pr(9) + Pr(10) ≈ .011

Taken alone, we would reject the null at the 5% level. She looks like a genuine expert.

Step 2: count the pundits. Around 50 pundits across TV, radio, and podcasts made picks for the same games. If all 50 are pure guessers, the chance that at least one compiles a record this good is:

1 - (1 - .011)^50 ≈ .42

So a field of clueless pundits produces a "9 out of 10 genius" about 42% of the time, and the network only hands a victory-lap segment to the one who got lucky. The impressive individual p-value of .011 was real arithmetic applied to a selected survivor, which makes it close to worthless as evidence of skill. The right responses: ask how many pundits were picking, adjust for the multiple comparisons, and see whether she can repeat the performance next season (replication).

How the exam asks this

The professor's own exercise deck for this chapter is one long scenario (a Niagara Region randomized pilot of a Rapid Housing Support program) followed by a chain of interpretation MCQs, and the midterms use the same template. Expect these patterns:

• "Would you like to know more?" questions: given a study with an impressive result, which extra information matters? Correct answers involve how many outcomes were examined, whether the study was preregistered, and whether one primary outcome was named in advance.
• Multiple-outcome confidence questions: a trial measures 25 outcomes and exactly one is significant. The right answer is that confidence should fall because many tests inflate the false positive risk. Distractors claim more outcomes make the study "more comprehensive," that significance is significance regardless, or that the other 24 effects are proven zero.
• Preregistration chains: first a question where a preregistered primary outcome being the significant one raises confidence, then a twist (an audit shows the primary outcome was switched mid-study) where the right answer is that this looks like p-hacking even if the team claims innocence. Distractors take the extreme positions: "researchers can update plans freely" or "any change is fraud."
• Advice-to-decision-maker finales: a council or executive asks whether to scale the program. The credited answer is always the measured middle: promising evidence, replicate before scaling. Distractors are the overconfident ("p < .05, it worked"), the dismissive ("certainly a false positive"), and the accusatory ("any revision means fraud").
• Calculation items: binomial streak probabilities like (1/2)^10, and at-least-one computations like 1 - (1 - p)^k for octopuses, mutual funds, or 20 outcome tests. Know these cold and be ready to identify which wrong option corresponds to which error (per-test rate, complement forgotten, wrong exponent).
• P-value meaning questions: a character asserts "p < .001 means a 0.1% chance the effect is zero" and you pick the response that correctly states the conditional: if the null were true, data this extreme would be very unlikely. This exact pattern appears on both practice midterms.
• Publication bias magnitude questions: published studies average one effect size, the full registry including unpublished studies averages something much smaller (the 3.3 versus 0.5 get-out-the-vote contrast), and you identify selective publication as the explanation.
• Recycled mechanics inside the scenario: regression tables with Estimate, Std. Error, and t value to interpret, and short R snippets (a filter(p_value < 0.05) pipeline is a natural way to dress up selective reporting as a code question).

Lecture 8: Reversion to the Mean

First, a quick upgrade to confidence intervals

The lecture opens by tightening up the CI formula you have been using:

[β̂₁ - 2 × SE(β̂₁), β̂₁ + 2 × SE(β̂₁)]

The 2 is a rough guide. Technically it should be 1.96, and 1.96 is itself just the t-critical value when the sample is large. The fully general two-tailed version is:

[β̂₁ - t(1-α/2, df) × SE(β̂₁), β̂₁ + t(1-α/2, df) × SE(β̂₁)]

For a one-tailed test the critical value is t(1-α, df) instead. Three things to know cold:

• α is the significance level (0.05 for a 95 percent CI).
• df (degrees of freedom) = number of observations minus number of coefficients, and the count of coefficients includes the intercept. A simple regression with one slope on n observations has df = n - 2.
• Why 1 - α/2 for two tails: at the 5 percent level you reject if your t-statistic lands in the most extreme 5 percent of the distribution, and a two-tailed test splits that into 2.5 percent on each side. So you look up the 97.5th percentile, written t(.975, df).

Course convention: assume the sample is large enough that 2 works fine. But the exam can ask you why 2 is an approximation, what it approximates (1.96, the large-sample t-critical value), and how df is counted.

The big idea: every outcome is signal plus noise

Almost any outcome you measure has two parts. Signal is the stable, systematic component (a school's true quality, a person's true height, a store's underlying customer base). Noise is transient luck (a flu outbreak on test day, a good night's sleep, one big tour bus of shoppers).

When you observe an extreme outcome, it is usually extreme for both reasons at once: strong signal and a lucky (or unlucky) noise draw. The signal sticks around for the next measurement. The noise does not, because a fresh noise draw is just as likely to go the other way. So the next measurement is expected to be less extreme, closer to the mean. That is reversion to the mean.

Same idea, three historical names you may see: reversion (or regression) to the mean, Galton's regression to mediocrity (exceptionally tall parents tend to have somewhat shorter children), and cosmic habituation (the eerie pattern where striking scientific findings shrink when researchers try to replicate them, as if the universe were getting used to being studied; the boring explanation is that the original result was an extreme draw, partly noise).

The one required condition

Reversion to the mean happens if, and only if, outcomes contain both signal and noise.

• Pure signal, no noise: no reversion. Measure an adult's height twice, three weeks apart, and the second measurement matches the first. Identical conditions give identical results.
• Pure noise, no signal: complete reversion. Ask people for an arbitrary number from 1 to 100 in two waves. Wave 1 tells you nothing about wave 2, so your best guess for everyone in wave 2 is simply the mean.
• A mix: partial reversion, and the noisier the outcome, the stronger the reversion.

The lecture's University of Chicago panel (two survey waves, three weeks apart) is the canonical illustration:

It is a statistical pattern, never a force

Nothing pulls extreme values back toward the mean. There is no gravity, no balancing pressure, no cosmic thermostat. The mechanism is only this: the transient luck that helped make the first observation extreme is unlikely to repeat. Two consequences the exam loves:

• It runs in both directions in time. If John Junior is exceptionally tall, your best guess is that his son is shorter than him, and also that his father is shorter than him. A causal force could only work forward; a statistical pattern of shared signal plus independent noise is symmetric.
• The population does not compress toward the average. Individuals with extreme draws revert, while other individuals get fresh extreme draws, so the overall spread stays the same generation after generation. Concluding that everyone will eventually be mediocre is the classic misreading of Galton.

Why it fools people: interventions are triggered by extremes

We act exactly when things are unusually bad (or pick winners when things are unusually good). Then noise fades, outcomes drift back toward normal, and the action gets the credit. The lecture's list, all of which are at least partly mean reversion:

• COVID-19 cases declined after states implemented shelter-in-place orders (orders came at the peak).
• Patients report less pain after knee surgery (surgery happens when pain is at its worst).
• Your headache went away after you took an aspirin (you took it at peak pain).
• Sports teams win more after firing their coach (coaches get fired during unusually bad streaks).
• Crime fell in NYC precincts targeted for extra policing (targets were the precincts with extreme crime spikes).
• People report less depression after starting medication (they start at a low point).
• The body appears to heal itself under a placebo (people enroll in trials when symptoms peak).

None of this proves the interventions do nothing. It means the naive before-and-after comparison is contaminated: in the decomposition Ȳ₁ᵀ - Ȳ₀ᵁ = ATT + Bias + Noise, selecting units because their outcomes were extreme (and extreme partly through noise) plants reversion inside the Bias term. The improvement you observe is the true effect plus the recovery that would have happened anyway.

Related trap: "cracking under pressure" and the Sports Illustrated cover jinx. An athlete gets studied (or put on a magazine cover) right after an extreme hot streak. The next stretch looks worse. Probably not choking, probably reversion.

Spotting reversion in a regression or a scatterplot

Plot wave-2 outcomes against wave-1 outcomes and fit a line. The course slope formula applies as always:

β = cov(x, y) / σ²ₓ

A fitted slope below 1 (a line flatter than the 45-degree line) is the signature of reversion: units that were far from the mean in wave 1 are predicted closer to the mean in wave 2. With equal variance in the two waves and independent noise, the slope equals the signal share of total variance:

slope = σ²(signal) / (σ²(signal) + σ²(noise))

So: all signal gives slope 1 (no reversion), all noise gives slope 0 (full reversion, the flat line at the mean), and a flatter line means a noisier variable. When a figure shows two groups, the group with the flatter fitted line is the group with more reversion. Careful: the test for reversion compares the slope to 1, while the default t-statistic in R output tests against 0.

Worked example

A university surveys students' happiness (1 to 10) in two waves, three weeks apart. Suppose the stable disposition component has variance 4 and the day-to-day noise component has variance 4. Then the slope from regressing wave 2 on wave 1 is:

slope = 4 / (4 + 4) = 0.5

The class mean is 6. A student who reported 9 in wave 1 (3 points above the mean) has an expected wave-2 score of:

6 + 0.5 × (9 - 6) = 7.5

Expected to drop 1.5 points with no intervention at all. Now imagine the university had enrolled every student who scored 3 or below in a wellness program. Those students sit far below the mean, so the same arithmetic predicts sizable improvement for them by wave 2 even if the program is useless. Contrast with height: noise variance is nearly 0, slope is nearly 1, and a student measured 3 inches above the mean is expected to be 3 inches above the mean again. And for the arbitrary number, signal variance is 0, slope is 0, and the best wave-2 prediction for everyone is the mean, roughly 50.5.

Classic traps and misconceptions

• Treating reversion as a force. "High prices are being pulled down toward the average like gravity." Wrong: outcomes move toward the mean only because random noise fades.
• Crediting a policy for improvement in units selected on extreme outcomes. If the treated zones, schools, or patients were chosen because they were extreme, part (possibly all) of the observed improvement is reversion.
• Expecting reversion everywhere. Variables that are essentially all signal (height, a store's location) barely revert. Saying "public opinion will revert just like prices" fails if opinion is mostly systematic.
• Thinking reversion shrinks the population toward the mean over time. The spread of the distribution stays put; only conditional expectations for extreme units move inward.
• Forgetting the time symmetry. Reversion predicts backward too (the exceptional son's father is also expected to be less extreme), which is how you can tell it apart from a causal story.
• Testing the wrong null. A wave-1 slope significantly different from 0 says persistence exists; reversion is about the slope being below 1.
• Confusing reversion with reverse causation. "Dangerous intersections got roundabouts" raises two distinct issues: past outcomes influenced treatment assignment (reverse causation in assignment), and extreme past outcomes would have moderated anyway (reversion). Midterm questions ask you to name each one correctly.
• CI sloppiness. 2 is an approximation to 1.96, which is the large-sample t-critical value t(.975, df); df counts observations minus all coefficients including the intercept.

How the exam asks this

The professor's chapter 8 deck wraps everything in one running policy scenario (a Niagara development-charge relief program applied to the neighbourhoods with the biggest price jumps) and then fires 4 to 8 interpretation MCQs at it. Expect these recurring patterns:

• Straight definition: "What does reversion to the mean describe?" The keyed answer always names random variation: extreme outcomes are followed by less extreme ones because part of what made them extreme was noise. Distractors say "because the government intervened," "outcomes move to equilibrium," or "the mean itself shrinks."
• Required condition: outcomes must be shaped by both systematic forces (signal) and random fluctuation (noise). A follow-up asks what happens with pure signal (reversion disappears: identical conditions yield identical results).
• "What's the best response?" A councillor, manager, or influencer claims a program worked because the extreme units improved. The right answer flags reversion as an alternative explanation without overclaiming that the program did nothing.
• "Why is this incorrect?" Someone describes reversion as gravity or a pull. Keyed answer: reversion is a pattern that appears because noise fades, and no force acts on the observations.
• Selection on extremes: you discover the treated zones were chosen for their extreme pre-period values; why does that raise the reversion worry? Because unusually extreme performers tend to moderate later even without intervention.
• Figure reading: a scatter of this year's outcome against last year's, with separate fitted lines by treatment group. The flatter line means more reversion. Distractors talk about distance from the 45-degree line "proving" the policy worked or tighter clustering meaning equalized values.
• Before-and-after with an unusually bad baseline (midterm 2 style): sales were unusually low, then treatment, then recovery. Keyed answer: a temporary negative shock faded on its own, making the recovery look like a treatment effect.
• Cross-topic connections: naming reversion inside the ATT + Bias + Noise decomposition, distinguishing it from reverse causation, and the fix (randomize among equally extreme units, or compare treated extremes to untreated extremes).
• The CI note: quick hits on why 2 approximates 1.96, what t(1-α/2, df) means, why two-tailed tests use α/2 (2.5 percent in each tail at the 5 percent level), and computing df as observations minus coefficients including the intercept.

Lecture 9: Why Correlation Doesn't Imply Causation

This lecture is the turning point of the course. Part II asked whether a correlation is real (or just noise and selective reporting). Part III asks a harder question: even when a correlation is genuine and statistically significant, why is it usually weak evidence of causation? The answer lives in one framework, potential outcomes, and one master equation, Estimate = Estimand + Bias + Noise. If you can write out that decomposition and identify each piece in a scenario, you can answer almost every question this chapter generates.

Potential outcomes and counterfactuals

For each unit i (a person, a school, an intersection) and a binary treatment T, imagine two parallel worlds:

• Y1i = outcome for unit i if T = 1 (the treated world)
• Y0i = outcome for unit i if T = 0 (the untreated world)
• The causal effect of T on Y for unit i is Y1i - Y0i

The catch: in real data you only ever observe one of the two. A student who attends a charter school shows you her Y1i; her Y0i (how she would have done in a public school) is a counterfactual you never see. This is the fundamental problem of causal inference, and it is why comparing charter students to public students is usually comparing apples to oranges: the kids who apply to charters differ from the kids who do not, before any teaching happens.

Three estimands: ATE, ATT, ATU

An estimand is the quantity you want. Chapter 9 uses three averages of individual effects, distinguished by which group you average over. Notation: Ȳ1T means "average of Y1 among the Treated," Ȳ0U means "average of Y0 among the Untreated," and so on.

• ATE (average treatment effect, everyone): ATE = Ȳ1 - Ȳ0
• ATT (effect on the treated): ATT = Ȳ1T - Ȳ0T
• ATU (effect on the untreated): ATU = Ȳ1U - Ȳ0U
• They connect through a weighted average: ATE = ATT × Pr(T) + ATU × Pr(U)

Notice that ATT needs Ȳ0T, the outcome the treated units would have had without treatment. That term is unobservable, which is exactly why naive comparisons go wrong.

The master decomposition

What the data actually hands you is the simple difference in means (SDM): the average observed outcome of the treated minus the average observed outcome of the untreated.

• Population difference in means = Ȳ1T - Ȳ0U
• Sample difference in means = Ȳ1T - Ȳ0U + Noise
• Estimate = Estimand + Bias + Noise
• Targeting the ATT: Sample difference in means = ATT + Bias + Noise, where Bias = (Ȳ1T - Ȳ0U) - (Ȳ1T - Ȳ0T) = Ȳ0T - Ȳ0U
• Targeting the ATU: Sample difference in means = ATU + Bias + Noise, where Bias = (Ȳ1T - Ȳ0U) - (Ȳ1U - Ȳ0U) = Ȳ1T - Ȳ1U
• Targeting the ATE: Bias = (Ȳ0T - Ȳ0U) × Pr(T) + (Ȳ1T - Ȳ1U) × Pr(U)

Read the ATT bias term out loud: it is the baseline difference, how the treated group would have compared to the untreated group if nobody had been treated. If charter applicants would have scored higher anyway, Ȳ0T - Ȳ0U > 0 and the SDM overstates the charter effect. Bias has nothing to do with sample size: it is about comparability of groups. Noise is the sampling error, and it is the only piece that shrinks as N grows.

Where does bias come from?

Two canonical sources, and the exam expects you to tell them apart:

• Confounders: a third factor that affects a unit's treatment status AND (for reasons unrelated to the treatment) also affects the outcome. Both links are required. Traffic volume confounds the crosswalk-accident relationship only if busy intersections are more likely to get crosswalks and busy intersections also have more accidents.
• Reverse causation: the outcome affects the treatment. "We installed crosswalks where accidents were already frequent" means the outcome variable drove treatment assignment. The arrow runs backwards.

Always think about both. A single scenario can contain each, and different question stems in the same set will target one or the other.

Signing the bias

Classic exam move: even when you cannot compute the bias, you can still sign it. For a confounder Z, ask two questions: how does Z correlate with the treatment, and how does Z correlate with the outcome? Multiply the signs.

• Both positive or both negative: the naive estimate is biased upward (too positive).
• One positive, one negative: the naive estimate is biased downward (too negative).

Example: traffic volume is positively correlated with crosswalk installation and positively correlated with accidents. Positive times positive gives upward bias, so the naive regression makes crosswalks look worse at preventing accidents than they really are (the estimated effect is pushed in the positive, more-accidents direction).

Confounders vs. mechanisms

A mechanism is a variable on the causal pathway: treatment changes it, and it in turn changes the outcome. Controlling for a confounder removes bias. Controlling for a mechanism removes part of the causal effect you are trying to measure, which creates a new problem. Quick test: does the third variable cause the treatment (confounder, control for it), or does the treatment cause the third variable (mechanism, leave it out)?

Traps and misconceptions

• "The SDM is the causal effect." Only when bias is zero, which requires Ȳ0T = Ȳ0U (for the ATT): treated and untreated units would have looked the same without treatment. Randomization delivers this; self-selection almost never does.
• "A bigger sample fixes it." More data shrinks Noise only. Bias is unaffected by N. A biased estimate from 5,000 people is a precise wrong answer.
• "It is statistically significant, so it is causal." Significance addresses the noise question (is the correlation real?). It says nothing about the bias question (is it causal?). Ask the chapter's three questions in order: significant? genuine rather than a false positive? causal?
• Calling something a confounder when it only has one link. A variable that affects the outcome without affecting treatment status does no confounding, and vice versa. Both arrows are required.
• Mixing up ATT, ATU, ATE on the table. ATT averages the effect column over treated rows only, ATU over untreated rows only, ATE over all rows. The SDM uses only the observed cells: Y1 for treated rows, Y0 for untreated rows.
• Wrong bias identity. For the ATT the bias is Ȳ0T - Ȳ0U (a baseline-outcomes comparison). For the ATU it is Ȳ1T - Ȳ1U (a treated-outcomes comparison). Students routinely pick the expression for the SDM (Ȳ1T - Ȳ0U) or the ATT itself (Ȳ1T - Ȳ0T) when asked for the bias.
• Controlling for mechanisms. Adding every available variable to a regression is a mistake if some of those variables sit on the causal path.

Worked example

A tutoring centre lets students opt into a free exam-prep program (T). Suppose we magically observe both potential test scores for four students:

• SDM (observed cells only): Ȳ1T - Ȳ0U = (80+76)/2 - (60+58)/2 = 78 - 59 = 19
• ATT (treated rows, both columns): Ȳ1T - Ȳ0T = 78 - 69 = 9
• ATU (untreated rows): Ȳ1U - Ȳ0U = 64 - 59 = 5
• ATE (average the effect column): (10+8+6+4)/4 = 7, and check the weighting: 9 × 0.5 + 5 × 0.5 = 7
• Bias for the ATT: Ȳ0T - Ȳ0U = 69 - 59 = 10, and indeed SDM = ATT + Bias = 9 + 10 = 19

The story: stronger students opted in. Even without tutoring they would have scored 10 points higher than the non-enrollers, so the naive comparison of 19 points more than doubles the true effect on the treated (9). Nothing about collecting more students would fix this.

How the exam asks this

The professor's Chapter 9 deck and both midterms recycle one reliable pattern: a realistic policy or business scenario (crosswalks in Niagara, roundabouts, gym programs, product placement) with a full potential-outcomes table, followed by a run of interpretation MCQs. Expect these specific moves:

• Notation identification: "Which expression is the SDM?" with Ȳ1T - Ȳ0U, Ȳ1T - Ȳ0T, and Ȳ1 - Ȳ0 all offered. Know which is SDM, which is ATT, which is ATE, and which is the bias.
• The identity: pick SDM = ATT + Bias + Noise from rearranged impostors, and state when bias is zero (Ȳ0T = Ȳ0U).
• Table computations: compute SDM (observed cells only, watch the sign), ATT, and ATE (average the effect column). Distractors are always the other estimands, so label your work.
• "A colleague says..." questions: someone claims the SDM is causal, or that a large sample or a small p-value proves causation. The right answers say the SDM equals the causal effect only when bias is zero, sample size only shrinks noise, and p-values speak to chance rather than to bias.
• Confounder logic: why does variable Z confound? The correct option always states that Z is correlated with BOTH treatment and outcome; distractors give only one link.
• Classification: "we treated the units that already had bad outcomes" must be labelled reverse causation, and a variable the treatment itself changes must be labelled a mechanism (do not control for it).
• Signing the bias: two setup questions establish the sign of corr(Z, T) and corr(Z, Y), then a third asks for the direction of omitted-variable bias. Multiply the signs.
• Regression versions (heavily used on Midterm 2): a table of models where the treatment coefficient shrinks as controls are added, and you explain the drop as confounding; plus R completion items using lm(y ~ treatment + confounder, data = df), filter(), and geom_smooth(method = "lm").

Lecture 10: Controlling for Confounders

A confounder is a variable that affects both the treatment and the outcome. Confounders are why a simple difference in means is a biased estimate of a causal effect: the treated and untreated groups would have looked different even if nobody had been treated. Lecture 10 covers the standard observational fix, controlling, and spends just as much time on what controlling cannot do.

The core idea

Controlling means comparing treated and untreated units that share the same value of the confounder, then averaging those within-group comparisons. The lecture's running example: are members of Congress extreme because party leaders whip them into line? Republicans cast conservative roll-call votes far more often than Democrats, and personal ideology is an obvious confounder, since conservative people join the Republican party in the first place. The fix is to compare legislators with similar personal ideology (measured with candidate surveys) across parties. Whatever gap survives that comparison is a much cleaner estimate of party pressure.

Multiple regression automates this. In Y = α + β*T + γ*X + ε, the coefficient β is the association between T and Y holding X constant. If X captures all of the confounding, β is an unbiased estimate of the causal effect of T on Y.

Omitted variable bias: the formulas

Suppose the regression you want (the long regression) is:

Y = α + β*T + γ*X + ε

and its β would be an unbiased estimate of the effect of T on Y. If you cannot observe X, you are stuck running the short regression:

Y = αS + βS*T + εS

The gap between what you get and what you want is the omitted variable bias:

βS - β = γ * Π

where Π is the slope you would get from regressing the omitted variable X on the treatment T:

Π = cov(T, X) / var(T)

The professor's exercise deck writes the identical result as:

Bias(β̂1) = β2 * cov(T, X) / var(T)

Notice that Π is just the bivariate slope formula from your formula sheet, β = cov(x, y) / σx², applied with the omitted variable as the outcome and the treatment as the regressor. Two ingredients are both required for bias: the omitted variable must matter for the outcome (γ ≠ 0) and it must be correlated with the treatment (Π ≠ 0). If either one is zero, omitting X causes no bias at all.

Signing the bias

Exams love this move. The sign of the bias is the sign of the product γ * Π:

• γ > 0 and Π > 0: upward bias, βS is too positive.
• γ < 0 and Π < 0: upward bias again, the negatives cancel.
• Opposite signs: downward bias, βS is too negative.

Upward always means "too positive." If the true effect is negative, upward bias makes it look less negative or even positive. This is why treatment coefficients so often shrink toward zero when confounders enter the model: adding the control removes the bias and moves the estimate back toward the truth.

Worked example

Suppose the true model of exam scores is score = 40 + 5*tutoring + 2*motivation, with motivation on a 0 to 10 scale. Tutored students are more motivated: regressing motivation on tutoring gives Π = 3. Omit motivation and the short regression delivers:

βS = β + γ*Π = 5 + 2*3 = 11

The naive comparison more than doubles the true effect of 5. You can sign this bias without any data: γ is positive (motivation raises scores) and Π is positive (tutored students are more motivated), so the bias is upward, +6 in this case.

What controlling cannot fix

• Unobserved confounders. You can only control for what you measure. If ambition, enthusiasm, or family background is missing from the dataset, it stays in the error term and keeps biasing the estimate. No number of observed controls repairs this.
• Reverse causation. If the outcome influences the treatment (dangerous intersections get the safety upgrades, successful neighborhoods get the tax designation), controlling for other variables does nothing about it.

The lecture's illustration: is social media bad for you? Researchers ran an experiment paying people to quit, and they also analyzed observational data with controls. The comparison matters because a controlled observational estimate only deserves the same trust as the experiment when the no-unobserved-confounders assumption is credible, and that assumption is untestable.

Confounders vs. mechanisms

A mechanism is a variable on the causal path from treatment to outcome: T causes M, and M causes Y. You do not control for mechanisms. The mechanism is part of what you are trying to estimate, so controlling for it blocks the pathway and subtracts part of the effect, leaving an estimate that understates the total impact. The exercise deck's version: Opportunity Zones work by cutting property taxes to zero, so controlling for the property tax rate erases the program's main channel.

The nasty case: some variables are both confounder and mechanism. Estimating the effect of economic prosperity on civil war, do you control for democracy? Democracy plausibly causes prosperity (confounder logic) and is also plausibly caused by prosperity on the way to affecting conflict (mechanism logic). There is no purely statistical fix. You have to reason about the substantive question and defend your choice.

Heterogeneous effects: you estimate a LATE

If the treatment effect differs across units, controlling implicitly puts more weight on units whose treatment varies more within groups of the control variable (in the whipping example, the middle ideology bin, where both parties are actually represented). The weighted answer is a local average treatment effect (LATE) for those heavily weighted units, which is generally different from the ATE for everyone. This is a general issue across causal inference methods, so always ask whose effect you are estimating and whether it generalizes. The slide's consolation: better LATE than nothing.

Flavours of controlling

• Regression: include the confounders as right-hand-side variables.
• Matching: match each treated observation to one or more untreated observations with similar values of the control variables, then compare.
• Entropy balancing: reweight the untreated units so the treated and untreated groups have similar distributions of the control variables.

All three do the same kind of thing: they balance observed variables. None of them touches unobserved confounders or reverse causation. As the slides put it, there is no magic.

Classic traps and misconceptions

• The R² trap. A higher R² means the model predicts Y better. It says nothing about bias or causality. Adding almost any variable correlated with Y raises R², including bad controls that make the causal estimate worse.
• The control-coefficient trap. Coefficients on control variables (income, crime, traffic) are there to clean up the treatment estimate. Each one is an association holding the other regressors fixed, and the design says nothing about its causal status, because nobody controlled for the confounders of the controls.
• The "control for everything" trap. The rule is to control for variables that affect both the treatment and the outcome and are not mechanisms. Throwing in every variable in the dataset risks controlling for mechanisms and other post-treatment variables.
• The mechanism trap. On multiple choice, the giveaway is a variable that the treatment itself changes. Controlling for it understates the total effect.
• "With controls, it is as good as an experiment." A coefficient that survives controls is causal only if no unobserved confounders remain, and that assumption cannot be verified from the data. Randomized experiments do not need the assumption, which is why they remain the benchmark.
• "The coefficient stopped moving, so we are done." Stability across specifications is comforting. It is still no proof that bias is gone, because every specification can share the same blind spot.

How the exam asks this

The professor's Chapter 10 exercise deck and both midterm practice exams use a consistent recipe. Expect:

• One scenario, many questions. A realistic setting (an Opportunity Zone tax program, a shelf-placement dataset, a roundabout safety study, an influencer claim) followed by 4 to 10 interpretation MCQs that walk from correlation vs causation, to short and long regressions, to bias direction, to caveats.
• Formula recognition. Pick the correct bias expression β2 * cov(T, X) / var(T) out of distractors that swap in β1, drop the variance denominator, or use the wrong pair of variables in the covariance.
• Sign-the-bias questions. You are told how the confounder relates to the treatment and to the outcome, and you say whether the naive estimate is too positive or too negative.
• Regression tables. Columns (1), (2), (3) with standard errors in parentheses and the treatment coefficient shrinking as controls enter. You are asked what the shrinkage implies, whether the final column is now causal (no: unobserved confounders may remain), and how to interpret the coefficient with controls ("holding income, unemployment, and crime constant"). Keep CI95 = [β̂ - 2*SE(β̂), β̂ + 2*SE(β̂)] handy for any is-it-distinguishable-from-zero follow-up.
• Trap statements to rebut. "R² rose, so the model is causal." "Crime's coefficient proves crime causes fewer businesses." "With these controls the regression must match an experiment." The correct option is always the calm one that names the specific limit.
• Which variable should NOT be controlled for? The answer is the mechanism, the channel through which the policy works (the professor's version: PropertyTaxRate in the Opportunity Zone scenario).
• Reverse causality spotting. One sentence reveals that the outcome influenced treatment assignment ("neighborhoods with more businesses were more likely to be designated"), and you must name the problem.
• R code. Complete lm(y ~ t + x, data = df) specifications (know that + adds controls while * adds interactions), filter() for subsetting rows, and geom_smooth(method = "lm") for fitted lines drawn separately by color group.

Lecture 11: Randomized Experiments

The last few lectures were the bad news: observational comparisons, even with many controls, almost never give convincing causal estimates because you can never be sure you controlled for everything. This lecture is the good news. If you randomize who gets the treatment, the bias term disappears by design, and a simple difference in means becomes an unbiased estimate of the causal effect.

Why randomization kills bias

Everything runs through the course's core decomposition. What you compute from data is the difference in mean outcomes between treated and untreated units, and it splits into pieces:

• Population difference in means = Ȳ1T − Ȳ0U
• Sample difference in means = Ȳ1T − Ȳ0U + Noise
• Average treatment effect (ATE) = Ȳ1 − Ȳ0
• Bias = (Ȳ0T − Ȳ0U) × Pr(T) + (Ȳ1T − Ȳ1U) × Pr(U)

Read the bias formula in words: bias shows up when the treated would have differed from the untreated even without treatment (different average Ȳ0), or when the two groups would respond differently to treatment (different average Ȳ1). Randomization makes treatment status independent of potential outcomes, so on average:

Ȳ0T = Ȳ0U and Ȳ1T = Ȳ1U, which forces Bias = 0.

The familiar decomposition Difference in means = ATT + Bias + Noise still applies. Randomization sets Bias to zero. Noise never goes away: a randomized estimate is unbiased, and it can still be imprecise.

Motivating puzzle from the slides: breastfeeding and infant health are strongly positively correlated in the developed world and negatively correlated in the developing world. Who breastfeeds, and what the alternative is (clean water or contaminated water), differs across settings: confounding. Researchers in Belarus resolved this with a large randomized experiment, randomizing encouragement to breastfeed rather than forcing anyone.

Ways to randomize

• Independent randomization: flip a coin for each unit. Simple, and group sizes can end up lopsided.
• Constrained group sizes: randomly sort the units, then put the first k units (or first x percent) in the treated group.
• Blocked (stratified) randomization: group units by covariates you think are correlated with Y0 and Y1 (say, by school or by baseline score), then randomize within each block. This guarantees balance on the blocking variables and improves precision.

Estimation and inference

Estimate the ATE with the sample difference in means, or equivalently regress the outcome on the treatment dummy: lm(outcome ~ treated) gives the identical number as the slope. This equivalence is a favorite exam fact.

With m treated units out of N total, the estimated standard error is:

SE = √( Var(Y0)/(N − m) + Var(Y1)/m )

Hypothesis testing works exactly as before: posit a null (usually an effect of zero) and ask how likely an estimate as large as yours would be by chance if the null were true. The two common tools are the t-test and randomization inference (reshuffle the treatment labels many times and see where your actual estimate falls in that distribution). All the earlier caveats about p-values and confidence intervals still apply. From the formula sheet:

CI95 = [β̂ − 2 × SE(β̂), β̂ + 2 × SE(β̂)]

Noncompliance: ITT, compliers, and the Wald estimator

Noncompliance means some people assigned to treatment never take it up, or some people in the control group get the treatment on their own. Two tempting fixes are both wrong:

• Comparing people who did and did not take the treatment: take-up is self-selected, so this reintroduces exactly the selection bias randomization was supposed to remove.
• Dropping the noncompliers: the groups that remain are no longer randomly comparable.

What you can estimate without bias is the intent-to-treat (ITT) effect: the effect of being assigned to the treatment group, computed as the difference in mean outcomes by assignment. ITT is often the more policy-relevant quantity anyway, because a government can offer a program far more easily than it can force usage.

To get closer to a treatment effect, classify subjects into four types by how they respond to assignment:

Assume there are no defiers (people who do the opposite of their assignment), which is reasonable in most settings. Then you can estimate the share of compliers, called the first stage:

Pr(treated | assigned to treatment) − Pr(treated | not assigned to treatment) = Pr(Complier) + noise

Now add the exclusion restriction: assignment affects the outcome only through its effect on the treatment. Under that assumption:

ITT = CATE × Pr(Complier) + noise

Divide the two estimates and you get the Wald estimator:

Wald estimator = ITT / First stage = ITT / Pr(Complier)

This is an unbiased estimate of the CATE, the complier average treatment effect: the causal effect of the treatment for compliers only. Since the ITT spreads the effect across everyone assigned (including never-takers who felt nothing), dividing by the complier share scales it back up.

Instrumental variables

The Wald estimator is a special case of instrumental variables (IV). You need an outcome of interest, a treatment of interest, and an instrument that affects the treatment (in an experiment, the instrument is random assignment). The necessary assumptions:

• Exogeneity: you can credibly estimate the effect of the instrument on the treatment and on the outcome (randomization delivers this).
• Exclusion restriction: the instrument moves the outcome only through the treatment.
• Compliers exist: the instrument actually shifts some people's treatment status.
• No defiers.

The more common implementation is two-stage least squares (2SLS). Key sentence from the slides: IV is a tool for addressing noncompliance; on its own it is a fix inside a design rather than a research design.

When experiments go wrong

Chance imbalance. Through bad luck, treatment and control can differ on pre-treatment covariates. Options: throw out the broken experiment, proceed as normal (randomization only promises balance on average), or control for the unbalanced covariates. A small imbalance in one covariate is expected and does not by itself mean randomization failed.

Imprecise estimates. Unbiased does not mean precise. Remedies: a bigger sample (and a more even split between treatment and control), a blocked or stratified design, and controlling for pre-treatment covariates to soak up outcome variance.

Attrition. Some subjects exit before you measure their outcome. Three scenarios, in rising order of trouble: attrition is random (just less precision), attrition is nonrandom but unrelated to assignment (still workable), and attrition is affected by the treatment itself. The slide example: subjects who took the exercise treatment and lost no weight were embarrassed and skipped the follow-up, so the treated sample you observe is selected on success. Responses: design the study to mitigate attrition, test whether attrition is related to treatment or to pre-treatment covariates, and estimate bounds.

Interference (spillovers). One unit's treatment status affects another unit's outcome: friends in an economic study pool resources, a New Jersey minimum wage increase affects Pennsylvania employment as workers cross the border, one voter in a household makes the others more likely to vote. Sometimes spillovers are the object of study; otherwise they contaminate the control group. The standard fix is to randomize at a higher level (classroom, store, village) so treated and control units do not interact.

Natural experiments

Sometimes the world randomizes for its own practical reasons and researchers get to piggyback: draft lotteries, charter school admission lotteries, gambling winnings, randomized ballot order. Treat these like experiments with messy compliance: think hard about the exclusion restriction (did draft assignment affect earnings only through military service?), and remember the compliers may be unusual people, which limits how far the CATE generalizes.

Classic traps and misconceptions

• "Randomization guarantees balanced groups." It guarantees balance on average, in expectation. Any single sample can show chance imbalances, and finding one does not prove randomization failed or that confounders lurk.
• Comparing takers to non-takers. Once people choose whether to take up treatment, that comparison is observational again. Same for dropping noncompliers.
• Confusing assignment with usage. The ITT is the effect of being assigned (offered) the treatment. Claims like "usage changed outcomes by the ITT amount for everyone" overreach twice: wrong quantity and wrong population.
• Multiplying instead of dividing. CATE = ITT / Pr(Complier). With partial compliance the CATE is larger in magnitude than the ITT. If your computed "CATE" is smaller than the ITT, you multiplied by mistake.
• Generalizing the CATE. It is the effect for compliers. Never-takers and always-takers may respond differently, so "the treatment changes outcomes by this much for everyone" is wrong.
• Picking the wrong estimand for policy. If the policy question is "should we offer or restrict access," the ITT answers it. If the question is "what does actually using the treatment do," you want the CATE.
• "Insignificant means zero effect." A small study has low power, so a modest true effect can easily produce an insignificant estimate. And hiding null results in the file drawer creates publication bias; publishing them keeps meta-analyses honest.
• p-value and CI misreadings. The p-value answers: if the null were true, how likely is an estimate at least this large by chance? It is never the probability that the null is true. Likewise a 95 percent CI describes the procedure's long-run coverage across repeated samples.
• Mixing up exogeneity and the exclusion restriction. Exogeneity is about the instrument being as good as random. The exclusion restriction is a separate claim: assignment touches the outcome only through take-up of the treatment.

Worked example

A utility company randomly selects 150 of 300 households to receive a free smart thermostat installation offer. Among the 150 offered, 90 install one; no control household installs. Average daily electricity use afterward: 32.4 kWh for the offered group, 34.2 kWh for the control group.

• ITT: 32.4 − 34.2 = −1.8 kWh. Being offered the thermostat reduced average use by 1.8 kWh per day.
• First stage: 90/150 − 0/150 = 0.60. An estimated 60 percent of households are compliers.
• Wald / CATE: −1.8 / 0.60 = −3.0 kWh. Among compliers, installing the thermostat reduced use by about 3 kWh per day.

The CATE interpretation leans on no defiers and on the exclusion restriction (the offer letter changed usage only by getting thermostats installed). If merely receiving the letter made households more energy conscious, the exclusion restriction fails and the Wald number overstates the thermostat effect.

How the exam asks this

The professor's exercise deck for this chapter runs one continuous scenario (a Ministry of Education policy team reviewing studies on social media and youth mental health) through a chain of interpretation MCQs. Expect the same moves on the final:

• "Why does random assignment matter?" The credited answer says assignment becomes independent of potential outcomes, eliminating observed and unobserved confounding. Distractors claim it equalizes sample sizes, equalizes everyone's outcomes, or raises R².
• Chance imbalance vignettes. Baseline anxiety (or steps, or sales) differs slightly across arms. The credited answer: balance holds on average, small imbalances happen. Follow-ups ask why "just control for it" or "scrap the study" may be hasty.
• Low power and the file drawer. A 60-person pilot finds nothing: the credited answer is low power, never "the effect is zero." A follow-up asks whether to hide the null study: no, publishing nulls reduces publication bias.
• Compute the ITT from a table of means. Watch the sign and the direction of the wording (an outcome 0.82 lower for the assigned group means ITT = −0.82).
• Compute the compliance rate (takers among assigned minus takers among controls) and then CATE = ITT / compliance. Distractors are always the multiplied version, the raw ITT, and a ratio using the wrong denominator.
• Name the missing assumption. Given randomization, compliers, and no defiers, the answer is the exclusion restriction. Also expect "which best distinguishes ITT from CATE" and "which effect should guide the Minister's decision" (ITT for an access policy, CATE for the effect of use).
• Spot the pathology from a footnote. Treated students convince classmates to quit social media: interference. The redesign answer is randomizing at the classroom (cluster) level.
• Revise the overclaiming summary. Junior analyst writes "use decreases scores by 1.37 for everyone"; the credited revision hedges to compliers and to access, with the right number attached to the right estimand.

On midterms, this material also appears with regression output tables (Estimate, Std. Error, t value) where the slope on the treatment dummy is the difference in means, CI and p-value meaning questions, ATT + Bias + Noise decompositions from potential outcomes tables, and R completion items using lm(), filter(), and ggplot() with geom_smooth(method = "lm"). A reliable test-taking pattern: distractors speak in absolutes (guarantees, proves, always, exactly), while credited answers are specific and hedged.

Lecture 12: Regression Discontinuity Designs

Experiments give unbiased causal estimates, but you often cannot run one: too expensive, impractical, or unethical. Regression discontinuity (RD) is the next best thing when a treatment is handed out by a cutoff rule. A scholarship goes to averages of 80 and above, extra prenatal monitoring kicks in at age 35, a remedial course is ordered at 9 demerit points. Wherever a bright line decides who gets treated, an RD design may let you estimate the effect of the treatment.

The setup: running variable, cutoff, treatment

Three ingredients, and the exam loves making you identify them:

• Running variable R: the continuous score that assignment depends on (admission average, age in days, demerit points).
• Cutoff c: the threshold value of the running variable where treatment status changes.
• Treatment D: what actually happens to you when you cross the line. In a sharp design, D = 1 if R >= c, D = 0 otherwise.

The logic: someone with an average of 79.9 and someone with 80.1 are essentially the same person. Same ability, same background, same everything on average. The only thing that changes discontinuously at the cutoff is treatment. So a jump in the outcome right at the cutoff can be attributed to the treatment. Comparing units just below to units just above approximates a randomized experiment, locally.

The price you pay: RD estimates a local average treatment effect (LATE), the effect for units right around the cutoff. It tells you nothing directly about people far from the line.

The identifying assumption: continuity of potential outcomes

Both potential outcomes, Y(1) and Y(0), must vary smoothly (continuously) with the running variable as it passes through the cutoff. Treatment probability is the only thing allowed to jump at c. If that holds, any jump in observed outcomes at the cutoff is the treatment effect for units at the cutoff.

What breaks continuity in practice:

• Manipulation or sorting of the running variable. Parents timing a delivery to dodge the "advanced maternal age" label, teachers bumping a 79.6 up to 80, applicants retaking a test until they clear the bar. Now the people just above and just below are different kinds of people, and the comparison is contaminated.
• Other things changing at the same cutoff. If turning 65 triggers both public health insurance and pension eligibility, a jump in health outcomes at 65 cannot be pinned on insurance alone.

Diagnostic: plot a histogram (density) of the running variable. A spike piled up on one side of the cutoff with a dip on the other is the classic fingerprint of manipulation. The exercise deck ends on exactly this question.

One more subtlety from the slides: continuity can be satisfied while treatment effects are heterogeneous away from the cutoff. The design is still valid, but the estimate only speaks for units near the cutoff. Do not extrapolate.

Three ways to implement an RD

• 1. Naive: pick a small window around the cutoff, compute the mean outcome just above and just below, take the difference.
• 2. Local linear: pick a (possibly larger) window, fit separate linear regressions on each side of the cutoff, and take the difference in the two fitted values at the cutoff. This can be done in one regression (see the formula below).
• 3. Polynomial: pick an even larger window and regress the outcome on the treatment while controlling for polynomials of the running variable.

The bandwidth tradeoff shows up on every exam: a narrower window reduces bias from misspecifying the functional form near the cutoff (a line is a good approximation over a short range), but it uses fewer observations, so variance rises and the estimate gets noisier. A wider window buys precision at the risk of bias. Good practice is to report estimates across several bandwidths and specifications and show the result is robust to those choices.

Key formulas

• Sharp RD assignment rule: D_i = 1 if R_i >= c, 0 otherwise
• RD estimand (difference in limits at the cutoff): tau_RD = lim(R -> c from above) E[Y | R] − lim(R -> c from below) E[Y | R]
• Local linear model with the running variable centered at the cutoff (R_i = score − c): Y_i = alpha + tau*D_i + beta1*R_i + beta2*(D_i × R_i) + e_i. Here tau is the jump in the outcome exactly at R = 0, beta1 is the slope below the cutoff, beta1 + beta2 is the slope above, and alpha is the predicted outcome just below the cutoff.
• Fuzzy RD estimator (a Wald / IV ratio): tau_fuzzy = (jump in E[Y] at cutoff) / (jump in Pr(D = 1) at cutoff). This is the same logic as the formula sheet entry Wald = ITT / Pr(Complier): the outcome jump plays the role of the ITT, and the jump in treatment probability plays the role of the complier share.
• Confidence interval (from the formula sheet, used to judge significance of tau): CI95 = [tau_hat − 2*SE(tau_hat), tau_hat + 2*SE(tau_hat)]

Sharp versus fuzzy RD

Sharp: the threshold completely determines treatment. Everyone above is treated, everyone below is untreated.

Fuzzy: the threshold influences treatment but there is noncompliance: some units below the cutoff get treated anyway (a below-cutoff student is so impressive she gets the scholarship), and some above the cutoff go untreated. The probability of treatment jumps at the cutoff without going from 0 percent to 100 percent.

Solution: combine RD with instrumental variables. Crossing the cutoff is the instrument for treatment. You need the IV assumptions:

• Exclusion restriction: crossing the cutoff affects the outcome only through the treatment, with no other channel.
• Monotonicity (no defiers): crossing the cutoff never reduces anyone's chance of treatment.

The resulting LATE is the effect of the treatment for compliers close to the cutoff: units whose treatment status is moved by crossing the threshold.

Two case studies from the lecture

• Are extremists more electable? You cannot just regress general election vote shares on candidate extremism: parties and districts choose candidates for reasons correlated with electability, so the estimate is biased. Instead, use close primary elections between an extremist and a moderate. The running variable is the extremist's primary winning margin, the cutoff is zero. A primary barely won by the extremist versus barely lost is as good as a coin flip, so comparing subsequent general election outcomes on either side of zero identifies the effect of nominating an extremist (the evidence says extremist nominees do worse).
• Bombing in Vietnam. U.S. officials scored each village on a continuous security scale, then converted scores into letter grades A through E using cutoffs. Officials deciding where to bomb saw the grades rather than the underlying scores. Villages just on either side of a grade cutoff had nearly identical scores but discontinuously different probabilities of being bombed: a fuzzy RD. The study found bombing was counterproductive, pushing villages toward the insurgency.

Classic traps and misconceptions

• Reading tau as a global effect. RD identifies the effect at the cutoff, for units near the cutoff. Any answer choice claiming the effect applies to everyone, at every value of the running variable, is wrong.
• Wide-band naive comparisons. Comparing all 30 to 34 year olds against all 35 to 39 year olds mixes the smooth age trend with the jump at 35. The whole point of RD is to isolate the discontinuity from the trend in the running variable.
• Confusing the first stage with the effect. A jump in the probability of treatment at the cutoff is the first stage. It is evidence the design has bite, and it is the denominator of the fuzzy estimator. It is never itself the treatment effect on the outcome.
• Forgetting to scale in a fuzzy RD. The raw outcome jump at the cutoff is an effect of crossing the threshold (like an ITT). Divide it by the jump in treatment probability to get the effect of the treatment on compliers.
• Believing a narrow bandwidth is free. It reduces functional form bias and it inflates variance. Nothing reduces both at once except more data.
• Ignoring the density check. A spike in the histogram just on the favorable side of the cutoff signals sorting, which breaks continuity. It is a threat to identification, and it does far more damage than just changing standard errors.
• Misreading the regression table. In Y = alpha + tau*D + beta1*R + beta2*(D × R) + e, only tau is the RD effect. beta1 is the below-cutoff slope, beta2 is the change in slope, and a significant slope says nothing about a treatment effect.
• Assuming near-cutoff units are identical. They are similar on average, and treatment probability is what changes discontinuously. Answer options that say the design "guarantees" identical risk factors or "reduces measurement error to zero" are wrong on purpose.

Worked example

A university awards a bursary to entrants with an admission average of 85 or above. Using students within 4 points of the cutoff, with dist = average − 85 and above = 1 if average >= 85, you estimate:

gpa = 2.40 + 0.15*above + 0.06*dist + 0.02*(above × dist), with SE(tau_hat) = 0.05.

• Predicted first-year GPA just below the cutoff: 2.40. Just above: 2.40 + 0.15 = 2.55. The RD estimate of the bursary effect at the cutoff is tau_hat = 0.15 GPA points.
• Confidence interval: CI95 = [0.15 − 2(0.05), 0.15 + 2(0.05)] = [0.05, 0.25]. Zero is outside the interval, so the jump is statistically distinguishable from no effect at conventional levels.
• Slopes: below the cutoff GPA rises 0.06 per point of average; above, it rises 0.06 + 0.02 = 0.08.
• Now suppose take-up is imperfect: the share actually receiving the bursary jumps from 0.10 just below to 0.60 just above (first stage jump = 0.50). The fuzzy RD effect for compliers at the cutoff is 0.15 / 0.50 = 0.30 GPA points. Same numerator, scaled by who actually changed treatment status.

How the exam asks this

The professor's exercise deck for this chapter is one long realistic scenario (an "advanced maternal age" cutoff at 35 that triggers extra prenatal monitoring) followed by interpretation MCQs, and the midterms use the same template: a policy story, a figure or regression table, then 4 to 8 questions. Expect these specific patterns:

• Figure reading: what is on the x-axis and what does the dashed vertical line mark (running variable and cutoff); what does the vertical gap in a treatment-probability plot represent (the first stage jump); how to describe an outcome jump cautiously (a local decrease near the cutoff, worded with "appears associated").
• Design logic: why compare people just below and just above instead of everyone (near-cutoff units are similar in all ways except treatment probability, approximating an experiment); which behaviour violates continuity (timing or manipulating the running variable to dodge the cutoff, never innocuous things like different ultrasound machines).
• Sharp versus fuzzy classification: given that the treatment probability jumps without going 0 to 100, pick "fuzzy" with the correct reason. Distractors offer "fuzzy because the cutoff is unknown" and "sharp because the rule is exact": both defensibly wrong.
• Fuzzy assumptions: pick the combination "exclusion restriction and monotonicity (no defiers)" over decoys like equal sample sizes or identical covariates at all ages.
• Model interpretation: given Y_i = alpha + tau*D_i + beta1*R_i + beta2*(D_i × R_i) + e_i, identify what tau represents (the local effect exactly at the cutoff), with the slope-difference reading of beta2 as a distractor.
• Bandwidth: the tradeoff question, nearly verbatim: narrower reduces bias from misspecified functional form near the cutoff but increases variance because it uses fewer observations.
• LATE scope: "for whom is this effect identified": units with running variable values close to the cutoff whose treatment is influenced by crossing it.
• Naive comparison bias: why comparing wide bins (ages 30 to 34 versus 35 to 39) is biased relative to RD: it mixes broad trends in the running variable with the jump at the cutoff.
• Histogram question: a spike just on one side of the cutoff suggests manipulation of the running variable and threatens identification.
• Midterm-style extras attached to any topic: regression output tables with Estimate, Std. Error, and t value columns; R code completion using lm(), filter(), and ggplot() + geom_smooth(method = "lm"); confidence interval and p-value interpretation questions where the correct answer is the careful frequentist statement and the wrong answers claim certainty, probability of the null, or guarantees.
• A reliable test-taking pattern: options containing "always", "proves", "guarantees", "eliminates all bias", or "reduces error to zero" are essentially always wrong; correct options say "local", "on average", "near the cutoff", and "under the continuity assumption".

Lecture 13: Difference-in-Differences Designs

The big idea. Sometimes a treatment turns on at different times for different units: one province raises its minimum wage while another does not, some cities get television before others. A difference-in-differences (DD or DiD) design compares the change in the treated group to the change in the control group. By working with changes instead of levels, DiD automatically strips out two whole families of confounders:

• Everything that differs across units but stays constant over time (California is simply different from Alabama, always).
• Everything that changes over time but hits all units equally (a national recession, a policy that applies everywhere).

What survives the double differencing is the differential change in the treated group, which is your treatment effect estimate, provided one key assumption holds.

The key identifying assumption: parallel trends. If the treatment had never happened, the treated group's outcome would have changed by the same amount as the control group's outcome. Read that carefully: it is a claim about counterfactual trends. The two groups are allowed to sit at completely different levels before treatment; that gap is constant and gets differenced away. What DiD cannot survive is the treated group being on a different trajectory for reasons other than the treatment.

The basic calculation. With two groups and two periods:

DiD = (Ybar_T,post - Ybar_T,pre) - (Ybar_C,post - Ybar_C,pre)

Equivalently, you can difference the other way first (post gap minus pre gap) and get the same number:

DiD = (Ybar_T,post - Ybar_C,post) - (Ybar_T,pre - Ybar_C,pre)

The regression version. The 2x2 DiD is exactly reproduced by one regression with an interaction:

Y = b0 + b1*Treated + b2*Post + b3*(Treated x Post) + e

Each coefficient has a fixed meaning, and exam questions love to ask you to match them:

• b0 = control group mean in the pre period.
• b1 = pre-period gap between treated and control (the constant level difference).
• b2 = change over time in the control group (the common time shock).
• b3 = the DiD estimate: how much more the treated group changed than the control group. This is the only causal quantity, and only under parallel trends.

In R: lm(y ~ treated * post, data = df). The * expands to both main effects plus the interaction, and the row labelled treated:post in the output is your estimate. Judge its significance the usual way: CI95 = [b3_hat - 2*SE(b3_hat), b3_hat + 2*SE(b3_hat)], and a t value beyond about 2 in absolute value means the interval excludes zero.

Three ways to implement a DiD (the lecture lists all three, and the exam expects you to match each method to its data format):

• By hand: plug the four group means into the formula above.
• First differences: data in wide format, one row per unit. Compute each unit's change in outcome and change in treatment, then regress one on the other: dY_i = a + b*dT_i + e_i, in R something like lm(d_outcome ~ d_treatment, data = wide_df).
• Fixed effects: data in long format, one row per unit-period. Regress the outcome on the treatment plus unit fixed effects and time fixed effects: Y_it = b*T_it + unit FE + time FE + e_it, in R lm(y ~ treatment + factor(unit) + factor(period), data = long_df). Unit fixed effects absorb constant unit differences; time fixed effects absorb common shocks. This version generalizes to many units and many periods.

Useful diagnostics. Parallel trends itself is untestable (it is about a counterfactual you never observe), so you build a circumstantial case:

• Look at pre-trends. Plot the two groups over several pre-treatment periods. If they moved together before treatment, parallel trends after treatment is more believable. One pre period alone tells you almost nothing about trends.
• Lead-treatment test. In the fixed effects setup, code a variable equal to the treatment's value in the next period and include it alongside the actual treatment. If the lead coefficient is not zero, outcomes were already changing before the treatment changed. A significant lead is bad news for the design.
• Unit-specific linear trends. Add a time variable for each unit (time for that unit, 0 for all others). If parallel trends holds, adding these barely moves your estimate of interest. If the estimate jumps around, worry.

The lecture's running examples. Minimum wage and employment: comparing California to Alabama in levels is hopeless, so compare employment changes when one state raises its wage against changes in states that did not. Television and kids: TV arrived in some American cities years before others in the 1940s and 50s, so compare cognitive outcomes of cohorts across early and late TV cities. Obesity contagion: Christakis and Fowler found that people whose friends became obese were more likely to become obese, a DiD-flavoured claim; the same method also "finds" that height and acne are contagious, which is a placebo test screaming that the identifying assumption fails (friends share environments and choose each other, so their trends were never parallel to begin with).

Classic traps and misconceptions.

• "The groups start at different levels, so DiD is invalid." Wrong. Level differences are exactly what DiD removes. Parallel trends is about slopes of the counterfactual, never about starting points.
• "Treated units differ from controls in X, so the estimate is biased." Only if X changes differentially over time during the study window. Any time-invariant difference (more bars, more rural roads, bigger population) is absorbed by the differencing or by unit fixed effects.
• Reading the wrong number. The within-treated before/after change mixes the treatment with every common time shock. The post-period treated/control gap mixes the treatment with every constant group difference. Only the double difference (the interaction coefficient) is the DiD estimate.
• Genuine threats are time-varying and group-specific: another policy or shock that hits only the treated group at the same time as the treatment. That is the shape of every "legitimate threat" answer.
• Overclaiming precision. DiD gives an average differential change (the ATT under parallel trends), never the exact effect for each unit, and never "proof" of the mechanism.
• Misreading the lead test. A significant lead-treatment coefficient does not confirm anything good. It signals pre-existing trends and undermines the design.

Worked example. Province A raises its minimum wage in January 2024; Province B does not. Teen employment rates:

Treated change: 50 - 52 = -2. Control change: 45 - 48 = -3. So DiD = -2 - (-3) = +1. A naive before/after look says the wage hike "cost" 2 points of employment. The control group reveals employment was falling everywhere by about 3 points, so relative to that counterfactual the treated province did 1 point better. In the interaction regression you would see b0 = 48, b1 = 4 (pre gap), b2 = -3 (common decline), b3 = +1 (the DiD). This sign flip between the naive change and the DiD is a favourite exam moment.

How the exam asks this

The professor's Chapter 13 exercise deck and the midterm style point to a very stable question template. Expect a realistic scenario (a police enforcement pilot, a transit program, a policy rollout across divisions or municipalities) followed by 4 to 8 interpretation MCQs built on a small 2x2 table, a plot, and a regression table. The recurring patterns:

• "What does this number represent?" You are shown the within-treated change (for example 4.0 to 2.8) and must recognize it as the before/after change that mixes treatment with other time-varying factors, and specifically that it is neither the causal effect nor the ATT nor pure noise.
• Arithmetic on the table. Compute the post-period treated/control gap, then the full DiD. Distractors are always the other differences you could have taken: the treated change alone, the control change alone, the cross-sectional gap, or a sign error.
• Name the assumption. Correct answer: the groups would have followed the same trend absent treatment. Standard distractors: equal pre-period levels, identical covariates, identical post-period shocks.
• Regression tables with models (1) to (3), adding Treated, then Post, then Treated x Post, with Estimate and Std. Error columns. You must identify the interaction as the DiD, interpret its sign and size, and check significance with the t value or the two-SE confidence interval.
• Colleague-correction questions. A claim like "traffic volume differs across divisions but is stable over time, so it cannot bias the DiD" (true: time-invariant factors are differenced out), or "the DiD shows the unit installed caused exactly 0.9 fewer fatalities" (overclaim: DiD is an average difference in changes). Pick the response that fixes precisely the error made.
• Spot the legitimate threat. Among options listing constant differences (more young drivers, more rural roads, bigger area), the correct threat is the time-varying, treated-only shock, such as a new policy in 2023 affecting only treated divisions.
• Event-study plots. Read pre-period slopes to judge parallel trends (similar pre slopes support it; groups moving in opposite directions pre-treatment violate it; a level gap does not), and recognize that the difference in slopes from pre to post is the DiD estimate itself.
• R code completion in the midterm style: complete lm() calls (interaction with treated * post, first differences, fixed effects with factor()), and occasionally ggplot with geom_smooth(method = "lm") for trend plots.

Lecture 14: Assessing Mechanisms

Everything before this lecture asked one question: does the treatment change the outcome, and by how much? Lecture 14 asks the follow-up question: why and how does it work? Knowing the mechanism matters for two reasons. First, substantive understanding: a program evaluation that ends at "it worked" tells you very little about the world. Second, portability: if you know why a program worked in one setting, you can predict whether it will work somewhere else.

The honest headline of this chapter: there is no algorithm for finding mechanisms. No regression spits out "the mechanism." You need clear thinking, theory, careful research design, and independent tests. The exam rewards guarded, humble conclusions and punishes anything that claims a mechanism has been "proven."

Causal mediation analysis (and why to distrust it)

The textbook example: do charter schools raise college attendance, and if so, is the mechanism access to AP classes? Mediation analysis runs two regressions:

• Short: college attendance = α1 + β1 * charter school + ε1
• Long: college attendance = α2 + β2 * charter school + γ2 * AP classes + ε2

The claimed interpretation:

• β1 is the average (total) effect of charter schools.
• β2 is the effect of charter schools over and above their effect on AP classes (the "direct" effect).
• (β1 - β2) / β1 is the share of the effect that goes through the mechanism of AP classes.

Take this with a grain of salt. Here is the core problem: the mediator (AP classes) is a post-treatment variable. Charter schools may be randomly assigned, but who takes AP classes is never randomly assigned. Students who take AP classes differ in unobserved ways (motivation, family support, ability) that also affect college attendance. So γ2 is confounded, which contaminates β2, which makes the decomposition unreliable. The slides make the point dramatically: you can build an example where charter schools affect AP class-taking, AP classes have zero causal effect on college attendance (so AP classes are truly not a mechanism), and yet the mediation regressions still report a healthy "share of the effect" flowing through AP classes. Controlling for a variable that the treatment itself created is the classic bad control mistake, and mediation analysis does exactly that on purpose.

So if mediation analysis is untrustworthy, what can we actually do? The lecture gives four practical strategies.

Strategy 1: Measure intermediate outcomes

Regress candidate mediators on the treatment. This part is legitimate: if the treatment was randomized, the effect of treatment on the intermediate outcome is well identified. The logic is asymmetric:

• If the treatment does not move a candidate mediator, that mechanism becomes very hard to defend. You can rule it out (or at least demote it).
• If the treatment does move a candidate mediator, the mechanism is plausible, and that is all. You still do not know how much that intermediate outcome affects the final outcome, so you cannot confirm the channel.

Lecture example: cognitive behavioral therapy for at-risk youth in Liberia. The randomized intervention improved economic outcomes and reduced crime. It showed no effect on self-control skills (so self-control looks like a weak candidate mechanism) and big effects on social networks and attitudes toward violence (so those channels are plausible).

Strategy 2: Test independent theoretical predictions

Think theoretically: if Mechanism A is at work, what else should be true in the data? Then go test those extra predictions.

Lecture example: discrimination against women in elections. Women who run for office win about as often as men. Does that prove no discrimination? No. Consider the theory: if discrimination exists and women know it, only exceptional women run. That theory generates an independent prediction: marginally elected women should outperform marginally elected men, because a woman who barely wins had to clear a higher hurdle. Anzia and Berry find exactly that: districts that elect a woman get a more productive legislator. Additional tests sharpen the case: widows of male members of Congress (who inherit the seat without clearing the hurdle) do not outperform, and the female performance premium is largest in more conservative districts, where the hurdle should be highest. Each extra confirmed prediction makes rival explanations less credible.

Strategy 3: Test mechanisms through design (multiple treatment arms)

Build the mechanism test into the experiment itself. Lecture example: Gerber, Green, and Larimer's voter turnout postcards. A postcard showing households their own and their neighbours' voting records (with a threat to publicize turnout) raised turnout by 8 percentage points. A milder postcard, without the neighbour comparison, raised turnout by 5 percentage points. Because there were multiple treatment arms plus a control group that got nothing, you can compare arms to each other: the gap between arms isolates the incremental effect of social pressure, while each arm versus control gives the total effect of that version. One arm alone could never separate "any postcard works" from "social pressure works."

Strategy 4: Disentangle competing mechanisms

Sometimes theory predicts two mechanisms pushing in opposite directions. Lecture example: does economic prosperity reduce violent conflict? Studies using commodity price shocks often find null or conflicting results. Two competing mechanisms can explain this: higher wages raise the opportunity cost of fighting (less conflict), while more resources mean there is more to fight over (more conflict). A null total effect can hide two large, offsetting channels. The way forward is to find variation that moves one mechanism at a time: for example, price shocks to labour-intensive commodities mainly move wages, while price shocks to capital-intensive resources mainly change the size of the prize.

Classic traps and misconceptions

• Controlling for a post-treatment variable. "Let's hold awareness fixed by controlling for it" sounds rigorous and is wrong. The mediator was created by the treatment and is selected, so controlling for it distorts the treatment coefficient. This is the single most tested trap in this chapter.
• Trusting the mediation share. (β1 - β2) / β1 only measures the mechanism share under assumptions that are almost never true (essentially, that the mediator is as good as randomly assigned). The share can even come out negative or above 1.
• "Treatment moves the mediator, therefore that is the mechanism." Moving the mediator makes the channel plausible. Confirming it would also require knowing the mediator's effect on the outcome, which you usually do not.
• "The biggest arm proves the only mechanism." If the social-comparison arm beats the information arm, social comparison plays a role. Smaller effects from other channels are still on the table.
• "A null effect means nothing is going on." Women win as often as men, and there can still be discrimination. Competing mechanisms or selection can hide real forces behind a null.
• Coefficients on mediators and controls are associations. γ2 is included to attempt a decomposition. It is confounded and should never be read as the causal effect of the mediator.

Worked example

A province randomizes a job-training program and studies annual earnings. Candidate mechanism: earning an industry certification.

• Short regression: earnings = α1 + β1 * training gives β1 = 4000.
• Long regression: earnings = α2 + β2 * training + γ2 * certification gives β2 = 2500 and γ2 = 3000.

Mediation arithmetic: share through certification = (β1 - β2) / β1 = (4000 - 2500) / 4000 = 0.375, so mediation analysis claims 37.5 percent of the effect flows through certification, and 2500 is the "direct" effect.

Now the grain of salt. Suppose certification has zero causal effect on earnings, and the trainees who complete certification are simply the most motivated ones, who would have earned more anyway. Then γ2 = 3000 reflects motivation, the drop from 4000 to 2500 reflects the same selection, and the 37.5 percent "mechanism share" is an artifact. Randomizing training does nothing to fix this, because certification itself was never randomized. Better moves: (1) test whether training raises certification rates at all (intermediate outcome), (2) derive an independent prediction (if certification is the channel, gains should concentrate in occupations that require the certificate), or (3) redesign with arms (training with exam access versus training without).

How the exam asks this

The professor's Chapter 14 exercise deck runs one long policy scenario (household water-use reports with an information mechanism versus a social-comparison mechanism) and asks 4 to 8 interpretation MCQs. Expect these recurring patterns:

• Read the arms, pick the guarded conclusion. You get results by treatment arm (comparison version cuts usage a lot, information-only version a little) and must choose the answer saying one mechanism "appears stronger, though the other may also operate." Options claiming a mechanism is "proven," "the only mechanism," "definitively" shown, or that mechanisms "must be equally strong" are always wrong.
• Spot the post-treatment control. An analyst proposes outcome = α + β*T + γ*Awareness + ε where awareness is surveyed after treatment. The correct answer says the mediator is post-treatment and possibly confounded with unobservables, so controlling for it distorts the treatment estimate and blocks part of the causal pathway. Distractors say the variable "is not numeric," "is too subjective," or "cannot vary."
• Counterfactual phrasing. Which question captures the mechanism logic? The right form is "would the outcome still change if households got the treatment while the mediator (their beliefs) stayed unchanged?" Distractors swap in irrelevant counterfactuals (richer households, different pricing).
• Pick the informative intermediate-outcome test. The right test checks whether the treatment moved the mediator, or whether outcome changes line up with mediator changes. Distractors test irrelevant correlates (liking the report's graphic design).
• Pick the design. The clean answer randomizes households into one arm per mechanism plus a control, then compares arms. Distractors randomize something irrelevant (billing cycles) or sacrifice the design for power (treat only the top quartile).
• Why mediation decomposition fails. The right answer: mediators are observed post-treatment and may be confounded, so the required assumptions are unrealistically strong. Distractors invent fake rules (mediators must be continuous, one mediator can never matter).
• Midterm-style furniture carries over. Regression tables with Estimate, Std. Error, and t value, lm() and ggplot code completions (geom_smooth(method = "lm")), short-versus-long regression language, and the standing rule that coefficients on controls are associations, never causal effects.

Lecture 15: Turning Statistics into Substance

This lecture opens the last part of the course: what to actually do with quantitative evidence. You now know how to measure correlations, worry about confounders, and estimate causal effects. The point of all that work is better decisions, and this chapter's core message is that a statistic is only useful when it maps onto the substantive thing you care about. Evidence alone never tells you what to do: you have to combine it with a clear objective, your prior beliefs, and your values.

Match the statistic to the objective

The lecture's flagship example is fuel efficiency. Suppose the EPA can improve small sedans by 2 MPG or large SUVs by 2 MPG (equal numbers of each, driven equally). It is tempting to call them equally effective. But the objective is fewer gallons of gas burned, and MPG is inversely related to gallons. Convert to the statistic that matches the objective:

gallons per year = miles driven / MPG

• Sedan, no regulation: 30 MPG at 10,000 miles gives 10,000 / 30 = 333 gallons per year. With regulation: 10,000 / 32 = 313 gallons. Saving: about 20 gallons per car.
• SUV, no regulation: 10 MPG gives 10,000 / 10 = 1,000 gallons per year. With regulation: 10,000 / 12 = 833 gallons. Saving: about 167 gallons per car.

The identical 2 MPG improvement saves roughly eight times more fuel on SUVs, because a 1/MPG relationship makes gains at low MPG worth far more. Bad statistics lead to bad decisions: always ask what quantity the decision actually turns on, then convert.

Percents and percentage points

Changes are often reported as percent changes:

percent change = (new - old) / old * 100

Example from the slides: fatalities rising from 200 to 300 is (300 - 200) / 200 * 100 = 50, a 50 percent increase. Percent changes come with several traps:

• Small base: 1 shark attack last year and 3 this year is a 200 percent increase, and also just 2 extra attacks. Big relative changes can be substantively tiny.
• Zero base: if last year had 0 attacks, the percent change is undefined.
• Negative values: percent changes of outcomes that can be negative produce nonsense.
• Percent vs percentage point: a drug "reduced serious cardiac problems by 44 percent." The control group's rate was 2.75 percent, so the absolute reduction is about 1.2 percentage points (the slides round this to about 1 percentage point). The percentage point reduction is what matters for weighing the drug's expected benefit against its costs and risks.
• Nonsensical shares of a net change: Governor Walker claimed "over 50 percent of U.S. job growth in June came from Wisconsin." U.S. net growth was about 19,000 and Wisconsin's was about 9,500, so the arithmetic checks out. But 5 states added more jobs than Wisconsin, and by the same logic 70 percent of U.S. job growth came from Minnesota and 150 percent came from California. When gains and losses offset each other in a net total, shares of that net can exceed 100 percent and mean almost nothing.

Reading figures without being fooled

Beautiful graphics are never a substitute for clear thinking. When you see a figure, ask: what data and analysis produced it, are the assumptions sound, would another statistic or visualization be more informative, does it address the question of interest, is the scale appropriate, and are there distracting features designed to mislead? Truncated axes are the classic trick: they make small differences look enormous. Rules of thumb for making figures: keep it simple, focus on substance, use a table when it communicates just as well (a table compares 89 and 90 fine), show the data, and convey uncertainty when possible.

Bayes' Rule: updating beliefs with evidence

Quantitative evidence alone never tells you what your beliefs should be, but Bayes' Rule tells you how to update them:

Pr(A|B) = Pr(A) * Pr(B|A) / Pr(B)

Pr(B) = Pr(A) * Pr(B|A) + Pr(not A) * Pr(B|not A)

In claim-and-evidence language:

Pr(Claim|Evidence) = Pr(Claim) * Pr(Evidence|Claim) / Pr(Evidence)

The lecture's centerpiece is People v Collins (Los Angeles, 1964). A couple matched an eyewitness description (blonde ponytail, yellow car, Black man with beard and mustache). A mathematician multiplied the population probabilities of each feature and testified there was a 1 in 12 million chance the couple was innocent. Two things are wrong:

• The features are dependent, so you cannot multiply their probabilities: Pr(beard and mustache) != Pr(beard) * Pr(mustache).
• More fundamentally, it answers the wrong question. The mathematician estimated Pr(Match|Innocent). The court needed Pr(Innocent|Match). Those are completely different quantities, and swapping them is the prosecutor's fallacy.

Bayes' Rule converts one into the other:

Pr(Innocent|Match) = Pr(Innocent) * Pr(Match|Innocent) / [Pr(Innocent) * Pr(Match|Innocent) + Pr(Guilty) * Pr(Match|Guilty)]

Plug in generous numbers: 2,000,000 innocent couples in LA and 1 guilty couple, Pr(Match|Innocent) = 1 in 1,000,000, and Pr(Match|Guilty) = 1. Then among all matching couples you expect 2 innocent matches and 1 guilty match, so Pr(Innocent|Match) = 2/3. Even with a one-in-a-million match probability, the couple is more likely innocent than guilty, because Pr(Innocent) starts out extremely high. You can always reason this way without the formula: count the expected number of couples in each scenario, then condition on what you observed.

Bayes' Rule and statistical significance

The same logic answers: how confident should I be that an effect is genuine, given a statistically significant result? Map the pieces: the claim is "the relationship is real," the evidence is "the study found a significant result." Then:

Pr(real | significant) = (Power * Prior) / (Power * Prior + Significance * (1 - Prior))

• Prior = Pr(relationship real): your belief before seeing the study.
• Power = Pr(significant result | relationship real): the chance the study detects a real effect.
• Significance level = Pr(significant result | relationship not real): the false positive rate, usually 0.05.

The denominator is the law of total probability: every significant result comes either from a world where the effect is real (with probability Power × Prior) or from a world where it is not (with probability Significance × (1 - Prior)).

Costs and benefits

Even a genuine, well-estimated benefit does not settle a decision. A quantitative analysis usually measures only one of many relevant costs and benefits. A program with significant benefits may still fail a cost test, may have side effects, and may carry costs that are hard to quantify and depend on people's values. Chapter 17 returns to whether costs and benefits should simply be added up.

Classic traps and misconceptions

• Confusing a percent change with a percentage point change. A 44 percent reduction of a 2.75 percent rate is about 1 percentage point.
• Treating a large relative change on a tiny base as substantively large (the shark attack trap).
• Quoting a share of a net change as if it were meaningful when gains and losses offset (shares can exceed 100 percent).
• Optimizing the headline statistic instead of the objective (maximizing MPG instead of minimizing gallons).
• The prosecutor's fallacy: reporting Pr(evidence|innocent) when the question is Pr(innocent|evidence).
• Multiplying probabilities of dependent events as if they were independent.
• Believing a result significant at the 5 percent level means there is a 95 percent chance the effect is real. The posterior depends on the prior and the power, and with a low prior it can be well under 50 percent.
• Treating power as the posterior, or treating the prior as if the evidence never arrived.
• Jumping from "the posterior is above 50 percent" straight to "act." The decision still requires comparing expected benefits with all the costs.
• Trusting a figure because it looks polished. Check the scale first.

Worked example: is the significant result real?

A supplement study reports a statistically significant benefit at the 5 percent level, with 80 percent power. You are skeptical of the supplement industry, so your prior that any given supplement works is 4 percent. How confident should you be now?

• Numerator: Power * Prior = 0.80 * 0.04 = 0.032
• Denominator: 0.032 + Significance * (1 - Prior) = 0.032 + 0.05 * 0.96 = 0.032 + 0.048 = 0.080
• Posterior: 0.032 / 0.080 = 0.40

Check it with counts, People v Collins style. Imagine 10,000 supplement ideas, of which 400 truly work. Power catches 0.80 * 400 = 320 of the real ones. Of the 9,600 duds, the 5 percent significance level lets through 0.05 * 9,600 = 480 false positives. So among the 800 significant results, only 320 / 800 = 40 percent are genuine. A significant result moved your belief from 4 percent to 40 percent, a huge update, and the supplement is still more likely useless than useful. Both things are true at once.

How the exam asks this

The professor's exercise deck for this chapter is one running consumer scenario (the Omega-X memory supplement: a skeptical you, an enthusiastic friend, and a study headline) followed by a chain of MCQs that walk through Bayes' Rule piece by piece. Expect the same architecture on the final:

• Headline interpretation first: a claim like "memory improved 7 percent" where the correct answer distinguishes a percentage point change (70 to 77) from a relative percent change. Distractors include the relative reading, an invented ratio, and "cannot tell without significance."
• Define the ingredients: one question each on what a prior is (belief before the study), what power is (Pr(result | relationship real)), and what the significance level is (Pr(result | relationship not real)). Distractors swap the conditioning, exactly the People v Collins error.
• Map to the formula: which quantity is the numerator of Bayes' Rule (Power × Prior), which is the denominator (Power × Prior + Significance × (1 - Prior)), and why the total probability decomposition is valid (every result comes from a real-effect world or a no-effect world).
• Plug and chug: compute the posterior with clean numbers (the deck uses prior 0.20, power 0.80, significance 0.05, giving 0.16 / 0.20 = 0.80), then recompute for a friend with a different prior (0.50 gives 0.40 / 0.425, about 0.94). The point being tested: same evidence, different priors, different posteriors. Standard distractors are the power itself (0.80), the unchanged prior, 1 minus significance (0.95), and 0.50.
• Close with judgment: a "which is the best response" question where the friend wants to buy immediately. Correct answers weigh expected benefits against all costs (price, side effects, opportunity cost, the risk the research is wrong). Wrong answers include "posterior above 50 percent means always buy" and "significance means ignore the price."
• Midterm-style seasoning: like the practice midterms, expect an influencer or policy scenario, at least one question on what a p-value or significant result does and does not mean, and possibly a figure or R snippet where the issue is a truncated axis or a statistic that fails to match the objective (an MPG-to-gallons conversion is very plausible as a two-step arithmetic question).

Memorize Posterior = Power * Prior / (Power * Prior + Significance * (1 - Prior)), practice the arithmetic as fractions, and when a conditional probability appears, say out loud which way the conditioning runs before choosing an answer.

Lecture 16: Measure Your Mission

This lecture is about a quiet failure mode: you can run a technically flawless analysis and still answer the wrong question. Before you trust any number, run four alignment checks. Does the outcome you measure match your goal? Does the treatment you study match the action you will actually take? Does the sample you study generalize to the people or places you care about? And once you act, will people strategically adapt in ways that break the relationship you estimated?

1. Outcomes: the danger of partial measures

A partial measure tracks the outcome only where you happen to be looking. Airports measure weapons caught at metal detectors, so smuggling shifts to routes the detectors do not cover, and "weapons detected" can look great while the real risk moves elsewhere. The professor's own exercise deck uses speed cameras: average speed falls 18 percent at camera sites, but drivers reroute to unmonitored roads and collisions can rise in other neighbourhoods. The metric improves while the mission (fewer serious collisions region-wide) may not.

• Ask: where am I not measuring? Could the problem simply relocate there (spatial spillovers)?
• A falling metric at monitored sites can even overstate success, because risk is shifted rather than removed.

2. Outcomes: intermediate outcomes as proxies

Often the true outcome is hard or slow to measure, so we study an intermediate outcome and hope it is a good proxy: poll numbers as a proxy for winning the election, blood pressure as a proxy for heart attacks. Two things can go wrong:

• Correlation is not causation for the proxy either. Basketball playing is correlated with height, but experimentally increasing basketball playing will not make anyone taller. A proxy that correlates with the goal in observational data may not move the goal when you intervene on it.
• Heterogeneity in who responds. The campaign message might only shift poll answers among people who will never vote. The drug might only lower blood pressure in healthy people who were never going to have heart attacks. The proxy moves, the mission does not.

3. Outcomes: does the target match the mission?

Slide example: 3 of the 8 richest people in the world dropped out of college to start tech companies, so your friend plans to do the same to get rich. Two layers of error here. First, the familiar mistakes from Chapters 4, 7, and 9: this is selecting on the dependent variable (looking only at the ultra-rich) with no comparison group and a tiny, wildly unrepresentative sample. Second, a Chapter 16 mistake: even if dropping out did maximize the probability of becoming a top-8 billionaire, is that your goal? A sensible goal is maximizing expected wealth, or minimizing the chance of poverty, and dropping out likely performs terribly on those objectives. Pick the objective first, then measure it.

4. Treatments: short run and long run are different treatments

Researchers found that hot days correspond with lower economic growth, used climate models to project more hot days, and multiplied the two to conclude climate change will cut global incomes 23 percent by 2100. The problem: a short-term weather fluctuation is a different treatment from a long-term change in climate. People and firms have no way to adapt to a random hot Tuesday. Over decades, though, they can adopt air conditioning, change crops, migrate, and redesign cities. The estimated effect of the treatment they studied (weather shocks) need not equal the effect of the treatment they care about (climate change).

5. Samples: external validity

The World Bank's nutrition program in southern India trained mothers to convert limited budgets into better nutrition, and randomized experiments showed it significantly reduced malnourishment. A nearly identical program in Bangladesh failed. A likely reason: in Bangladesh the father and mother-in-law, rather than the mother, control food purchasing and preparation, so training mothers changed nothing. The India RCT was internally valid, yet its result did not transport because the mechanism depended on local context. Always ask whether the treatment, sample, and setting you studied match the ones you will act in.

6. Samples: selection into your sample

Think hard about how observations got into your data. The Dean of Admissions finds SAT scores only weakly predict GPA among enrolled students and proposes ignoring the SAT. But the admissions decision requires knowing whether the SAT predicts performance among applicants. Suppose admission depends on SAT scores plus other qualities (essays, grit, recommendations). Among admitted students, someone with a mediocre SAT must have been strong on the other things to get in, so within the selected sample, SAT scores and those other qualities are negatively related. That induced negative relationship masks the SAT's real predictive power in the applicant pool.

Same logic, baseball edition: in 2017 MLB pitchers batted .125 versus .259 for other positions, yet in Chicago-area high schools in 2018 pitchers hit .322 versus .317 for everyone else. High school pitchers are simply the best athletes and hit fine. The major leagues select pitchers purely on pitching ability, so hitting ability is ignored at the selection step, and the selected sample of MLB pitchers ends up below average at hitting. The gap tells you about the selection process, and it tells you nothing like "pitching skill causes bad hitting".

7. Strategic adaptation

Once a measure becomes the basis for policy, people respond to the policy, and previously stable relationships can break. Slide examples: England's window tax (window count was a fine proxy for wealth until it was taxed, then people bricked up windows), the infield shift in baseball (positioning based on past spray charts changed batters' incentives), and the war on drugs (crackdowns on one drug or route push activity to substitutes). Always ask: if I act on this relationship, who has an incentive to change behaviour, and does my analysis account for that?

Formulas

Lecture 16 introduces no new formulas; it is a chapter about judgment. Two course formulas still do the work in the background. Selection into a sample or into treatment shows up as the bias term in Difference in means = ATT + Bias + Noise, and the selected-sample examples are statements about how selection changes corr(x, y) = cov(x, y) / (sd(x) * sd(y)) inside the sample you happen to observe.

Worked example: how selection weakens a predictor

Suppose every applicant has an SAT score and a grit score, each from 1 to 4, and the two are unrelated in the applicant pool. True college performance is performance = SAT + grit. The college admits anyone with SAT + grit >= 6. The admitted students are:

Look at the pattern: the admitted student with SAT 2 has grit 4, while students with SAT 4 have grit as low as 2. Among admits, cov(SAT, grit) is about -0.28, even though it was zero among applicants. The consequence: in the applicant pool, corr(SAT, performance) is about 0.71, but among admitted students it falls to 0.50 (and with a stricter cutoff it shrinks further). The Dean who studies only enrolled students concludes the SAT is a weak predictor when it is actually doing exactly its job in the pool where the decision is made.

Classic traps and misconceptions

• "The metric improved, so the mission is succeeding." Partial measures can improve while the underlying problem relocates or worsens.
• "The proxy correlates with the goal, so moving the proxy moves the goal." Basketball and height. Intervening on a correlated proxy is a causal claim that needs its own evidence.
• Ignoring heterogeneity. A treatment that moves the proxy only for people irrelevant to the mission (non-voters, healthy patients) looks successful and achieves nothing.
• Selecting on the dependent variable and choosing the wrong objective. Studying only billionaires, and forgetting to ask whether "chance of being top 8" is even the quantity you want to maximize.
• Extrapolating short-run estimates to long-run policies. Weather effects are estimates for a treatment without adaptation; climate change is a treatment with decades of adaptation.
• Assuming what worked there works here. An internally valid experiment can fail to generalize when the mechanism depends on context (India versus Bangladesh).
• Evaluating a predictor inside a selected sample. Selection on multiple criteria induces negative correlation among those criteria within the sample, weakening each predictor's apparent power (SAT dean, MLB pitchers).
• Forgetting strategic adaptation. When a measure becomes a target for policy, people game it, and pre-policy relationships stop holding (window tax, the shift, war on drugs).

How the exam asks this

The professor's Chapter 16 exercise deck is one long running scenario (automated speed cameras in Niagara Region) followed by interpretation MCQs, and the midterms use the same format with influencer claims and business datasets. Expect these patterns:

• "Which critique is most appropriate?" A policy shows an improvement in a narrow metric and an official declares success. The correct answer points at mission misalignment (the metric may not capture the broader goal). Distractors are technical measurement error, "sample too small", and seasonal or volatility complaints: all defensible-sounding, all beside the point.
• "Which risk does this pattern illustrate?" Behaviour shifts around the measurement (rerouting drivers, relocated vaping). The answer is strategic adaptation or spatial spillovers. Distractors: attrition bias, reverse causality, regression to the mean.
• "Which outcome is most aligned with the mission?" Pick the region-wide, representative measure of the actual goal (severe collisions across the whole region) over site-specific metrics, activity counts (tickets issued), or opinion surveys.
• "Why might the estimate overstate the true impact?" Look for answers where risk is displaced rather than reduced. Note the mirror question: general deterrence at unmonitored sites would make the site-specific estimate understate total impact, so read the direction carefully.
• Selected-sample questions. A dean, employer, or league studies a predictor inside the already-selected group and concludes the predictor is useless. You must name the applicants-versus-attendees problem and the induced negative correlation among selection criteria.
• Recommendation questions. "To better evaluate progress, the Region should..." The answer widens measurement to the mission (region-wide severe collisions, a representative sample of road types), and rejects options that intensify the narrow metric (more sensitive cameras, more tickets).
• On midterms these ideas also hide inside regression questions: a small, insignificant coefficient estimated on a selected sample, where the trap answer treats a large p-value as proof of no effect and the right answer flags both the selection problem and what a p-value can and cannot say.

Exam strategy: state the mission first, in one sentence. Then check outcome, treatment, sample, and adaptation, in that order. The correct option is almost always the one that connects the measured quantity back to the stated mission.

Lecture 17: The Limits of Quantification

This is the final chapter of Thinking Clearly with Data, and it flips the course's usual question. For sixteen lectures the question was "how do we learn from data?" This lecture asks "what can data never tell us?" There are two big answers: sometimes the evidence is too weak to settle a question, and even perfect evidence cannot tell you what your goals and values should be. Every exam question on this chapter tests whether you can spot where the numbers stop and the value judgments begin.

Making decisions with limited evidence

Quantitative analysts tend to "look where the light is": they study questions where good data happens to exist. That means many important empirical questions have no conclusive answer, and you will still have to make decisions about them.

The single most testable idea in this lecture: the phrase "there is no evidence for X" can mean two very different things, and you must diagnose which one you are in.

• Situation 1: Nobody has run a compelling study. We have little evidence either way. X might be true, X might be false.
• Situation 2: Rigorous, high-powered studies have been run, and they give strong evidence that X is wrong.

These call for completely different responses, yet headlines collapse them into the same phrase. The bridge back to earlier chapters: failure to reject the null is never proof of the null. A statistically insignificant estimate with a wide confidence interval is consistent with a zero effect and also with a large effect. It is an absence of evidence, and absence of evidence is weak evidence of absence at best.

The flossing case study. In 2016 the New York Times reported that there was only "very unreliable" evidence that flossing reduces plaque. Wright's slides walk through why you might floss anyway, and each reason is a diagnostic tool you should be able to apply to any "no evidence" claim:

• The experiments had small sample sizes, so estimates were imprecise and statistical power was low.
• Compliance was low (people assigned to floss often did not), which dilutes the measured effect and hurts power further.
• The studies only measured short-term outcomes, while the benefits of flossing (avoiding decay and gum disease) accumulate over years.
• There are good theoretical and biological reasons to expect flossing to help.
• The evidence was more mixed than the headline: the studies did find statistically significant reductions in gingivitis, and they never examined tooth decay or gum separation at all.

The lesson: when evidence is inconclusive, combine whatever evidence you have with theory and substantive knowledge, then make your best decision. Doing nothing, or defaulting to the status quo, is itself a decision, and inconclusive evidence does not automatically support it.

The OIRA example. The U.S. Office of Information and Regulatory Affairs requires a cost-benefit analysis before approving regulations. If no study convincingly quantifies the harm of a particular pollutant, OIRA ignores that pollutant. Notice what this does: an unmeasured benefit gets an implicit weight of zero, which is a substantive (and probably wrong) assumption smuggled in as procedure. The inclination to quantify is good, but it should never stop us from acting on benefits we reasonably believe exist just because we cannot measure them precisely.

Quantification and values

Quantitative evidence should serve our goals and values. But if we are careless, the desire to quantify starts steering the values instead. This happens two ways.

1. Quantification can sneak in values we do not share: algorithmic bias. Health-care providers use algorithms to enroll high-need patients in high-cost care programs. A standard approach predicts a patient's future health-care costs from previous claims, diagnoses, procedures, and medications. Race is deliberately excluded to avoid racial bias. It fails anyway. Why? Because predicted cost is a proxy for access, and access differs by race. Patients from groups with historically worse access to care spend less money for the same level of sickness. The algorithm reads their low past spending as low need, so equally sick patients get scored below the enrollment cutoff. Dropping the race variable did not remove the bias; it hid the bias inside the outcome the algorithm was built to predict. This is the famous Obermeyer et al. (2019) result shown in the slides.

2. Quantification can shape our values in undesirable ways: cost-benefit analysis. Tempting argument: quantify all costs and benefits, add them up, and let the data decide. No values required. The chapter's rebuttal: who says we want the policy that maximizes the simple sum of costs and benefits? That objective is itself a value choice (the book calls it being a "crass utilitarian"). Summing ignores distribution: who pays the costs, who receives the benefits, and when. The slides give two sharpening examples: should we remove children from abusive homes (the child's welfare is not just a line item to net against costs), and should rich countries dump toxic waste on poor countries (a raw dollar-sum CBA can say yes, and most people's values say no). Quantifying is fine. Pretending the aggregation rule is value-free is the error.

The closing charge of the course. These skills could be used to exploit people who have not learned to think clearly with data. Do the opposite: be transparent about the strengths and weaknesses of your evidence, be clear about what the data can and cannot show, and be honest about which parts of your conclusion came from your values rather than from the evidence. The last slide also lists the course's greatest hits, which double as a final-exam checklist: the philosophy of causation, selecting on the dependent variable, substantive versus statistical significance, reversion to the mean, publication bias, why correlation is not evidence of causation, and how to get credible causal estimates.

Formulas

Lecture 17 introduces no new formulas. Two earlier tools get reused when exam questions dress this chapter's ideas in numbers:

• The 95 percent confidence interval, for judging whether "no evidence" means imprecision: CI95 = [beta_hat - 2*SE(beta_hat), beta_hat + 2*SE(beta_hat)]
• The logic of the p-value: the probability of an estimate at least this extreme if the null were true, which is why an insignificant result never proves the null.

The professor's Chapter 17 exercise deck also uses two invented decision-rule formulas, and the point of both is that they embed value judgments:

• A fiscal ranking ratio: Score = ProjectedTaxRevenueAfterYear5 / TaxRevenueWaived (counting only fiscal quantities is itself a choice about what matters)
• A weighted scoring algorithm: Score = 0.6 * PredictedJobs + 0.4 * ProjectedLongTermTaxRevenue (the weights 0.6 and 0.4 are a political statement about how much jobs matter relative to revenue)

Classic traps and misconceptions

• "No evidence for X, therefore X is false." Diagnose first: is this a strong-evidence-against situation or a weak-evidence-either-way situation? Small samples, low compliance, and short follow-up all point to the second.
• "The estimate is insignificant, so the effect is zero." Failure to reject the null is never proof of the null. Check the confidence interval: if it contains both zero and substantively large effects, the study is simply uninformative.
• "The evidence is inconclusive, so do nothing." The status quo is also a choice. The chapter says to combine the weak evidence with theory and substantive knowledge and decide.
• "We removed the sensitive variable, so the algorithm cannot be biased." Bias enters through proxies. Past health spending proxies for access to care, so predicting cost reproduces racial disparities without any race variable.
• "Algorithms and formulas are value-neutral." Someone chose the objective, the inputs, and the weights. Each of those choices encodes what (and who) counts.
• "If we cannot quantify it, leave it out." Leaving it out assigns it an implicit weight of zero, which biases the whole evaluation toward whatever is easy to measure (the OIRA problem).
• "Just add up all costs and benefits and pick the maximum." That aggregation rule is crass utilitarianism: a value system, chosen silently. It ignores distribution, fairness, and rights.
• Exercise-deck distractor pattern: the wrong options are usually absolutist ("a single metric is never useful", "algorithms are objective", "future residents cannot be considered"). The correct option is almost always the nuanced one that keeps the quantitative tool while naming the hidden value choice.

Worked example: weights are values

Niagara Regional Council must give a tax waiver to exactly one of two applicant firms, ranked by the algorithm Score = w1 * PredictedJobs + w2 * ProjectedRevenue. A mayor claims this "removes politics from the decision." Here are the applicants (both inputs on comparable scales):

With the proposed weights w1 = 0.6, w2 = 0.4:

• Firm A: 0.6*90 + 0.4*20 = 54 + 8 = 62
• Firm B: 0.6*40 + 0.4*85 = 24 + 34 = 58

Firm A wins. Now suppose Council instead values long-run revenue more: w1 = 0.3, w2 = 0.7:

• Firm A: 0.3*90 + 0.7*20 = 27 + 14 = 41
• Firm B: 0.3*40 + 0.7*85 = 12 + 59.5 = 71.5

Firm B wins. Nothing about the data changed; only the weights did. The winner of the waiver is determined by how much the Region values jobs relative to revenue, which is exactly the political question the algorithm claimed to eliminate. The algorithm did not remove the value judgment. It froze one particular value judgment into arithmetic and hid it from the debate. The honest fix is transparency: state the weights, state what was left out (job quality, who benefits, neighbourhood effects), and let Council argue about the values directly.

How the exam asks this

The professor's Chapter 17 exercise deck is one long policy scenario (a Niagara property tax waiver for new businesses) followed by eight to ten interpretation questions with stems like "Which critique is strongest?", "Which reply is most accurate?", and "Which answer best reflects the Chapter 17 framework?" Expect the final to reuse this format. The recurring question patterns:

• Single-metric critiques: someone proposes judging a policy by one number (net jobs in 3 years). The right answer says the metric undervalues other outcomes the decision-maker also cares about. Wrong answers say single metrics are "never" useful or that the metric is "unrelated" to anything.
• "Value-neutral" formula or algorithm claims: a ratio or weighted score is offered as objective. The right answer identifies the embedded value choice (what is counted, how it is weighted, whose welfare matters). Wrong answers either declare algorithms objective or demand the weights be "estimated statistically."
• Distributional reasoning: options compare who benefits (large external firms versus small local owners, current versus future residents, vulnerable neighbourhoods). The right answer is the one about the distribution of benefits and costs, and it is distinct from options about totals or averages.
• Hard-to-quantify effects: revitalized corridors, neighbourhood identity, displacement of existing firms. The right answer includes them and names the bias from omitting them; wrong answers exclude them entirely or bury them in a fake score.
• The synthesis question: how to use quantitative tools without letting them override values. The answer is always transparency: document what is included, what is excluded, and whose welfare is weighted, so value choices stay visible.
• Midterm-style crossovers: expect "no evidence" headlines paired with a small regression table (Estimate, Std. Error, t value) where you must compute a wide confidence interval, correctly state what a p-value does and does not mean, and conclude that the study is uninformative rather than proof of no effect. The flossing rebuttals (small n, low compliance, short horizon, supportive theory, unmeasured outcomes) are a ready-made checklist for these.

One reliable tiebreaker across all of these: the correct option almost always keeps quantification while making the hidden value judgment explicit. Options containing "never", "always", "cannot", "guaranteed", or "purely objective" are nearly always distractors.