Ugg draw ONE line through berry-cloud, feed whole tribe

Long ago, tribe carry too many rock: t-test rock, ANOVA rock, correlation rock, one for each hunt. Ugg back hurt. Then elder show ONE master tool: Regression — Ugg draw one straight line (or flat stone-sheet, when many things) through the cloud of dots so line can guess Y from X. This one tool eat all other tool. t-test, ANOVA, ANCOVA, correlation — all just baby regression in costume. Elder call the big family General Linear Model (GLM). Ugg call it "no more back hurt."

How Ugg pick the best line

Dots: X = hours hunt, Y = meat caught. Ugg want line to guess meat from hours.

Intercept (${\beta }_0$) — where line start when X is zero. Slope (${\beta }_1$) — how much meat go up per one extra hour.

But INFINITE lines Ugg could draw! Which best? Each dot miss the line a little. That miss is residual (${\epsilon }_i=Y_i-{\hat{Y}}_i$) — the up-down gap from dot to line. Some miss up, some miss down. Add them raw → they cancel, useless. Use plain distance → math turn ugly and pointy. So Ugg SQUARE each miss, then pile them up:

Pick the line that make this pile SMALLEST. That line is OLS (Ordinary Least Squares). Squaring good — it punish big miss more than small, and give smooth answer Ugg solve with fire-math:

Slope = how X and Y move together, divided by how much X wiggle alone. Line must pass through the middle-dot ($\bar{X},\bar{Y}$), and that fix the start-point. Secret: if Ugg first shrink X and Y to same z-size, the slope become EXACTLY Pearson's r (correlation). And with one X, $R^2=r^2$. Same beast, different fur.

Many X, one idea

Add more: hours, brain-size, sleep. Line become flat stone-sheet:

Magic word: each ${\beta }_j$ mean "how much Y move per one-step of $X_j$, holding all other X frozen still." Not raw friendship — the UNIQUE push of that one X after the others already counted. This freezing is why regression smarter than plain pairwise correlation.

Does the line hunt well?

R² (coefficient of determination) — the headline shiny-stone:

How much of the meat-wiggle the line explain. 0 = useless (just guess average meat for all). 1 = every dot sit on line. R² = 0.65 mean "line explain 65 of every 100 wiggle."

Ugg WARN: R² sneaky! It NEVER go down when Ugg add more X — even garbage X, even random pebble, R² only climb. So use Adjusted R²:

This one charge a tax for every X ($k$ = how many X). Useless X barely move R² but pay big tax → adjusted R² go DOWN. Honest stone for comparing models. Ugg WARN: never compare models by raw R² when they have different number of X!

Three question-fires

1. Model F-test — does whole line beat "just guess the average"? $F={\text{MS}}_{\text{reg}}/{\text{MS}}_{\text{res}}$. 2. Coefficient t-test — does each X pull its own weight? $t={\hat{\beta }}_j/\text{SE}({\hat{\beta }}_j)$, with $df=n-k-1$. 3. Confidence interval — ${\hat{\beta }}_j\pm t^{\ast }\cdot \text{SE}({\hat{\beta }}_j)$. If this range skip past zero, that X matter.

Ugg WARN: strong overall F do NOT mean every X good — still check each t! To compare X of different size (meat in kg vs brain in points), z-shrink all → Standardised coefficient (β): "one SD of X push Y this many SD." Now Ugg compare fair. (Careful: not same as $\beta /\text{SE}$ — that one is the t-number.)

Six rules, or the line lie

Line tell truth only if six rules hold — Ugg call them LINeM + one:

• Linearity — coefficients enter straight (X may still bend with square or log, that fine).
• Independence — each dot its own dot, not same mammoth counted twice.
• Normality of residuals — the MISSES bell-shaped. NOT X, NOT Y. Ugg WARN loud: tribe always check X or Y — WRONG! Check the residual (use Q-Q plot).
• Equal variance — Homoscedasticity: miss-spread same width everywhere. When spread grow like a fan → Heteroscedasticity. Fix: log the Y, or use tough robust HC standard errors (keep β, fix the SE).
• no Multicollinearity — X not be twins. Check VIF > 5–10 = bad. Twins flip signs and make single β lie. (For guessing future meat, twins okay; for knowing WHICH X matter, twins bite.)
• Plus Exogeneity ($E[\epsilon \mid X]=0$) — no hidden confounder-beast pushing both X and error. Deepest rule. Random-assign in experiment force it true.

Ugg's four looking-plots

After every line, Ugg stare at four plots:(1) residual-vs-fitted (bend = linearity broke, fan = hetero),(2) Q-Q (straight diagonal = normal miss),(3) scale-location (sharper fan-check),(4) residual-vs-leverage with Cook's distance — one dot with Cook's d > 1 is a bully dragging the whole line. Ugg WARN: look and understand, NEVER silently throw a dot in the fire.

More stone-tricks

Dummy coding: for a group-thing with k tribes, make $k-1$ little 0/1 dummy stones; the left-out tribe is the reference, and each β = that tribe minus reference. Ugg WARN: never make k dummy for k tribe — that the trap!

Simpson's paradox: education→money look mighty, β = 5000. Add job-type, β crash to 800. The push was really "school lead to fancy job," not school itself. Add or drop the wrong X and the whole story can flip — so Ugg pick X by thinking, not by luck.

Pick the simplest line that fit well (Occam razor). Judge with adjusted R², AIC (lower better; gap > 10 strong), or a nested F-test to see if extra X earn their place. Stepwise auto-pick is only a hint, never the law.

Ugg remember

• OLS = draw the line that make Σ(miss)² smallest. Slope $=\text{cov}(X,Y)/\text{Var}(X)$; line passes through the middle-dot.
• Each β = change in Y per one-step X, holding other X frozen. With one X only: $R^2=r^2$ and standardised β $=r$.
• R² always climbs with more X — judge honest with adjusted R², AIC, or nested F. Coefficient t-test df $=n-k-1$.
• Six rules (LINeM + exogeneity). Normality is about the RESIDUAL, not X or Y. Fan-shape = heteroscedasticity → robust SE.
• Always eyeball the residual plots; Cook's d > 1 = bully dot. And remember — t-test, ANOVA, ANCOVA, correlation are all just regression.