When many rocks look same, tribe get confused

Ugg use many guess-rocks to hunt the answer. But some rocks TOO ALIKE. Big mess. Sit by fire. Ugg untangle.

The tangle — two rocks carry same berries

Multicollinearity — fat word, mean the guess-rocks (the predictors, things Ugg use to guess with) match EACH OTHER too much. This NOT about a rock matching the prey you hunt (the outcome) — that match is GOOD, that what you want! Trouble is rocks matching one another.

Like: tall-ness and heavy-ness of hunter. Or family food-pile and family water-drink. Two rocks carry SAME berries — bring both to fire, second rock add almost nothing.

Why bad? Ugg count the ways (tribe elders LOVE ask this):

1. Sap the power — need MORE hunt-counts to see same truth. 2. Sign flip! Rock that alone push answer UP can show pushing DOWN when its twin-rock sit beside it. Ugg scratch head. 3. Wobble balloon — the standard error (Ugg's honest wobble-guess) blow up, sure-fences go wide, tests say "not sure." 4. Less sharp on each rock's OWN push. 5. Need bigger hunt-count $n$. 6. Whole trap look strong (F-test loud) even when NO single rock look strong. Sneaky!

Ugg picture: two mammoth on ONE weigh-stone that show only TOTAL. Ugg cannot tell which mammoth heavy. Same berries — machine cannot split the credit.

Sniff the tangle — the VIF footprint

For each rock $j$, Ugg try guess THAT rock from ALL other rocks, get $R_j^2$ (how much others explain it). Then the VIF (Variance Inflation Factor):

Mean: if other rocks explain rock-$j$ lots ($R_j^2$ near 1), bottom go tiny, VIF blow UP. Read footprint: VIF $=1$ → rock stand alone, best. 1 to 5 → little tangle, fine. Over 5 (some say 10) → BAD tangle, fix it.

Cousin trick: SMC (Squared Multiple Correlation) — most of a rock's wobble the others can explain, with ${\text{VIF}}_j=1/(1-{\text{SMC}}_j)$. SMC near 1 → rock is copy. Fast eye-trick: a correlation heat-map — any two rocks with $|r|>0.7$ glow red mean tangle likely.

Fix the tangle

1. Drop one twin-rock (keep the one tribe care more about). 2. Mash rocks into one composite — PCA or FA score, or just add-them-up. 3. Ridge — a tamer-rope that punish big coefficient; trade tiny wrong for less wobble. 4. Hunt more — more data shrink wobble. 5. Center rocks — help tangle made by mix-terms.

Why smash many rocks into few?

Even with no tangle, too many rocks bad. Curse of dimensionality — data Ugg need grow EXPLODING-fast as rocks pile up. Many rocks also mean overfit (trap fit THIS hunt's noise, fail next hunt), slow counting, and eye cannot see past 3 directions. So Ugg squeeze many rocks into few meaning-full ones. Two ways: FA and PCA.

Factor Analysis — the hidden spirit behind the rocks

Factor Analysis (FA) say: rocks Ugg SEE are made BY hidden spirits Ugg CANNOT see — latent factors — and many rocks may be shadow of the SAME spirit. Like "intelligence" no one can touch, but word-smart, number-smart, shape-smart CAN be measured — all share hidden spirit g. Or "customer-happy" hidden, but answers on seat, food, price cluster into few hidden happy-spirits.

The spirit-recipe for each rock $X_i$:

$F_j$ are hidden spirits. ${\lambda }_{ij}$ the factor loadings — how hard spirit-$j$ pull rock-$i$. ${ϵ}_i$ the rock's own private error. Each rock's wobble split into communality $h^2$ (shared part) and private part $1-h^2$, where

Mean: square each loading, add up — that the shared part (spirits do not overlap, so wobble just add).

Factor loadings = match of rock to spirit. Over 0.4 (loose) or 0.6 (strict) = rock belong strong. Cross-loading under 0.3 = rock belong to ONLY one spirit — clean "simple structure." Factor scores = each hunter's score on each spirit; Ugg can feed these to later magic like new rocks.

Explore first, confirm later

EFA (Exploratory Factor Analysis) — Ugg no know spirits yet, let data show. CFA (Confirmatory Factor Analysis) — Ugg already have story of which rock belong which spirit, now TEST if data agree. EFA explore; CFA confirm — wise to do EFA on one tribe-group, CFA on a DIFFERENT group. Ugg warn: no use EFA answer as if it confirm! That cheat, fit only THIS group. (Small note: R-type groups rocks that move together; Q-type groups PEOPLE with same answer-pattern.)

How many spirits keep?

1. A priori — story say how many. 2. Kaiser rule — keep eigenvalue over 1 (eigenvalue = wobble a spirit catch). Crude — grab too many. 3. Scree plot — draw eigenvalues falling big-to-small; keep spirits ABOVE the elbow-bend, throw the flat tail. 4. Parallel analysis — make FAKE random data same shape, get its eigenvalues, keep only real spirits taller than the random ghost. Most trusty. 5. % variance — keep enough to reach target total wobble.

Ugg warn: Kaiser grab too many! Trust parallel analysis > Kaiser > scree. After picking, rotate to read easy: Varimax keep spirits un-linked (orthogonal); Oblimin / Promax let spirits lean together (oblique) when real constructs overlap. Before all: KMO must be over 0.6 (better 0.8) and Bartlett's test must be LOUD (significant) — else no structure to dig. Need continuous rocks, straight-line links, ~5–10 hunters per rock.

PCA — the variance-hungry cousin

Principal Component Analysis (PCA) chase fewer rocks a DIFFERENT way — NO hidden spirit. It build new rocks (components) that are mixes of old ones: PC1 catch the MOST wobble; PC2 un-linked to PC1 catch most LEFT wobble; on down.

Mean: find the arrow-direction $w$ (length 1) that make the spread biggest. Deep magic: components are the eigenvectors of the covariance matrix, and each eigenvalue = wobble that component catch. First few catch most → keep them, toss rest.

PCA vs FA — the trap elders LOVE

Get this or lose easy marks. PCA: no hidden model, ALL wobble chopped into components, components are just MATH shapes. FA: hidden-spirit model, only SHARED wobble modeled plus private error, factors are REAL constructs. Short: PCA smash-and-shrink; FA find hidden spirit. Use PCA to compress or draw picture; use FA to test a real hidden trait.

Two last warnings. If PCA rocks have different units (foot vs shiny-stone), standardize first or the big-unit rock bully all. And a Heywood case — loading over 1 — is IMPOSSIBLE (loadings are matches, live in $[-1,1]$), so it mean broke model or too little data. For CFA, judge fit by MANY numbers, never one: CFI > 0.95, RMSEA < 0.06, SRMR < 0.08 (and χ²/df under 2, 3 ok).

Ugg remember

• Tangle (multicollinearity) is guess-rocks alike EACH OTHER, not rock-to-prey — it balloons wobble and flips signs.
• **VIF $=1/(1-R_j^2)$**, over 5–10 is bad. Fix: drop, mash, ridge, hunt-more, center.
• FA finds hidden spirits ($X_i={\sum }_j{\lambda }_{ij}{F}_j+{ϵ}_i$): loading > 0.4 strong, cross-loading < 0.3 clean, communality $h^2={\sum }_j{\lambda }_{ij}^2$.
• EFA explore, CFA confirm (on a fresh group). Pick spirit-count by parallel analysis > Kaiser > scree; check KMO > 0.6 and Bartlett loud.
• PCA smash wobble into math components (eigenvectors); FA models real hidden spirits. Never mix the two up.