Courses/Computer Vision

Computer Vision

CSE471

Prof. Makarand Tapaswi + Prof. Charu Sharma•Spring 2025-26•4 credits

Revision Notes/Unit 10 — Transformer Advances (ViT-5 era)

Unit 10 — Transformer Advances (ViT-5 era)

The seven upgrades that take you from 2017 Transformer to modern LLM/VLM stack: PreNorm, RMSNorm, LayerScale, QK-Norm, Registers, RoPE, GQA + KV-cache + Flash Attention.

Modern Transformer Upgrades

12 min

Original Transformer (2017) → modern LLM (2024+) is roughly seven independent upgrades. Each fixes a specific failure mode of the original — training stability, attention sharpness, position generalisation, or inference efficiency.