Saral Shiksha Yojna
Courses/Computer Vision

Computer Vision

CSE471
Prof. Makarand Tapaswi + Prof. Charu Sharma•Spring 2025-26•4 credits
Revision Notes/Unit 11 — Multimodal LLMs (PaliGemma / Qwen2-VL / Gemma 4)

Unit 11 — Multimodal LLMs (PaliGemma / Qwen2-VL / Gemma 4)

Vision-Language Models: the 3-pillar blueprint, Prefix-LM masking, SigLIP vs CLIP loss, dynamic resolution + M-RoPE for video, and the move toward native multimodal models.

VLM Architecture — Encoders, Connectors, Positional Encoding

14 min

Modern VLMs stitch a pretrained vision encoder, a small connector, and a pretrained LLM together — three pillars. The connector is the only randomly initialised piece. Then mask attention so the image+prompt are 'free context' (bidirectional) and only the answer is generated causally.