Overview
This video explains Google's Gemini Embedding 2 — the first natively multimodal embedding model — and why simply plugging it into an existing RAG pipeline doesn't deliver the results most people expect. The presenter walks through how embeddings and vector databases work, identifies the critical gap in naive multimodal RAG architectures, and demonstrates a proper pipeline that pairs video embeddings with Gemini-generated text descriptions for meaningful question-answering.
Key Concepts
What is Gemini Embedding 2?
- Google's first natively multimodal embedding model
- Can embed text, images, videos, audio, and documents into vector databases
- Previously, non-text data required hacky workarounds (e.g., embedding text descriptions of videos)
- Limitations: videos capped at 120 seconds per chunk, text up to 8,192 tokens
How Embeddings and Vector Databases Work
- An embedding model converts data into a vector — a point in high-dimensional space (1,526 dimensions for this model)
- Vectors are placed based on semantic meaning — similar content clusters together
- When you query, your question becomes a vector and the database finds the nearest matches
- The matched vectors return their paired documents for the LLM to use in its answer
The Problem with Naive Video RAG
- When a video is embedded as-is, the retrieved result is just a video clip — not a text answer
- Most LLMs (except Gemini in specific contexts) cannot ingest and analyze raw video to generate text responses
- The naive approach returns "here's the clip, good luck" instead of a detailed text-based answer
- Even with Gemini, re-analyzing the video at query time is slow and wasteful
The Correct Architecture
- During ingestion (not query time), run non-text media through Gemini to generate text descriptions/transcripts
- Store both the video embedding and its text description as paired data in the vector database
- When a query retrieves a video vector, the LLM gets the accompanying text to generate a real answer, plus the video clip as a media reference
- This is a front-end pipeline enhancement — process once at ingestion, benefit every query
Video Chunking — An Unsolved Challenge
- Long videos must be split into chunks (like text chunking in traditional RAG)
- The presenter's approach: 2-minute segments with 30-second overlap
- Intelligent video chunking (e.g., by topic/scene) remains an open problem
- Re-rankers and more sophisticated strategies will likely emerge
Implementation
- A GitHub repo is provided with the complete multimodal RAG architecture
- Two setup paths: clone the repo, or copy a Claude Code blueprint prompt
- Stack: Python, FFmpeg, Supabase (vector DB), Gemini API
- Supabase can be swapped for Pinecone or other vector databases
- Claude Code handles database setup and configuration automatically
Key Takeaway
Embedding 2 is a genuine leap forward for multimodal RAG, but only if you architect the ingestion pipeline correctly. The embedding model's job is search and similarity (finding the right content), not explanation (generating answers). You need Gemini on the ingestion side to bridge that gap — creating text descriptions that travel with the video through the vector database so the LLM can actually answer questions about it.