Building Multimodal RAG for Production App
If you’ve played with RAG (Retrieval-Augmented Generation) in demos, you already know it feels powerful.
But taking it to production, handling real users, real data, and real costs is different. In this episode Juan Rivillas demonstrated how he built a multimodal RAG system (text, images, videos) and what you can learn from it.
What Makes RAG So Useful in Real Apps? #
Let’s start simple. As Juan explains:
Juan: RAG is a technique you use in AI applications. You take your data, files, images, videos, convert them into embeddings, and store them in a vector database. Then you convert the user query into an embedding and compare both to find relevant results.
Think of it like this:
- Traditional search → exact matches (SQL, keywords)
- RAG → semantic matches (meaning, context)
That’s why it works so well for modern apps.
Juan: It made me feel that I could have a search engine in my hands with a very easy implementation.
From Text to Multimodal: Searching Images and Videos #
Most tutorials stop at text. But real apps need more.
Juan built a system that searches:
- Images
- Videos
- Natural language queries (like “mom’s birthday”)
How Image Search Works #
Instead of storing raw images directly:
- Convert image → small thumbnail
- Send thumbnail to an LLM
- Generate a text description
- Store that description as an embedding
Juan: I convert images into very tiny thumbnails and ask OpenAI, ‘What’s inside this?’ It returns a very informative description.
This gives you:
- Lower cost (small images)
- Fast processing
- Good semantic understanding
How Video Search Works #
Videos are more complex. Juan breaks them into layers:
- Visual layer → snapshots (frames)
- Audio layer (optional)
Juan: I take snapshots of the video, three to six images, and send them together. The model can interpret the sequence and infer what happens in the video.
So instead of processing thousands of frames:
- Sample a few frames
- Let the model infer the story
Simple, but effective.
Why Scope Matters More Than You Think #
A big mistake in RAG systems is trying to do everything.
Juan avoided that.
Juan: Having something really generic leads to worse results. If you scope it like family photos or pets, RAG works much better.
His app focuses only on:
- Families
- Babies
- Life moments
This improves:
- Prompt quality
- Search accuracy
- User experience
Lesson: Narrow scope = better results.
Hybrid Search: RAG + Traditional Search #
Here’s a practical trick many developers miss.
RAG is not always the best choice.
Juan: RAG doesn’t work well with single words. It tends to hallucinate more. For short queries, I use traditional search.
So the system uses:
- Keyword search → for short queries (e.g., “dog”)
- RAG search → for longer queries (e.g., “playing with my dog in the park”)
This hybrid approach gives:
- Better accuracy
- Faster responses
- Less confusion
Making It Production-Ready #
Building a demo is easy. Production is hard.
Here are key decisions Juan made.
1. Asynchronous Pipelines #
Processing images and videos takes time. So everything runs async.
Juan: I created a multi-pipeline system. First download and thumbnail, then extract information, then store in the database.
Pipeline stages:
- Stage 1: Download + thumbnail
- Stage 2: Metadata extraction + embedding
- Stage 3: Store in DB
This prevents:
- Blocking requests
- Server overload
- Poor user experience
2. Scale Each Step Independently #
Different tasks need different resources.
Juan: You can scale each pipeline independently. The extraction stage consumes more resources than others.
That means:
- CPU-heavy tasks scale separately
- API-heavy tasks scale separately
3. Use the Right Stack (Even If It’s Uncommon) #
Juan used Elixir, which is not typical for AI apps.
Juan: Elixir gives fault tolerance, scalability, and robustness out of the box.
Even if you don’t use Elixir, choose tools that simplify concurrency and scaling.
Keeping Costs Under Control #
Production systems must be cost-efficient.
Key optimizations:
- Use thumbnails instead of full images
- Process data only once (embed once, reuse forever)
- Store embeddings in Postgres with vector extension
Juan: The beautiful thing is you only embed once. All future queries just hit the database.
He also avoided expensive infrastructure:
- S3 → storage
- Cloudflare → caching
- Fly.io → cheaper compute
- Supabase → database
Multimodal RAG Feels Like a Superpower #
One of the most powerful ideas from the session:
Juan: Right now you can search anything, files, images, videos. It’s impressively powerful.
And it really is.
You’re no longer limited to:
- Structured data
- Predefined queries
You can build:
- Smart search engines
- Context-aware apps
- Personalized experiences
AI Helps You Build Faster, But Architecture Still Matters #
AI tools helped speed up development a lot.
Juan: Using tools like Cursor and Claude Code allowed me to focus on architecture instead of small code details.
But here’s the key:
Juan: The architecture you design is what makes the project successful.
So:
- AI writes code
- You design systems
That’s your real value as a developer.
Key Takeaways #
- RAG is basically semantic search for anything
- Multimodal RAG = text + images + videos
- Use LLMs to generate metadata, not just answers
- Combine RAG + traditional search
- Keep your scope small and focused
- Build async pipelines for production
- Optimize costs early (thumbnails, one-time embeddings)
- Focus on architecture, not just code
Final Thoughts #
Multimodal RAG isn’t just a cool experiment, it’s becoming a core pattern for modern applications. If you understand it well, you can build products that feel far more intelligent than traditional systems.