Virtual Lens Try-On
Goggle VTON is a production-grade virtual try-on proof of concept for smart eyewear and goggles. The platform combines MediaPipe face detection (478-point mesh), Three.js WebGL 3D rendering, and a LangGraph agentic pipeline wrapping GPT-4.1 to deliver real-time 3D glasses overlay on a live camera feed, automated face-shape classification, and expert-optician style recommendations. Built to give enterprise clients a hands-on look at what an end-to-end AI try-on experience feels like — from camera frame to fit analysis to personalized SKU picks.
Automated product explainer · MediaPipe face mesh + Three.js overlay in action
Core capabilities
Hands-on features available when you launch the live demo.
Real-time 3D glasses overlay
Live camera feed with head tracking at 30–60 FPS. Three.js + React Three Fiber render GLB goggle models with lighting, shadows, and occlusion.
Photo mode with Lenskart-style browsing
Capture a photo and cycle through 15+ frame models in a grid layout — compare looks side-by-side without re-calibrating the camera.
AI face-shape classification
Geometric analysis of width, height, ratios, and jaw angle classifies the face as oval, round, square, heart, or oblong to power fit scoring.
Expert-optician recommendations
A LangGraph agentic pipeline wraps GPT-4.1 with an expert-optician persona to recommend the top SKUs for the detected face shape and fit.
2D asset upload + auto SVG overlay
Upload a 2D goggle image; the backend runs Pillow + landmark detection to generate an SVG overlay and landmarks automatically.
Multi-photo 3D reconstruction
Upload multiple photos of a frame to trigger a COLMAP-style point cloud + mesh generation pipeline for new SKUs.
Lens tint visualization
Preview clear, blue-light, polarized, sunglass, gradient, and mirror tints on the current frame in real time.
Session persistence + analytics
Every try-on session is persisted via SQLAlchemy + PostgreSQL for history, recommendations memory, and usage analytics.
Technology stack
Every layer of the stack — from database to 3D renderer.
| Technology | Role & contribution |
|---|---|
| Python 3.12 + FastAPI | Async REST API with auto-generated OpenAPI docs |
| PostgreSQL 16 + SQLAlchemy 2.0 | Async ORM for sessions, assets, and reconstruction jobs |
| LangGraph + LangChain + GPT-4.1 | Agentic DAG pipeline for fit analysis + style recommendations |
| MediaPipe (server + client) | 478-point face mesh detection via WASM at 30–60 FPS |
| React 18 + TypeScript + Vite | Hot-reload frontend with type-safe API integration |
| Three.js + @react-three/fiber + drei | WebGL 3D rendering of GLB goggle models with R3F |
| Pillow + trimesh | Image compositing + 3D model dimension extraction |
| Docker Compose v2 | Orchestrates backend, frontend, and PostgreSQL containers |
| Playwright 1.51 + pytest 8 | End-to-end browser tests + backend unit/integration |
Launch the live Virtual Lens Try-On demo
Runs in your browser · camera processed locally · never stored.
Build your own AI experience
Explore the full AIXcelerator platform — agents, skills, MCP servers, and the modular capability layers that power demos like this one.