See it in a video, find it in a click
WindowShop_AI
Real-time in-video product discovery: point at a product in a video, find it, buy it.
- Identification accuracy
- 92%
- HackOn with Amazon
- Top 10 national
- Team
- 3, led by me
the problem
You see a product you like in a video and there is no way to find it short of screenshotting and reverse-image searching. For the HackOn with Amazon hackathon, we wanted discovery to happen inside the video itself, in real time, without breaking playback.
the solution
A detection-to-recommendation pipeline: YOLOv8 localizes products frame by frame, CLIP embeds each detection into a shared image-text space, and a similarity search matches it against a product catalog. The whole flow runs behind a Flask REST API that the video player calls as you watch. Product embeddings are precomputed and cached in PyTorch, so per-frame matching stays fast. The pipeline identifies products at 92% accuracy, and the project reached the national semifinals (Top 10 teams).
tech stack
- YOLOv8Real-time object detection on video frames
- CLIPJoint image-text embeddings for product matching
- PyTorchModel inference and cached embedding store
- OpenCVFrame extraction and preprocessing
- FlaskREST API orchestrating detection to recommendation
- JavaScriptIn-player overlay UI
key engineering decisions
Precompute and cache product embeddings
Embedding the catalog on every request would have made real-time matching impossible. Precomputing embeddings once and caching them in PyTorch turned per-frame matching into a fast similarity lookup, which is where most of the latency win came from.
Two-stage detection then matching, not end-to-end
Splitting the problem (YOLOv8 finds products, CLIP identifies them) meant each stage could be swapped and tuned independently. It also meant we could hit 92% accuracy with pretrained models instead of training something end to end during a hackathon.
Backend as the contract for a team of 3
I designed the REST API first so the ML and frontend work could proceed in parallel against a stable contract. In a hackathon, the API surface is the coordination mechanism.