Claude Code Video Vision
A Claude Code plugin that extracts frames via ffmpeg and processes audio via multiple backends: Gemini API, local Whisper, or OpenAI API.
Verified
Core facts
Read by a script from the official API, not rewritten by a model.
- License
- MIT
- Languages
- TypeScript
- Maintainer
- jordanrendric
- Verified
What it is
Claude Code Video Vision is a Claude Code plugin that extracts frames via ffmpeg and processes audio via multiple backends (Gemini API, local Whisper, or OpenAI API). Claude receives frames as images and audio transcription with timestamps. The plugin is a perception layer, not an interpretation layer.
Main features
When a YouTube URL is passed, the MCP server downloads it with yt-dlp, preserving source metadata and captions for context.
Backends let the user choose between cloud APIs or fully local processing.
Claude adjusts fps, time range, and resolution based on the question.
Whisper models download automatically on first use.
The slash command setup-video-vision walks the user through configuration.
In conversation, when a video file or YouTube URL is mentioned, Claude will detect it.
Getting started
The plugin install slash commands are run one at a time inside Claude Code; the setup wizard walkthrough covers backend selection, whisper configuration if local, frame options, and dependency verification.
The Gemini API backend requires a free API key.
The OpenAI backend requires an API key from OpenAI.
The Local backend requires whisper-cpp (macOS) or an equivalent runtime, plus an automatic model download; the model download does not replace installing the runtime.
ffmpeg is auto-detected, and install instructions are provided by the setup wizard.
The example request 'the first second' maps to extraction at original fps from 00:00:00 to 00:00:01.
The example request 'summarize this 1h lecture' maps to low fps and full duration.
/plugin marketplace add https://github.com/jordanrendric/claude-video-vision/plugin install claude-video-vision/claude-video-vision:setup-video-vision/watch-video path/to/video.mp4
/watch-video tutorial.mp4 "what language is used in this tutorial?"
/watch-video https://www.youtube.com/watch?v=... "summarize this video"Who it is for
The MCP server requires Node.js 24 or newer. yt-dlp is optional and required only for YouTube URLs. The plugin exposes 6 MCP tools.
video_watch extracts frames and processes audio, and is the main tool. video_analyze analyzes video structure with ffmpeg filters before extraction. video_detail drills into specific cached or newly extracted moments.
video_info gets video metadata without processing. video_configure changes settings. video_setup checks and guides dependency installation.
The frame format can be jpeg, png, or webp. jpeg remains the default for backwards compatibility. png is useful for screen recordings where text and sharp UI edges should stay lossless.
For YouTube URLs, manual YouTube subtitles are a transcript source when an English track is available. YouTube automatic captions are a transcript source when manual subtitles are not available. The configured audio backend is a transcript source when captions are missing, empty, or cover too little of a longer video.
Audio results label provenance with a transcription source field, so Claude can treat manual subtitles as stronger evidence than auto-captions. The example request 'what text is on screen at 1:30?' maps to high resolution and a narrow time window. CI runs on Ubuntu and macOS with Node.js 24 and 26.
The project is under the MIT license.
Related tools
Related tools listed on this site.
EveryGen AI
Video generator and creative studio describing scene planning, clips, image editing, voices, music and video assembly.
QwenPaw
Personal AI assistant and agent workstation running in your own environment. Its primary description presents local/cloud deployment, multiple channels and extension through Skills and MCP.
Cursor
A coding agent for understanding repositories, planning features, changing code, and reviewing diffs.
CodeBuddy IDE
AI coding workspace for product, design and development. Its dedicated IDE documentation describes technical conversations, code completion, Figma-to-code, diagnostics and a design-to-deployment workflow.
InfCode
Enterprise AI coding agent from TokensInfinity. Its site describes using enterprise code and knowledge through IDE, extension and CLI forms for completion, cross-file changes and engineering tasks.
阿里云百炼
Alibaba Cloud Bailian is a model-service and application-development platform for model inference, knowledge bases, agents and workflows.
千问云
The discovered Qianwen Cloud entry currently presents the Qianwen AI Platform for model discovery, online trials, API access, Skills/CLI tooling and usage/cost management.
Figma AI
Collection of AI workflows and capabilities within Figma products for design exploration, content editing and connecting prototypes with code.
Pixso AI
AI design capability integrated into the Pixso canvas. Its official page describes generating editable web and mobile interfaces from text, sketches or screenshots and continuing to refine existing designs.
Ardot
AI design and collaboration tool. Its official site describes text/image-to-editable UI, vector editing, dynamic layouts and design systems, plus collaboration, annotations and design-to-development handoff.
纳米AI
Nano AI is an AI search and agent assistant whose site says it can generate videos, reports and slides from one sentence and integrates several large models.
Exa AI
Exa is a web-retrieval service for AI applications and agents, describing search, content retrieval, deep research and structured-output APIs.
Sources
This page is written from the official sources below. Open them to check the original text.
- GitHub READMEhttps://github.com/jordanrendric/claude-video-visionFetched
Verified ·
This page is written by AI from the official sources above and checked automatically. This site has not tested it. Found a mistake? Tell us.