TrueGrowth/openspec/changes/replace-funclip-with-videoagent-clip-workbench/design.md

Context

VideoAgent is a broad Python agent framework for video understanding, editing, audio processing, and creative remaking. It is not a stable desktop SDK, so TrueGrowth must wrap it behind a local JSON task adapter and keep heavy dependencies isolated from Electron and the web renderer.

Goals / Non-Goals

• Goals: make VideoAgent the clip workbench primary runtime, provide a customer-facing intent-first editing flow, and write structured deliverables back to tasks/assets.
• Goals: support a useful MVP even when the full VideoAgent model stack is not installed.
• Non-Goals: bundle full Python, CUDA, PyTorch, ONNX, or model packages into the default desktop/web process.
• Non-Goals: expose VideoAgent, FunClip, CUDA, model paths, or command-line details in customer-facing UI.

Decisions

• Decision: run VideoAgent as an independent videoagent-local sidecar managed by the existing local API process.
• Decision: use a stable TrueGrowth JSON task shape instead of directly driving VideoAgent's interactive CLI.
• Decision: treat capability groups as installable/readiness-gated packages: basic clipping, video understanding, creative remaking, and voice/audio processing.
• Decision: the first implementation must provide a deterministic MVP plan/result fallback so the clip center can complete a real task loop before the full third-party runtime is installed.
• Decision: source inputs are workflow-aware. Understanding, transcription, text editing, and most long-to-short jobs use one primary video/audio source; B-roll, platform packaging, and remaking jobs use a primary source plus references; montage, batch processing, and script-to-video jobs may use multiple videos, images, and audio assets.
• Decision: structured plans, transcripts, SRT, timeline, titles, cover prompts, and publish copy are task deliverables first. They are written to task metadata and only become asset-center media records when a real playable media file or publishable image exists.
• Decision: first-screen quick scenarios show only workflows that can close with the current adapter and model-center capabilities: transcription/subtitles, long-to-short planning, transcript editing, video summary/Q&A, course/interview/ecommerce/podcast planning, B-roll planning, multi-source montage planning, and script-to-video planning.
• Decision: VideoAgent demo features that require heavy optional stacks, such as ImageBind semantic retrieval, Seed-VC voice conversion, DiffSinger singing/music-video, or specialized voice-remake models, remain hidden from first-screen quick scenarios until the model center reports the required pack as initialized.
• Decision: model requirements are mapped to TrueGrowth capability packs instead of raw third-party names. Whisper/FunASR map to 字幕转写, CosyVoice/FishSpeech/IndexTTS-style models map to 配音音色, LatentSync/MuseTalk/Wav2Lip-style models map to 数字人口播, ImageBind maps to 素材语义检索, Seed-VC maps to 声音重制, and DiffSinger maps to 音乐视频.
• Decision: customer first use bundles only the lightweight adapter, media handling, and checks that are safe for the desktop package. Larger Mac/Windows model packs are initialized through the existing model center so storage, platform, license, and GPU constraints can be shown in one place.

API Contract

• GET /local-api/videoagent returns readiness, visible capability groups, service status, and setup progress.
• POST /local-api/runtime/videoagent/start starts or prepares the sidecar and returns an async startup state.
• POST /local-api/runtime/videoagent/stop stops the sidecar if it is running.
• POST /local-api/videoagent/tasks accepts source, goal, scene, platform, output options, model routing, and package flags.
• POST /local-api/videoagent/tasks also accepts sources, primarySourceId, sourceMode, acceptedSourceTypes, and supportingSourceIds for workflows that need multiple materials.
• GET /local-api/videoagent/tasks/:id returns task status, progress, generated plan, transcript, segments, deliverables, and asset ids.

Risks / Trade-offs

• VideoAgent dependencies are heavy and platform-sensitive. Mitigation: isolate runtime, detect capability groups independently, and keep a deterministic MVP fallback.
• VideoAgent is research-oriented. Mitigation: wrap it behind TrueGrowth task adapters and avoid exposing internal graph prompts or CLI details.
• Existing FunClip tasks may exist in history. Mitigation: keep history display able to read legacy funclip-local records while new submissions use videoagent-local.

Migration Plan

1. Add VideoAgent local runtime status and task endpoints.
2. Switch the clip workbench submission path to videoagent-local.
3. Replace capability-card-first UI with source/goal/plan/result workflow.
4. Preserve existing assets, task history, and publish handoff behavior.
5. Remove visible FunClip copy and provider jargon from the page.