41 lines
4.9 KiB
Markdown
41 lines
4.9 KiB
Markdown
## Context
|
|
VideoAgent is a broad Python agent framework for video understanding, editing, audio processing, and creative remaking. It is not a stable desktop SDK, so TrueGrowth must wrap it behind a local JSON task adapter and keep heavy dependencies isolated from Electron and the web renderer.
|
|
|
|
## Goals / Non-Goals
|
|
- Goals: make VideoAgent the clip workbench primary runtime, provide a customer-facing intent-first editing flow, and write structured deliverables back to tasks/assets.
|
|
- Goals: support a useful MVP even when the full VideoAgent model stack is not installed.
|
|
- Non-Goals: bundle full Python, CUDA, PyTorch, ONNX, or model packages into the default desktop/web process.
|
|
- Non-Goals: expose VideoAgent, FunClip, CUDA, model paths, or command-line details in customer-facing UI.
|
|
|
|
## Decisions
|
|
- Decision: run VideoAgent as an independent `videoagent-local` sidecar managed by the existing local API process.
|
|
- Decision: use a stable TrueGrowth JSON task shape instead of directly driving VideoAgent's interactive CLI.
|
|
- Decision: treat capability groups as installable/readiness-gated packages: basic clipping, video understanding, creative remaking, and voice/audio processing.
|
|
- Decision: the first implementation must provide a deterministic MVP plan/result fallback so the clip center can complete a real task loop before the full third-party runtime is installed.
|
|
- Decision: source inputs are workflow-aware. Understanding, transcription, text editing, and most long-to-short jobs use one primary video/audio source; B-roll, platform packaging, and remaking jobs use a primary source plus references; montage, batch processing, and script-to-video jobs may use multiple videos, images, and audio assets.
|
|
- Decision: structured plans, transcripts, SRT, timeline, titles, cover prompts, and publish copy are task deliverables first. They are written to task metadata and only become asset-center media records when a real playable media file or publishable image exists.
|
|
- Decision: first-screen quick scenarios show only workflows that can close with the current adapter and model-center capabilities: transcription/subtitles, long-to-short planning, transcript editing, video summary/Q&A, course/interview/ecommerce/podcast planning, B-roll planning, multi-source montage planning, and script-to-video planning.
|
|
- Decision: VideoAgent demo features that require heavy optional stacks, such as ImageBind semantic retrieval, Seed-VC voice conversion, DiffSinger singing/music-video, or specialized voice-remake models, remain hidden from first-screen quick scenarios until the model center reports the required pack as initialized.
|
|
- Decision: model requirements are mapped to TrueGrowth capability packs instead of raw third-party names. Whisper/FunASR map to `字幕转写`, CosyVoice/FishSpeech/IndexTTS-style models map to `配音音色`, LatentSync/MuseTalk/Wav2Lip-style models map to `数字人口播`, ImageBind maps to `素材语义检索`, Seed-VC maps to `声音重制`, and DiffSinger maps to `音乐视频`.
|
|
- Decision: customer first use bundles only the lightweight adapter, media handling, and checks that are safe for the desktop package. Larger Mac/Windows model packs are initialized through the existing model center so storage, platform, license, and GPU constraints can be shown in one place.
|
|
|
|
## API Contract
|
|
- `GET /local-api/videoagent` returns readiness, visible capability groups, service status, and setup progress.
|
|
- `POST /local-api/runtime/videoagent/start` starts or prepares the sidecar and returns an async startup state.
|
|
- `POST /local-api/runtime/videoagent/stop` stops the sidecar if it is running.
|
|
- `POST /local-api/videoagent/tasks` accepts source, goal, scene, platform, output options, model routing, and package flags.
|
|
- `POST /local-api/videoagent/tasks` also accepts `sources`, `primarySourceId`, `sourceMode`, `acceptedSourceTypes`, and `supportingSourceIds` for workflows that need multiple materials.
|
|
- `GET /local-api/videoagent/tasks/:id` returns task status, progress, generated plan, transcript, segments, deliverables, and asset ids.
|
|
|
|
## Risks / Trade-offs
|
|
- VideoAgent dependencies are heavy and platform-sensitive. Mitigation: isolate runtime, detect capability groups independently, and keep a deterministic MVP fallback.
|
|
- VideoAgent is research-oriented. Mitigation: wrap it behind TrueGrowth task adapters and avoid exposing internal graph prompts or CLI details.
|
|
- Existing FunClip tasks may exist in history. Mitigation: keep history display able to read legacy `funclip-local` records while new submissions use `videoagent-local`.
|
|
|
|
## Migration Plan
|
|
1. Add VideoAgent local runtime status and task endpoints.
|
|
2. Switch the clip workbench submission path to `videoagent-local`.
|
|
3. Replace capability-card-first UI with source/goal/plan/result workflow.
|
|
4. Preserve existing assets, task history, and publish handoff behavior.
|
|
5. Remove visible FunClip copy and provider jargon from the page.
|