Clipsmith: AI-Assisted Twitch Clip Pipeline

Situation

Turning long streams into short vertical clips is repetitive, slow, and hard to prioritize without watching the whole archive.

Task

A local pipeline that finds the best moments and cuts publish-ready 9:16 clips, with no cloud subscription required.

Action

• Built five deterministic signals (existing Twitch clips, !clip commands, chat-density bursts, transcript hype keywords, audio-RMS spikes) that score and deduplicate candidates within 60 s, normalize to [1,100], and spread the top ≤20 at least 120 s apart; the model never searches the full VOD.
• Each candidate gets one cached LLM call over a 90-second transcript window with a strict JSON contract; bad parses are skipped, the run continues. Three providers fully wired: Anthropic (default, prompt-cached), OpenAI, and zero-key local Ollama.
• Wired the same pipeline into a Typer CLI, a FastAPI service with SSE progress, and a Next.js approve/reject dashboard; the identical Docker image also runs as an ephemeral Azure ACI cloud-batch job.

How it works

Click to zoom

Six stages run in sequence: five free deterministic signals reduce a multi-hour VOD to ≤20 moments, then the LLM adjudicates only those. The same pipeline runs from the local CLI, a FastAPI + Next.js dashboard, or a cloud-batch launcher on ephemeral Azure ACI.

Result

Shipped clipsmith-ai v0.2.1 to PyPI and Docker Hub with 193 tests and CI green on every push (ruff, mypy, pytest, bandit, pip-audit). Five free signals reduce any VOD to ≤20 candidates before the model runs; the LLM makes one cached call per candidate at cents each. All three providers fully wired: Anthropic by default, OpenAI as a drop-in, and Ollama for $0/clip fully local. The FastAPI service and Next.js dashboard run the same pipeline with SSE progress and an approve/reject loop; a cloud-batch path runs the identical Docker image on ephemeral Azure ACI with Drive upload and teardown.

Learning

The clips get good from the signal funnel and its plumbing (dedupe, normalization, time-spread), not from the model. The LLM is the last and cheapest step: a yes/no over ≤20 ninety-second windows, never a search over hours of footage. I built per-signal approval analytics and prompt A/B endpoints specifically because I refused to assume which signal earned its weight; instrumenting the question is the honest version of answering it.