InsightsPipeline · Data engineering

Ten steps from recording to episode

The whole flow is one command. Inside it is a deliberate, auditable ten-step journey in which every step maps to a checkbox in the run book and produces an artefact you can point at.

  • August 2026
  • 8 min read
  • Engineering, ops

Four stages, ten steps

Governance comes first and publication last, with the model doing its work in a single step near the end. That ordering is the point: by the time anything is generated, the system already knows what the source was, that it is unaltered, and on what basis it may be used.

  1. 1–2 Governance Add the URL — the only external input. Validate and identity-check: extract the video ID, guard against duplicates, and record the legal basis and attribution notice before any processing begins.
  2. 3–5 Ingestion Fetch metadata — title, channel, upload date, duration, views. Fetch the transcript through the caption fallback chain. Hash the source with SHA-256 and write it to the run book: the audit anchor for everything downstream.
  3. 6–7 Retrieval Chunk the transcript, then embed into a FAISS index held per recording. Chunk count and index path are recorded, so the retrieval state of any episode is inspectable after the fact.
  4. 8 Generation Retrieve the top-ranked chunks and generate a script with the local model, under a prompt engineered to report rather than republish. The model and its settings are recorded with the artefact.
  5. 9–10 Publication A person reads the script against the transcript before anything is rendered. Only then is audio synthesised, show notes written, and the Published checkbox ticked by hand.

Compliance precedes work. The legal basis is recorded in step 2, not attached to the output in step 10.

What each stage leaves behind

StageArtefactRecorded in the run book as
IngestionTranscript JSON + SHA-256Transcript Fetched · Transcript File · Transcript SHA-256
EmbeddingFAISS index directoryEmbedded · Chunk Count · FAISS Index Path
ScriptMarkdown scriptScript Generated · Script File
AudioMP3 + show notesAudio Generated · Audio File
PublicationThe live episodePublished — ticked by a person, never by the pipeline

The symmetry is deliberate. Each stage writes exactly one artefact and exactly one set of run-book columns, which is why a half-finished run is diagnosable: the last ticked checkbox tells you where it stopped, and the artefact directory tells you what it managed to produce before it did.

Fig. 08 — a run that stopped, read two ways

STEP 2 — LEGAL BASIS AND ATTRIBUTION, RECORDED BEFORE ANY OF THIS RUNS ARTEFACT RUN BOOK INGESTION transcript + sha-256 Transcript Fetched EMBEDDING faiss index dir Embedded (FAISS) SCRIPT markdown script not written Script Generated empty AUDIO mp3 + show notes not written Audio Generated empty PUBLICATION the live episode not written Published empty THE RUN STOPPED HERE WHAT AN OPERATOR READS OFF THIS Last tick is Embedded, so it stopped in generation. Two artefacts on disk, not three, which agrees. Nothing was half-written, and no one has to open a log to find that out.

Based on House of Lords proceedings (UK Parliament). Not affiliated with, or endorsed by, the UK Parliament.

Apply this to your problem.

Sponsors bring a real problem class to a challenge day and get a cohort working on it, with the research behind them.

Or email info@inno-forum.co.uk