Why a spreadsheet, on purpose
Excel is the rare tool that a human operator, a data engineer and a pipeline script can all edit, review and version without ceremony. On a team using OneDrive it is also the file people actually open — which matters more than elegance. A governance record nobody reads is not a governance record.
The workbook has three sheets: Run Book, Instructions & Use, and Audit Log. The pipeline reads and writes it through openpyxl, so the same file is simultaneously the operator's interface, the scheduler's state and the auditor's evidence. There is no second system to reconcile it against, which removes an entire class of drift.
One row per recording, thirty columns
The columns fall into four groups. This is the canonical schema the pipeline reads and writes — the contract shared by the workbook generator, the orchestrator and the database mirror.
| Group | Columns | Purpose |
|---|---|---|
| Identity | # · Video URL · Video ID · Title · Channel | What the source is. The URL is the only column a human must supply for a new recording. |
| Source metadata | Published (UTC) · Duration (s) · Duration (mm:ss) · Views | Filled automatically on first run. |
| Workflow | Date Added · Added By · five ☐ checkboxes · Status | The checkbox columns drive the pipeline; Status summarises — Not started / In progress / Completed / Failed. |
| Audit & artefacts | Date Processed · Processed By · Transcript File · Transcript SHA-256 · Chunk Count · FAISS Index Path · Script File · Audio File · Legal Basis · Attribution Notice · Notes / Errors · Last Modified · Modified By | The verifiable trail: what ran, when, by whom, with which artefacts, under which legal basis. |
Two of those audit columns do more work than the rest.
- Transcript SHA-256 — a content hash of the raw transcript. Every downstream artefact can be shown to descend from one unaltered source. Change the source and the hash changes; the run book shows it. This is the audit anchor the rest of the trail hangs from.
- Legal Basis / Attribution Notice — every row records the basis on which the source is used and the attribution the output must carry. Compliance travels with the data rather than living in someone's head.
Compliance travels with the data, not in someone's head.
The idempotency contract
The orchestrator implements one rule, and the rule is what makes the pipeline safe to run repeatedly on a schedule:
A row with Status = Completed and all five checkboxes ticked is skipped. A row is processed only while any checkbox is still empty. To force a re-run, deliberately clear a checkbox.
That is what makes the ongoing workflow work. Once the first batch is done, an operator adds a URL in a new row and the next run picks it up — fetching metadata, filling the identity and source columns, and processing stage by stage. Re-runs never touch completed rows, and a re-run is something you ask for explicitly by clearing a box.
Fig. 04 — five boxes, and only one of them is a person
Note the asymmetry in the five checkboxes: four are ticked by the pipeline as artefacts appear, but the fifth — Published — is ticked by a person after the episode is live. A row can be Completed and still awaiting publication. The machine reports what it produced; only a human asserts that it went out.
That is governance by construction, not by discipline. Discipline degrades under deadline; construction does not.
Based on House of Lords proceedings (UK Parliament). Not affiliated with, or endorsed by, the UK Parliament.