- The problem with "the model said so"
- What gets recorded
- Prompts are versioned, never edited
- Why cost is computed, not reported
#The problem with "the model said so"
An AI feature that cannot be replayed is not a research artifact — it is an anecdote. Six months after a paper is submitted, "we used GPT-4o-mini" is not enough to reproduce a number in a table.
#What gets recorded
Every LLM and embedding call writes a row capturing the provider, model, prompt version, token counts, latency, and computed cost:
[$response, $run] = $logger->track(
feature: AiFeature::Tldr,
provider: $resolved['provider'],
model: $resolved['model'],
callback: fn () => $agent->prompt($prompt, ...),
subject: $post,
);
#Prompts are versioned, never edited
Templates are append-only. Changing a prompt inserts a new version and flips the
active flag; the old text stays readable forever. A run recorded against
prompt_version = 3 can always be reconstructed exactly.
#Why cost is computed, not reported
Providers return token counts, not money. Pricing lives in config and is applied at write time, so the cost dashboard reflects what the operator was actually billed under their own rate card.