Preserve Evidence and Normalize Unknowns

What you will complete

Complete the Evidence and Normalization Contract, including stable IDs, raw pointers, date fields, dedupe key, and unknown vocabulary.

Decision rule: Missing evidence stays missing. It never becomes false, zero, unpopular, irrelevant, or low value.

Save before interpretation

Write untouched provider or feed output before parsing, deduplication, scoring, or summarization. A crash after collection should leave raw evidence available. Each normalized record points back to its raw index, source URL or identifier, collection timestamp, and run identifier.

Give facts durable identities

Use stable run, record, and evidence identifiers. Attach evidence to field paths so a reviewer can see which source supports a title, date, quotation, or product change. Keep the original date string when normalizing to ISO format. Do not invent a publication date from the collection time.

Deduplicate without destroying the audit trail

Choose a stable key such as canonical URL or source identifier. Keep duplicates in raw data and write a duplicate-of pointer. Similar titles alone are not enough to merge events. Report requested, returned, normalized, duplicate, excluded, unknown, and error counts.

Worked example: Northstar Workflow Studio

Fictional teaching fixture: Northstar's fixture returns three records. Two use the same canonical release URL and become one primary development with a duplicate pointer. The third lacks a publication date, so publishedAt is Unknown while collectedAt remains available. No field is filled from a plausible guess. The fixture is fictional and cannot be cited as live market evidence.

Northstar is not a Flowgrammer client, and its sources, weights, records, scores, timing, and results are not live evidence. The example exists so you can inspect how one choice changes the same capstone from lesson to lesson.

Workbook exercise

Open section 4 of the Morning Intelligence Agent Launch Pack. Complete the following work for your own approved topic:

1. Define run, record, and evidence identifier patterns.
2. List required raw and normalized fields.
3. Choose and justify a stable dedupe key.
4. Define Unknown, Not returned, Not applicable, and Failed to collect for your workflow.

Write assumptions as assumptions. Keep any untested field marked untested. Ask another person to challenge one rule before you continue.

Common failure modes

• Overwriting raw evidence with cleaned text.
• Using collection time as publication time.
• Merging different events because headlines look similar.
• Turning a missing field into a negative classification.

Repair the smallest responsible layer: scope, source policy, collection control, evidence model, ranking rule, report, or operating gate. Do not hide a failed condition inside a more confident summary.

Capstone connection

Complete the Evidence and Normalization Contract, including stable IDs, raw pointers, date fields, dedupe key, and unknown vocabulary. This section must be specific enough for another operator to inspect, test, and reject. A complete worksheet is not automatically a passing worksheet; the evidence and critical gates still control the decision.

Check your application