Deduplicate Entities, Events, and Repeat Runs

What you will complete

A memory and reconciliation design that suppresses unchanged alerts without discarding legitimate events.

Make the operating decision

Choose source-specific duplicate keys such as source record ID, canonical URL plus event fields, or another stable identifier. Keep duplicates in raw data and add duplicate-of pointers in normalized data. Reconciliation should explain why returned and primary counts differ.

Separate the organization from the event. A new location and a later service request are different events for one entity. The same opening syndicated by two sources can be grouped when identity and event match are supported. Similar titles alone are not enough to merge records.

Persist event fingerprints, first seen, last seen, last alerted version, and review status. An old event discovered today remains old. Reprocessing identical evidence creates no new alert. A defined material change creates one new event-version alert while preserving dismissed, reviewed, or contacted state.

Worked fictional example: Harborline Safety Training

Northstar's fictional announcement appears twice and becomes one event with two evidence records. A later distinct support request remains another event. Dockside's undated page stays research needed on repeat scans and does not become fresh.

Harborline Safety Training, Northstar Distribution, Dockside Logistics, Westbank Storage, all events, excerpts, people, and URLs are fictional fixtures. They do not prove a live source, lead, contact route, or business outcome.

Complete workbook section 5

Use the evidence available for your own bounded job. Write unknown when the evidence is missing, and record the person or action that can resolve it.

1. Define entity and event keys.
2. Write duplicate and syndicated-event rules.
3. Store first-seen, event date, fingerprint, and review state.
4. Test identical, changed, and distinct-event repeat runs.

Open the course workbook PDF or read the workbook details.

Critical gate before continuing

• Raw duplicates remain available.
• Counts reconcile.
• First-seen cannot refresh event age.
• Review state survives repeat runs.

If a gate fails, repair the current section, narrow the scope, leave the route manual, or record a blocked or stop decision. Continuing is not the only successful learner action.

Common failure modes

• Expanding beyond a memory and reconciliation design that suppresses unchanged alerts without discarding legitimate events. before the current artifact can be graded.
• Turning a missing value, unavailable source, or blocked integration into a confident conclusion.
• Treating a prompt instruction as proof that the effective tool or permission boundary works.
• Marking a manual, simulated, or untested route as live.

Check your application