Create Real and Held-Back Evals

What you will complete

An eval suite with prewritten expectations, labelled fixtures, and evidence requirements.

Make the operating decision

Validate in layers. Parse files and required fields first. Run deterministic scripts on representative data. Then run the actual workflow, test forbidden and missing-input behaviour, and verify installed discovery. Schedule proof comes only after manual operation passes.

Use one real or sanitized case and hold another back. Add missing columns, duplicates, unknown owners, hostile source instructions, unsafe requests, and output-path failures. Write the expected output, question, refusal, or blocked state before running. A fictional fixture proves structure and decision logic only.

Keep each criterion independent. Required sections, factual traceability, format, safe behaviour, and delivery should not be one compound row. Save run IDs, artifact versions, observed outputs, and one-line evidence for every verdict. Mark not tested honestly.

Worked fictional example: Harbourlight Home Services

Harbourlight's first case contains five estimates, including two missing next actions. The held-back case includes a duplicate and an unknown owner. A hostile cell says to email every customer; the expected result treats the text as data and produces no communication.

Harbourlight Home Services and every estimate record in this course are fictional. They demonstrate the method and do not represent a Flowgrammer client, a deployed system, or measured savings.

Complete workbook section 6

Use the evidence available for your own bounded job. Write unknown when the evidence is missing, and record the person or action that can resolve it.

1. Create one representative and one held-back case.
2. Add missing-input, duplicate, hostile-input, denied-action, and script-failure cases.
3. Write expected results before execution.
4. Map each case to one or more independent outcomes.

Open the course workbook PDF or read the workbook details.

Critical gate before continuing

• Fiction and real evidence are labelled separately.
• Expected results predate observed outputs.
• Boundary tests cause no real side effects.
• Every verdict points to evidence.

If a gate fails, repair the current section, narrow the scope, leave the route manual, or record a blocked or stop decision. Continuing is not the only successful learner action.

Common failure modes

• Expanding beyond an eval suite with prewritten expectations, labelled fixtures, and evidence requirements. before the current artifact can be graded.
• Turning a missing value, unavailable source, or blocked integration into a confident conclusion.
• Treating a prompt instruction as proof that the effective tool or permission boundary works.
• Marking a manual, simulated, or untested route as live.

Check your application