D4 — Prompt engineering and structured output (20 %)

An automated reviewer flags a pattern on Monday and approves the same pattern on Thursday. Nothing changed in the code, and nothing changed in the prompt. What changed is that "be careful about risky migrations" has no fixed meaning, so every run reinvents the threshold.

The remedy the guide names is not a better adjective. It is explicit categorical criteria: named conditions, each one something you could check by hand. The guide's own example is the pair to memorise: "flag comments only when claimed behavior contradicts actual code behavior", against "check that comments are accurate".

# Vague — the version that produces a different verdict every run
Check code comments for accuracy. Be conservative — report only
high-confidence findings.

# Explicit criteria — the version that produces the same verdict twice
Flag a comment ONLY IF one of these holds:
  1. It describes behaviour that contradicts what the code does.
  2. It refers to a function or variable that does not exist.
  3. A TODO or FIXME refers to a bug already fixed in the code.

DO NOT flag:
  - dated phrasing or style that does not contradict the code;
  - imprecise wording that is still consistent with the code;
  - missing comments (a separate category, reported separately).

For each finding, output: file:line, which condition matched, the quoted
comment, and the code fragment that contradicts it.

The two halves are equally load-bearing. Criteria that only say what to report leave everything else to inference; the skip list is what stops a reviewer from drifting into minor style and local convention.

Severity that classifies consistently

The guide's third skill is severity, and it asks for two things: explicit criteria for each level, and a concrete code example per level. The definition alone leaves "medium" to taste; the worked example fixes it.

CRITICAL — fails at runtime for users.
  Example: payment_total = order.get("total") + order["tip"]   # KeyError when no tip
HIGH — security vulnerability.
  Example: db.execute("SELECT * FROM users WHERE id = " + user_id)
MEDIUM — logic bug with no immediate outage.
  Example: for i in range(len(items) - 1):   # last item never processed
LOW — code quality only.
  Example: the same six-line block duplicated in three handlers.

Why one bad category poisons the good ones

The guide's knowledge point on false positives is a claim about people, not about models: a high false positive rate in one category undermines developer confidence in the accurate categories too. Once the bot has cried wolf about naming conventions three times, its genuine SQL-injection finding is skimmed with the same eye.

That is what makes the guide's remedy counter-intuitive and worth remembering: temporarily disable the high false-positive category, keep the precise ones running, improve the prompt for the disabled one, and re-enable it once its criteria are specific. Reviewing less, on purpose, is how the remaining output keeps being read.

Explicit criteria fix what to report, and they leave a second failure untouched: the criteria are written, they are detailed, and the output still comes back in a different shape every run, or comes back empty on the case nobody thought to list.

The guide's answer is unambiguous, and it is the phrase to recognise in an option: few-shot examples are the most effective technique for consistently formatted, actionable output when detailed instructions alone produce inconsistent results. One example is one-shot prompting; several is multi-shot.

Two to four is the guide's own count, and the examples are targeted: they show the ambiguous cases, not the easy ones. Each carries the reasoning for why that action was chosen over a plausible alternative, because the reasoning is what transfers.

<examples>
  <example>
    <request>My order is broken</request>
    <action>get_customer, then lookup_order, then check status</action>
    <rationale>"Broken" may mean damaged goods or a failed delivery. The
    details decide, so gather them before routing anywhere.</rationale>
  </example>
  <example>
    <request>Get me a manager</request>
    <action>escalate_to_human, immediately</action>
    <rationale>The customer asked for a person explicitly. Attempting a
    resolution first overrides a stated preference.</rationale>
  </example>
</examples>

The XML tags are the course's habit and they earn their place here: examples are interpolated content sitting next to instructions, and descriptive delimiters — <sample_input>, <ideal_output>, <examples> — are what keeps the model from reading an example as a request.

The four jobs the guide gives them

The last two are the ones people forget, and each answers a specific failure.

Acceptable versus genuine is the false-positive lever of 4.1 seen from the other side: rather than enumerating every benign pattern, show one benign and one genuine, and let the distinction generalise.

{"location": "src/auth/login.ts:42",
 "issue": "SQL injection in the username parameter",
 "severity": "critical",
 "suggested_fix": "Use a parameterised query"}

Varied document structures is the guide's extraction case. A rate reported as the rate is 42% (Smith, 2023) and the same rate reported as The rate is 42%. [1] with a bibliography at the end are one fact in two shapes, and a model shown only the first returns null on the second. The guide names that symptom directly: examples are what addresses empty or null extraction of required fields on documents whose format varies.

Ask for JSON in prose and you get JSON most of the time: a trailing comma here, a fenced code block there, a field renamed on the third document. The guide's answer is a mechanism rather than a phrasing, and it states it without hedging: tool use with JSON schemas is the most reliable approach for guaranteed schema-compliant structured output, and it eliminates JSON syntax errors.

The move is a small inversion. You declare a tool whose input_schema is the shape you want out, you force the model to call it, and you read the extraction from the call's input. The tool is never executed. Its whole purpose is to be a schema the model must fill.

{
  "tools": [{
    "name": "extract_invoice",
    "description": "Records the fields extracted from an invoice document.",
    "input_schema": {
      "type": "object",
      "properties": {
        "invoice_number": { "type": "string" },
        "issue_date":     { "type": "string", "description": "ISO 8601, YYYY-MM-DD" },
        "total":          { "type": "number" },
        "company_name":   { "type": ["string", "null"] },
        "document_kind":  { "enum": ["invoice", "credit_note", "other", "unclear"] },
        "kind_detail":    { "type": ["string", "null"] }
      },
      "required": ["invoice_number", "total", "document_kind"]
    }
  }],
  "tool_choice": { "type": "tool", "name": "extract_invoice" }
}

The reply carries stop_reason: "tool_use" and a tool_use block whose input is your record. No parsing of prose, no repair pass:

{ "type": "tool_use", "id": "toolu_01A", "name": "extract_invoice",
  "input": { "invoice_number": "INV-2024-0912", "issue_date": "2024-09-12",
             "total": 1249.90, "company_name": null,
             "document_kind": "invoice", "kind_detail": null } }

tool_choice

This is the field D0 and D1 both deferred here. It decides whether calling a tool is optional, mandatory, or mandatory and named.

The two uses the guide names are worth memorising as scenarios, because that is how they are examined. any is the answer when a pipeline receives invoices, purchase orders and delivery notes through the same door: you have a schema for each, you cannot say in advance which applies, and what you cannot afford is a conversational reply instead of a call. A forced named tool is the answer when order matters: extract_metadata must run first, and the enrichment steps come in later turns.

Designing the schema so it does not invite invention

A schema is a contract, and a badly drawn contract makes fabrication the compliant answer.

The guide's last skill on this task statement is easy to skip and cheap to apply: format normalization rules go in the prompt, next to the strict schema. The schema says a field is a string; only the prompt can say which string.

Normalise before you fill the schema:
  dates       → ISO 8601 (YYYY-MM-DD); resolve "yesterday" to an absolute date
  money       → numeric amount plus a separate ISO 4217 code; "five bucks" → 5, USD
  percentages → decimal fraction; "half" → 0.5
  absent      → null, never a placeholder, never a guess

The payload conforms and the values are wrong. Forcing a call through a JSON schema guarantees the shape of the record and nothing about what is in it: total reads 150 while the line items sum to 145; a supplier name sits in the customer field. Nothing in the schema can see it, because nothing in the schema knows arithmetic.

The mechanism is a loop, and the guide tests its four steps:

1. The model produces an extraction.
2. Your code validates it, first with JSON Schema for structure, then with business rules for meaning. Pydantic is the appendix's named tool, and it does both: types and required fields in the model, cross-field rules in a validator.
3. On failure, you send a follow-up request carrying three things: the original document, the failed extraction, and the specific validation error.
4. You track which errors retries actually fix.

Step 3 is the whole of "retry with error feedback", and its adjective is the substance. "There is an error, try again" re-rolls the same dice. The error has to name the field and the gap.

try:
    invoice = Invoice.model_validate(tool_use.input)   # types, then @model_validator
    return invoice
except ValidationError as e:
    messages.append({"role": "assistant", "content": response.content})
    messages.append({"role": "user", "content": [{
        "type": "tool_result",
        "tool_use_id": tool_use.id,
        "is_error": True,
        "content": (
            "total is 150.00 but the line_items sum to 145.00. "
            "Re-read the line items and resubmit; do not adjust total to fit."
        ),
    }]})

The follow-up rides the ordinary conversation: the assistant message holding the tool_use block goes back in the history, and the answer to it is a tool_result block carrying the same tool_use_id, with is_error: true. The document is already in the history, so the model sees all three of the guide's ingredients at once.

When a retry cannot work

This is the judgement the task statement is built around, and it is a question about the source document, never about the number of attempts.

The left column shares one property: the model had the information and shaped it badly. Feeding back the exact error is enough. The right column shares the opposite one, and a retry there can only produce a more confident invention. The answer is a nullable field and a null, which is 4.3's design rule arriving as an operational decision.

Two fields that make the pipeline self-diagnosing

The guide names both, and both are cheap additions to a schema.

stated_total beside calculated_total, plus conflict_detected. Have the model extract the figure the document declares and the sum it computes from the lines, then flag the disagreement instead of silently picking one. The conflict survives into your code, where a human can arbitrate.

{"stated_total": 150.00, "calculated_total": 145.00,
 "conflict_detected": true, "line_items": [ … ]}

detected_pattern on every finding. A code-review pipeline that records which construct triggered each finding turns dismissals into data: when developers reject findings, you can group the rejections by pattern and see that one construct produces most of the noise, which is precisely the category 4.1 tells you to disable and rewrite.

{"location": "src/auth/login.ts:42",
 "issue": "Possible null dereference",
 "severity": "medium",
 "detected_pattern": "no optional chaining on user.profile.email",
 "suggested_fix": "user?.profile?.email"}

The Message Batches API processes large volumes of Messages API requests asynchronously. Three properties decide every question about it:

The third row is the one that decides fit, and it is the one distractors attack. "Most batches finish within the hour" is an observation about queues; the twenty-four hours bound how long processing may run. Neither is a promise about any single submission.

POST /v1/messages/batches
{ "requests": [
    { "custom_id": "invoice-10231",
      "params": { "model": "claude-sonnet-5", "max_tokens": 1024,
                  "messages": [{ "role": "user", "content": "Extract: …" }] } },
    { "custom_id": "invoice-10232",
      "params": { "model": "claude-sonnet-5", "max_tokens": 1024,
                  "messages": [{ "role": "user", "content": "Extract: …" }] } }
] }

// Results come back unordered; pair by custom_id, never by position.
{ "custom_id": "invoice-10232", "result": { "type": "succeeded", "message": { … } } }
{ "custom_id": "invoice-10231", "result": { "type": "errored",  "error": { … } } }
// → resubmit invoice-10231 alone.

custom_id is not decoration. Because results are unordered, it is the only reliable way to say which response belongs to which document, which is also what makes selective resubmission possible.

Which workload goes where

The criterion is a single question: is anyone waiting? Cost savings never promote a blocking workflow into a batch.

Submission cadence under an SLA

The guide asks you to compute one. The arithmetic first: with a 30-hour commitment and processing that may take up to 24 hours, a submission can wait at most 30 − 24 = 6 hours in the queue before it is sent.

Six hours is the ceiling, not the answer. A batch submitted exactly at the six hour mark and processed at the cap consumes the entire budget, leaving nothing for a document that failed and must be resubmitted. The guide's own example answers 4-hour windows for a 30-hour SLA, the ceiling minus a margin.

Note which number entered that calculation. Most batches finish in under an hour, and that observation has no place in the arithmetic: you size on the 24 hours the platform bounds, never on the hour you have been getting.

Handling failures

Four result types come back: succeeded, errored, canceled, expired, and only succeeded is billed. One failed request does not affect the others.

• Identify failures by custom_id and resubmit only those. Re-running the whole batch pays again for everything that worked.
• Resubmit with a modification, not identically, when the cause is in the request: a document that exceeded the context limit goes back as chunks. An invalid_request_error means the body must be corrected before it is sent again; a server error can be retried as it stands.
• Refine the prompt on a sample first. Dry-run one representative request through the synchronous Messages API, get the extraction right, and only then submit the thousand; otherwise you discover a prompt defect after paying for every document in the batch.

Two operational numbers worth carrying: a batch holds up to 100 000 requests or 256 MB, whichever comes first, and results stay retrievable for 29 days. And a short list of parameters a batch refuses, each for the same reason, since they are synchronous-only settings: stream: true (results come back as files), speed, store and previous_thread_event_id, cache_hint and context_hint, and max_tokens: 0.

Ask the session that just wrote the code to review it, and it will find little. Not because it is careless, but because of what it carries: the model retains its reasoning context from generation, which makes it less likely to question its own decisions in the same session. It already considered the alternative and rejected it. Re-reading confirms; it does not test.

The guide's remedy is architectural rather than instructional, and the comparison it draws is the point: an independent review instance, with no prior reasoning context, catches subtle issues better than a self-review instruction or extended thinking. Telling the model to be critical of itself, or giving it more room to deliberate, both operate inside the transcript that produced the bias.

"Independent" has an operational meaning: a separate session, or a subagent, which receives the artifact and the criteria and not the conversation that produced them. In Claude Code that is a reviewer subagent, running in its own context window, restricted to read-only tools, and checked into the repository so the team shares one reviewer. D3's TS 3.6 puts the same instance in a pipeline; what belongs here is why the instance has to be a different one.

Splitting a review that is too large for one pass

A pull request touching thirty files reviewed in a single pass produces inconsistent output: a pattern flagged in one file and approved in another, findings that thin out towards the end. The cause the guide names is attention dilution, and the remedy is to shrink each pass rather than to enlarge the context.

# Pass 1 — local, one invocation per file.
for file in $(git diff --name-only main...HEAD); do
  review --scope="$file" \
         --prompt="Local issues only: bugs, security, quality in THIS file."
done

# Pass 2 — integration, one invocation over the changed set.
review --scope="$(git diff --name-only main...HEAD)" \
       --prompt="Cross-file only: type mismatches across module boundaries,
                 circular dependencies, broken data flow. Ignore local issues."

Splitting a task into focused passes is D1's task decomposition (TS 1.6) applied to review; what is specific here is the cut, local against cross-file, because those are the two things one undifferentiated pass does worst.

Confidence, self-reported

The third skill is a verification pass in which the model reports a confidence alongside each finding. The value is not the number itself but the routing it enables: high-confidence findings go straight to the developer, low-confidence ones to a human check, and the threshold moves as you observe which band is worth reading.

{"file": "src/billing/refund.ts", "line": 88,
 "issue": "Refund may exceed the original charge when a partial refund precedes it",
 "confidence": 0.42,
 "detected_pattern": "no accumulated-refund check before subtraction"}

Two cautions keep this honest. A confidence reported from inside the generating context inherits that context's blind spot, so the verification pass is worth running in the independent instance. And a self-reported number is a signal for routing, not a measurement: what turns it into one is comparing it against findings whose verdict you already know.