Problem
For each request: is it really a deletion request, was the deletion complete at audit time, and if not, which findings, fixes and checks are needed, in what order? Evidence was spread across database exports, logs, JSON, screenshots and PDFs, and the workflow’s own “complete” flag couldn’t be trusted. So I measured how far the evidence could be trusted before writing a prompt.
The 24-hour individual challenge judges the code, output accuracy, the working transcript and a 30-minute interview with an AI judge.
Result / Impact
- Full run, 518 requests: 41 minutes, every answer passing its automated checks on the first try, at about $0.12 per request (estimated from token counts).
- 32 solved samples (my development set, so optimistic): status accuracy 1.000, remediation-step F1 0.955 (precision 0.984, recall 0.927).
- A correctness rule with a price: the audit-time filter lowered my rules engine’s sample row accuracy from 0.750 to 0.625. I kept it; an audit that uses later evidence isn’t an audit.
- After the challenge, I built a small tool to track the published final standings.
Need a Claude agent built like this? See the services I offer
What I built
- A rules engine first, as fallback and baseline.
- My own tool loop with seven read-only tools, so every call, retry and fallback is testable. Claude can investigate (the request, the person’s records across changed email addresses, the proof for each) but change nothing.
- Code for arithmetic and the fixed step order: stop data flows, block restores, then delete or redact.
- An audit-time cutoff in code. A leak check across 72 cutoffs changed 0 answers.
- A validator on every answer: one retry, then the rules engine.
- Prompt caching served 93.6% of input tokens; cached responses made reruns free.
- Built with Claude Code: one session, ten phases, starter code to a full run in under three hours; ten failures found and fixed, each with a test or check.
Tools: Claude (Opus 5.5), tool use, prompt caching, a vision step for images and PDFs, Python, Claude Code.
How I analysed the problem
- The answer key. Every relevant record gets a verification; each record not in its required state also gets a finding and a fix (steps = records + 2 × violations). So: the hard part is which records exist and their state. Code writes the steps; I scored record by record.
- What “SUCCESS” is worth. A SUCCESS log line raised the odds a record was gone by a likelihood ratio of only about 3. A FAILED redact or anonymize never turned out to have worked; “disable flow” and “block restore” always said SUCCESS, so carried no information. Trusting the workflow’s status got 13 more of 32 samples wrong. So: state comes from independent evidence.
- Does deleted data come back? A Kaplan–Meier estimate, cut off below 10 records at risk; I inspected all 10 reappearances, and at least one was my own parser bug. None followed a logged sync or restore. So: the agent looks for the mechanism (a flow still on, a restorable backup).
- Where copies go. The logs form a source-to-target graph. So: it shows where leftovers land (search, analytics, cache), and why upstream causes are fixed first.
- Practice versus real. The real set had twice the share of old-email requests and about 10 non-deletion requests the samples lacked. So: an identity-linking tool, my own test cases, and sample scores treated as optimistic.
- What 32 examples can prove. Wilson intervals, McNemar’s test: winning 2 of 2 disagreements is p = 0.5; the 7-case and 13-case differences were real. So: I treated small differences as leads to investigate, not proof, and kept a change only when I could explain why it was right.
- Future evidence is a leak. So: each case sees only evidence from before its audit time.
Most steps are exact rules; the model only reads requests in several languages, weighs evidence and reads images.
Decisions
Claude chooses, code calculates. Context: answers mix judgment and bookkeeping. Options: model does everything; rules only; split. Decision: split. Consequences: no invented steps; small, checkable model output.
Audit-time cutoff, even at a cost. Context: later evidence scores better but isn’t knowable at audit time. Options: all evidence; filter. Decision: filter, proved by the leak check. Consequences: sample accuracy 0.750 → 0.625; answers an auditor can accept.
Rules engine as fallback. Context: model output can fail validation. Options: model only; rules only; both. Decision: one retry, then rules. Consequences: every request answered. On the full run, agent and rules engine differed on only 2 of 518 rows.
Read-only tools. Context: the agent sees personal data. Options: act; investigate and recommend. Decision: recommend. Consequences: a smaller security-review surface; humans make the changes.
How I direct and review AI agents
- Rules first: each agent gets a role, allowed files and a definition of done. This portfolio is built by a team of Claude agents, including a fact-checker that never reviews its own work.
- Review before merge: lint, strict types, unit and end-to-end tests on every pull request.
- Checks over trust: validators, leak checks and baselines catch confident wrong answers.
- Human in the loop: in a KMC admin-console design, Claude drafts email wording, a personal note field is never sent to Claude, and staff send themselves a test first.
The KMC DC Events app, live since 16 November 2026, was built with Claude Code the same way.
Proof
- Challenge format: HackerRank blog (external site).