Case study · HackerRank Orchestrate · Bronze medal · Claude agents

A Claude agent built in 24 hours

Working alone, I built a Claude agent that audited 518 personal-data deletion requests and won a bronze medal: rank 363 of 4,090.

  • Bronzemedal, Oct 2026
  • 363 / 4,090final rank, top 9%
  • 24 hindividual challenge
  • 518requests audited
  • 41 minfull run, every answer passed its automated checks first try
  • ≈ $0.12per request (est. from tokens)

How the agent works: Claude chooses, code calculates

  • Claude (judgment)
  • Python (calculation)
  • Guardrail
  1. Request + evidence

    One deletion request; evidence in database exports, logs, JSON, screenshots and PDFs.

    518 requests
  2. Audit-time filter (Guardrail)

    Each case sees only evidence from before its audit time.

    72 cutoffs checked · 0 answers changed
  3. Claude judges (Claude (judgment))

    Is it a deletion request? Whose records are these, across changed email addresses? What does the evidence show? A vision step reads images and PDFs.

    7 read-only tools
  4. Code calculates (Python (calculation))

    Record states and the steps, in a fixed order: stop data flows, block restores, then delete or redact.

    steps = records + 2 × violations
  5. Validator (Guardrail)

    Checks every answer before it is accepted.

    every answer checked

If an answer fails its checks

  1. Fails validator
  2. Retry once
  3. Still fails
  4. Rules engine answer (also the baseline)
  • Prompt caching served 93.6% of input tokens
  • My own tool loop, every call testable
  • Claude can investigate but change nothing
Why a strict audit-time cutoff, even at a cost?

Later evidence scores better but isn’t knowable at audit time. The filter lowered my rules engine’s sample row accuracy from 0.750 to 0.625. I kept it: an audit that uses later evidence isn’t an audit.

Why read-only tools?

The agent sees personal data. Read-only tools keep the security-review surface small; it recommends, and humans make the changes.

Why does code, not Claude, write the steps?

No invented steps, and the model’s output stays small and checkable.

Why keep a rules engine?

It was built first, as fallback and baseline, so every request gets an answer. On the full run, agent and rules engine differed on only 2 of 518 rows.

Diagram as text: a request and its evidence are filtered to the audit time, judged by Claude with seven read-only tools, turned into record states and ordered steps by Python, then validated. A failed answer is retried once, then replaced by the rules engine’s answer.

Analysis before prompting

How I analysed it, in seven steps

  1. Study the answer key first

    steps = records + 2 × violations

    Changed: the hard part is which records exist and their state, so code writes the steps and I scored record by record.

  2. What a "SUCCESS" line is worth

    Changed: trusting the workflow’s status got 13 more of 32 samples wrong, so state comes from independent evidence.

  3. Does deleted data come back?

    10 reappearances, each inspected; at least one was my own parser bug

    Changed: none followed a logged sync or restore, so the agent looks for the mechanism: a flow still on, a restorable backup.

  4. Where copies go

    Changed: upstream causes are fixed first: stop the flows and block restores, then delete or redact.

  5. Practice cases ≠ real cases

    share from an old email: 2× · about 10 non-deletion requests only in the real set

    Changed: an identity-linking tool, my own test cases, and sample scores treated as optimistic.

  6. What 32 examples can prove

    Changed: small wins became leads, not proof; McNemar’s test showed the 7- and 13-case differences were real.

  7. Future evidence is a leak

    solid = used · dashed = excluded (illustrative)

    Changed: each case sees only evidence from before its audit time; a leak check across 72 cutoffs changed 0 answers.

Rules-engine row accuracy on the 32 samples

All evidence
0.750
Audit-time cutoff (kept)
0.625
I kept it: an audit that uses later evidence isn’t an audit.

Claude chooses, code calculates.

The model judges meaning and evidence; Python decides record state and writes the fix steps, so the model never invents what can be computed.
On this page

Problem

For each request: is it really a deletion request, was the deletion complete at audit time, and if not, which findings, fixes and checks are needed, in what order? Evidence was spread across database exports, logs, JSON, screenshots and PDFs, and the workflow’s own “complete” flag couldn’t be trusted. So I measured how far the evidence could be trusted before writing a prompt.

The 24-hour individual challenge judges the code, output accuracy, the working transcript and a 30-minute interview with an AI judge.

Result / Impact

  • Full run, 518 requests: 41 minutes, every answer passing its automated checks on the first try, at about $0.12 per request (estimated from token counts).
  • 32 solved samples (my development set, so optimistic): status accuracy 1.000, remediation-step F1 0.955 (precision 0.984, recall 0.927).
  • A correctness rule with a price: the audit-time filter lowered my rules engine’s sample row accuracy from 0.750 to 0.625. I kept it; an audit that uses later evidence isn’t an audit.
  • After the challenge, I built a small tool to track the published final standings.

Need a Claude agent built like this? See the services I offer

What I built

Seven build choices, from a rules engine first to Claude Code
  1. A rules engine first, as fallback and baseline.
  2. My own tool loop with seven read-only tools, so every call, retry and fallback is testable. Claude can investigate (the request, the person’s records across changed email addresses, the proof for each) but change nothing.
  3. Code for arithmetic and the fixed step order: stop data flows, block restores, then delete or redact.
  4. An audit-time cutoff in code. A leak check across 72 cutoffs changed 0 answers.
  5. A validator on every answer: one retry, then the rules engine.
  6. Prompt caching served 93.6% of input tokens; cached responses made reruns free.
  7. Built with Claude Code: one session, ten phases, starter code to a full run in under three hours; ten failures found and fixed, each with a test or check.

Tools: Claude (Opus 5.5), tool use, prompt caching, a vision step for images and PDFs, Python, Claude Code.

How I analysed the problem

The full reasoning behind the seven steps
  1. The answer key. Every relevant record gets a verification; each record not in its required state also gets a finding and a fix (steps = records + 2 × violations). So: the hard part is which records exist and their state. Code writes the steps; I scored record by record.
  2. What “SUCCESS” is worth. A SUCCESS log line raised the odds a record was gone by a likelihood ratio of only about 3. A FAILED redact or anonymize never turned out to have worked; “disable flow” and “block restore” always said SUCCESS, so carried no information. Trusting the workflow’s status got 13 more of 32 samples wrong. So: state comes from independent evidence.
  3. Does deleted data come back? A Kaplan–Meier estimate, cut off below 10 records at risk; I inspected all 10 reappearances, and at least one was my own parser bug. None followed a logged sync or restore. So: the agent looks for the mechanism (a flow still on, a restorable backup).
  4. Where copies go. The logs form a source-to-target graph. So: it shows where leftovers land (search, analytics, cache), and why upstream causes are fixed first.
  5. Practice versus real. The real set had twice the share of old-email requests and about 10 non-deletion requests the samples lacked. So: an identity-linking tool, my own test cases, and sample scores treated as optimistic.
  6. What 32 examples can prove. Wilson intervals, McNemar’s test: winning 2 of 2 disagreements is p = 0.5; the 7-case and 13-case differences were real. So: I treated small differences as leads to investigate, not proof, and kept a change only when I could explain why it was right.
  7. Future evidence is a leak. So: each case sees only evidence from before its audit time.

Most steps are exact rules; the model only reads requests in several languages, weighs evidence and reads images.

Decisions

Four decisions with context, options and consequences

Claude chooses, code calculates. Context: answers mix judgment and bookkeeping. Options: model does everything; rules only; split. Decision: split. Consequences: no invented steps; small, checkable model output.

Audit-time cutoff, even at a cost. Context: later evidence scores better but isn’t knowable at audit time. Options: all evidence; filter. Decision: filter, proved by the leak check. Consequences: sample accuracy 0.750 → 0.625; answers an auditor can accept.

Rules engine as fallback. Context: model output can fail validation. Options: model only; rules only; both. Decision: one retry, then rules. Consequences: every request answered. On the full run, agent and rules engine differed on only 2 of 518 rows.

Read-only tools. Context: the agent sees personal data. Options: act; investigate and recommend. Decision: recommend. Consequences: a smaller security-review surface; humans make the changes.

How I direct and review AI agents

Rules first, review before merge, checks over trust
  • Rules first: each agent gets a role, allowed files and a definition of done. This portfolio is built by a team of Claude agents, including a fact-checker that never reviews its own work.
  • Review before merge: lint, strict types, unit and end-to-end tests on every pull request.
  • Checks over trust: validators, leak checks and baselines catch confident wrong answers.
  • Human in the loop: in a KMC admin-console design, Claude drafts email wording, a personal note field is never sent to Claude, and staff send themselves a test first.

The KMC DC Events app, live since 16 November 2026, was built with Claude Code the same way.

Proof

Next case study

  • Volunteer full-stack engineer · 2026 · live product

    48 swebsite edit to app (one test run, development)

    KMC DC Events app

    Installable and offline-first, with isolated environments. Live since 16 November 2026 and owned by the center.

Hiring for a full-time role? Hire me