Challenge
State the contract, constraints, and success conditions.
Field guide · Harness engineering
The workflow
that learns.
Red Planet Labs did not search for one perfect prompt. They built an instrumented loop in which every agent run produces evidence, every failure can improve the skill or workflow, and every improvement must survive a harder test.
measuredfailurebecomes a design input
challengeruntraceskill Δregress8explicit phases
3private test dimensions
5telemetry layers
1repeatable learning loop
01 / The object being engineered
The model is only one component
A capable model can still fail inside a weak environment. The task may be underspecified. The relevant knowledge may arrive too early and disappear from attention. The evaluator may reward superficial completion. A single context may let the agent defend its first idea long after evidence has turned against it.
model+target+skill+workflow+tools+evaluator= observable capability
Rama matters here because it reduces the number of backend systems the model must coordinate. But the transferable idea is the improvement machinery around the model. Red Planet Labs created a benchmark, runner, skill, phased workflow, private oracle, and transcript laboratory. That system can tell them not only that a run failed, but where the workflow made failure likely.
Do not ask only, “Did the LLM produce a good answer?” Ask, “What evidence did this run create for improving the next one?”
Run
Give the agent a controlled environment, tools, and a versioned skill.
Observe
Capture phases, files, tests, timing, tokens, cost, and tool use.
Diagnose
Find the earliest decision that made the failure likely.
Amend
Change general guidance, workflow structure, or evaluation.
Regress
Rerun old challenges before moving to a harder one.
02 / Harness anatomy
Give every claim somewhere to collide with reality
A challenge, not a vibe
Each Rama challenge exposes a problem statement and protocol. Operations, latency targets, scale facts, and invariants make success falsifiable.
Private tests stay private
Reference implementations and tests are encrypted during the run. The final work is checked for correctness, performance, and fault tolerance without leaking the answer.
The agent can work freely
Runs happen in a clean Docker environment with Clojure, Rama tooling, a linter, and a long-running REPL. Fast feedback becomes part of the harness.
Every run leaves evidence
The harness saves phase transcripts and artifacts. A separate analysis tool reconstructs timelines, edits, test runs, errors, reasoning notes, and compaction events.
VISIBLE TO THE AGENT
trust boundaryChallenge brief · protocol · skill · tools · its own tests
VISIBLE AFTER SUBMISSION
Reference design · private correctness, performance, and fault-tolerance tests
03 / Instrumentation
Turn a run into a dataset, not a memory
The runner records each phase's duration, verdict, transcript, token use, estimated cost, tool count, skills, and reference files. It aggregates those into challenge reports and an append-only result history. The analysis script can then reconstruct what happened.
Outcome
Public and private tests, challenge score, iterations, alignment
Execution
Phase verdicts, retries, tool calls, skills and references used
Economics
Wall-clock duration, tokens, cache use, estimated model cost
Decision trail
Plans, validations, implementation history, REASONING.md, confusion notes
Environment
Challenge, model, reasoning level, skill version, runner behavior
challenge private phases duration tools cost verdict bank-transfer PASS 8 19m 42s 81 $... PASS time-series PASS 8 23m 08s 96 $... PASS auction PASS 10 31m 16s 127 $... PASS inspect: reasoning | confusions | timeline | errors | test-runs
Illustrative values. The fields and analysis commands come from the open-source runner.
Instrumentation changes the question
Weak signal
“The answer was bad.”
Useful signal
“The plan chose an object-heavy cache, validation failed to challenge it, and implementation later diverged from the reviewed design.”
04 / Process as a product
The skill carries knowledge. The workflow carries discipline.
Implicit spec
Derive edge cases and invariants without prematurely designing the solution.
Plan
Choose state, data structures, topologies, and performance strategy.
Plan validation
A fresh agent attacks the plan using an explicit checklist.
Implementation
Build to the reviewed plan, then load and lint the namespace.
Implementation validation
Check correctness, efficiency, fault tolerance, and plan conformance.
Tests
Cover the contract and the implicit spec.
Test validation
Look for weak assertions, missing cases, and tests that excuse the implementation.
Finish
Run, repair localized failures, and submit to the private evaluator.
Put the roadmap on disk
Templates and signposts survive context compaction. Whatever file the agent reads next tells it which phase it is in and what comes after. Critical process knowledge no longer depends on chat memory.
Separate creation from validation
A new context reviews the plan or implementation. It has less incentive to rationalize the decisions made by the previous agent. Explicit checklists make the review repeatable.
Do not restart the world for a small defect
Fresh contexts improve independence but add latency. The second report lets a validator repair a localized plan defect in place, reserving full retries for major failures.
05 / The fanout case
The best skill update does not contain the answer
The difficult fanout challenge required compact in-memory timelines, recovery from durable data, balanced work across a skewed social graph, and fair delivery when one account has millions of followers.
The agent reached for familiar structures
- Stored post content inside each timeline cache.
- Used an object-heavy tree map instead of packed primitives.
- Delivered to every follower eagerly, so a huge account could starve smaller ones.
The workflow forced resource reasoning
- Estimate bytes per entry and total memory per task.
- Account for object graphs, boxing, and garbage-collection pressure.
- Ask whether cached fields can be fetched from durable state at read time.
- Have an independent validator redo the calculation.
The agent derived the domain-specific design
The skill never said “use a packed timeline array.” General principles led the agent to compact post references, fetch content on read, and spread large fanouts over time. Report 2 says the challenge then one-shotted consistently across several models.
A useful skill narrows the search space with principles, costs, defaults, and checks. It should not smuggle benchmark answers into the context.
06 / Build the learning loop
Start with one repeatable workflow
You do not need a backend benchmark or a custom model. Choose work you already repeat: research briefs, support replies, data analysis, code migrations, article drafts, or design reviews. Then make quality observable.
- 01
Define a challenge suite
Use representative cases, edge cases, and at least one case the current workflow cannot solve reliably.
- 02
Create an independent evaluator
Combine deterministic checks, domain rubrics, reference artifacts, and human judgment. Keep some tests hidden from the agent.
- 03
Version the whole environment
Record the model, prompt, skill, workflow, tool surface, source material, evaluator, and runtime settings for every run.
- 04
Log outcomes and trajectories
Capture quality, duration, token and monetary cost, retries, tool calls, artifacts, validation verdicts, and explicit reasoning or confusion notes.
- 05
Diagnose the earliest divergence
Find the first point where a successful and failed run meaningfully differ. Later errors are often consequences, not causes.
- 06
Make the smallest general change
Add a principle, cost model, default, negative constraint with a reason, artifact, checklist, tool, or phase boundary.
- 07
Regress before escalating
Rerun the existing suite several times. Only then add a harder challenge that stresses a different capability.
Does it satisfy the real contract across repeated runs?
Where does fresh context, re-reading, or unnecessary rework dominate?
Did quality improve by focusing the search or merely spending more?
Does the workflow work consistently, not only in one celebrated run?
07 / Necessary skepticism
This is workflow learning, not model training
Beware benchmark overfitting
Repeatedly editing a skill against the same tasks can produce a disguised answer key. Keep guidance general, hold out challenges, and test transfer.
Your evaluator becomes the product
A harness optimizes what it can see. Weak tests create confidently wrong progress. Include non-functional constraints and human review where judgment matters.
Reasoning logs are evidence, not truth
An agent's explanation may be incomplete or post-hoc. Correlate it with tool use, edits, timings, artifacts, and test behavior.
More ceremony can make work worse
Fresh phases and exhaustive validation add latency. Use them where failure is expensive, and repair local defects without replaying the entire workflow.
The durable idea
Make failure legible enough to edit.
The Red Planet Labs experiment is interesting because it treats agent capability as an engineered system. Knowledge lives in skills. Process lives in workflows. Evidence lives in transcripts, artifacts, metrics, and tests. When the output fails, the team can change one part of that system and measure whether the change transfers.
That is the beginning of a workflow which does more than use an LLM. It gradually becomes better at using one.
Read the Red Planet Labs series ↗