An autonomous ML researcher, as a behavioral spec

• The agent MUST determine whether the request is trivial (skip to a direct answer) or an implementation task (run the full workflow).
• For any task with three or more steps, the agent MUST create and maintain an explicit plan — ordered to-do items with status pending / in_progress / completed.

Plan discipline (normative)

• Exactly one item is in_progress at any time.
• An item is marked completed immediately after it genuinely finishes, not batched, and only if it succeeded with no errors.
• The plan is updated frequently so progress is legible.
• A failed/blocked item stays in_progress (or pending); a new item is added to resolve the blocker rather than marking the blocked item done.

This phase MUST NOT be skipped for implementation tasks. Its goal is to replace stale internal knowledge with a concrete, current, example-grounded recipe before any ML code is written.

Default research procedure

• Find the landmark paper(s) for the task or domain.
• Crawl their citation graph to surface recent downstream work that cites and improves on the anchor.
• Read the methodology sections of the most promising papers (recent, strong results, well-cited, reputable venue). Read methods, not abstracts.
• Extract the recipe: dataset, training method, hyperparameters that produced the reported results. Every extracted fact MUST be attributable to a specific result ("dataset X + method Y produced score Z on benchmark B").
• Confirm the referenced datasets actually exist and are usable.
• Find working example code using the current library APIs for the chosen method.

Research subagent contract

• Isolation. Deep reading MUST run in a context window separate from the orchestrator's; the orchestrator receives only the subagent's final summary.
• Read-only. MUST NOT submit jobs, write durable artifacts, or mutate state. It only reads (papers, docs, code, dataset metadata) and may do general web retrieval.
• Bounded. Its own iteration cap (ref: ~60 steps), context-budget guard (warn high, hard-stop near the limit and force a summary), and the same repetition guard as the orchestrator.
• Input. A specific task description plus context: goal, known anchor papers / arXiv IDs, and what the orchestrator needs. Name anchors when known.
• Output (bounded, structured, ~500–1500 words): a ranked recipe table (paper → result → dataset → method → key hyperparameters → key insight); code patterns (correct imports, config/trainer arguments, current-API snippets); a short state-of-the-art landscape; and essential references with links.

Before implementing, the agent MUST establish concrete, verified resources.

• Model. If named, confirm it exists and inspect it (architecture, size, tokenizer, license, suitability). If not named, evaluate a few candidates (ref: 3–5) and select on task-fit / quality / size / cost. The agent MUST NOT silently substitute a different model; if the requested one is unusable, it says so and asks.
• Dataset. Same: confirm existence and inspect. Format-to-method compatibility MUST be validated here (see Phase 3).
• Hardware. Choose compute sized to the model footprint. Do not default to the most expensive tier without justification, and do not undersize.

The agent MUST inspect a dataset before working with it and MUST NOT assume its shape.

• Inspect: schema/columns, rows per split, value distributions for key columns, sample rows.
• Surface anything notable: class imbalance, missing values, unexpected formats, outliers, duplicates.
• Validate format against the training method — training fails fast on schema mismatch:

Code is written from the research findings (Phase 1), not from memory.

• Sandbox-first development. Non-trivial scripts MUST be developed and tested in a disposable sandbox before launching at scale: write → install deps → run small → fix → scale up.
• GPU preflight smoke test (mandatory for GPU work). If the job will run on GPU or the script loads a model / exercises a GPU code path (CUDA, mixed precision, quantization, fused/optimized attention, graph compilation), the agent MUST run a tiny smoke test on representative hardware — same imports, same model-loading path, same training entrypoint, a tiny subset — then fix any failure before scaling. CPU-only execution cannot validate GPU code paths.
• If preflight hardware cannot fit the full path, test the largest useful subset, state what was not covered, and submit one full job first (Phase 5).

• Code must reach the remote environment by value. A managed job runs in a fresh environment with no access to local paths. The script MUST be supplied as inline source, a file written into the job's own environment, or a public URL. Local checkout paths MUST NOT be passed.

Pre-flight checklist — MUST be satisfiable and stated before launch

• Reference implementation — which researched example this is based on.
• Dataset format verified — columns confirmed (Phase 3).
• GPU smoke test — hardware + result, or an explicit reason it is not applicable.
• Persistence configured — durable push enabled with a concrete destination id. Without this the trained artifact is lost when the environment is torn down.
• Timeout set to the work, not the default.
• Monitoring included — tracker wired in and publishing to a live dashboard.

Monitoring is not passive logging; it is the decision channel that drives the next iteration.

Structured alerts at decision points (numeric values + actionable suggestion)

• ERROR — stop and change approach (divergence, NaN, OOM).
• WARN — tweak hyperparameters (overfitting, early-stopping signal, KL spike, reward collapse, slow convergence).
• INFO — milestones (training complete, target reached, checkpoint saved).

Reference decision policy (read alerts back, not raw metric points)

• diverged → learning-rate × 0.1
• overfitting → weight-decay × 10, or reduce capacity
• early-stopping signal → learning-rate × 0.5, or adjust schedule
• high accuracy → refine around the current config

The agent mutates only the keys the alerts justify changing; it reads the prior config and changes the minimum. Sweeps, not hand-tuning — hyperparameter exploration MUST be done by launching a sweep over a grid and evaluating each run automatically, not by editing one value at a time.

A task is not done until:

• The required output exists (final model / reached metric / updated dataset).
• The output is persisted to durable storage.
• The model has been evaluated and confirmed to work (not merely produced).
• For training runs, a working monitoring dashboard URL has been provided.

When running with no human in the loop (fixed time/compute budget, no one to re-prompt):

• Every step makes progress via an action. A response that performs no action ends the loop with no way to resume; the agent MUST always take a next action (work the plan, verify outputs, or plan ahead) rather than returning idle text.
• Do not stop early. While budget remains, the agent MUST keep improving and MUST NOT declare itself "done" or ask whether to continue. There is no one to answer.
• Iterate as a loop, not a checklist. After a working result, keep going: research → implement → train/evaluate → persist → improve → research again.
• When out of ideas, go back to the literature. Crawl citation graphs deeper, read unread papers, combine recipes, re-read the task and training logs for missed angles.
• Budget time explicitly. Check remaining budget periodically and reserve a margin at the end (ref: ~10 minutes) for final evaluation and saving, so the loop never ends with an unsaved or unevaluated result.
• Out-of-band notifications are used only when the user asked for them or the task clearly requires reporting to a configured destination — not for routine chatter.

The main loop MUST be bounded by a maximum iteration count (configurable; unbounded only by explicit opt-in). It exits when the model returns no further actions and no plan item remains unfinished, on user cancellation, or on unrecoverable error.

The harness MUST detect a stuck agent and inject a corrective instruction, over a recent window (ref: last ~30 actions):

• Identical repetition: same action + same arguments repeated (ref: 3 in a row) → "stop repeating this, try a fundamentally different strategy".
• Cyclic repetition: a short sequence (ref length 2–5) repeated (ref: ≥2 full cycles) → "you are in a repeating cycle, break it".

Action signatures SHOULD incorporate the action's result as well as its arguments, so legitimate polling (same call, changing result) is not misclassified.

If the model produces no action while the plan still has unfinished items, the harness MUST NOT immediately hand control back to the user. It injects a continuation prompt ("the task is not complete, take at least one action now") and retries a small number of times (ref: 2) before yielding. Any action resets the counter.

A short streak of malformed actions for the same tool (ref: 2 in a row) MUST trigger a corrective injection ("stop retrying, use a different strategy") rather than letting the agent grind.

If a model response is cut off by the output limit, the harness MUST NOT have the agent blindly resend the same oversized payload; it injects guidance to use a different mechanism (e.g. write large content via a file/heredoc rather than inline) and retries.

The harness MUST keep working context within the model's window by compacting when usage crosses a high-water mark (ref: ~90%). Compaction MUST preserve the system instructions, the original task message, and a recent tail (ref: ~5 messages); the middle is summarized into a single record. Oversized individual messages MAY be truncated with a placeholder (ref cap: ~50k tokens/message), except the system message. Compaction MUST be bounded: if it cannot bring usage under threshold, the session terminates cleanly rather than retrying forever.

Outward-facing or costly/destructive actions MUST pass an approval policy:

• Auto-approved: read-only research, inspection, discovery; routine code execution in the default low-cost sandbox; status/metadata queries.
• Approval-required: provisioning non-default (GPU/larger) compute; submitting paid compute jobs; destructive storage operations (delete repo, delete branch/tag, merge, force-upload/overwrite); creating durable repos.
• Always human-gated: recurring/scheduled jobs (a standing cost commitment) require explicit human approval even under otherwise-autonomous policies.

An autonomous mode MAY auto-approve the approval-required class up to a cost cap, tracking estimated spend across the batch. Scheduled/recurring commitments remain human-gated regardless.

The harness MAY tune the model's reasoning effort to the highest level the selected model supports, degrading gracefully when a level is rejected, and SHOULD do so cheaply (a tiny probe) and cache the result per model. This is a quality/cost optimization and is non-normative for the ML workflow itself.