Inside ml-research.js

One agent() call returns a PLAN_SCHEMA object: it classifies the task, picks taskType (llm | vision | embedding | eval | data | other) and method, and freezes the baseline {model, dataset, method, sequenceLength} exactly as the user expressed it — the frozen object the scope-change guard later compares against.

• isTrivial=true only for a pure factual question / status check / resource lookup → return directAnswer and skip everything (spec's trivial-request branch).
• resolveProfile(taskType) selects the skill, reference scripts, and schema rules.

A parallel([...]) of four read-only Explore finders, each in its own context, each returning a FINDER_SCHEMA object:

• landmark — anchor papers, read methodology (not abstracts).
• citations — downstream work; no full citation-graph API, so it uses paper-page links + host WebSearch/WebFetch and sets fidelity="reduced".
• examples — working code with current APIs, preferring the skill's shipped reference scripts.
• datasets — which datasets produced the reported results; confirm they load.

A synthesis agent folds the finders into one RESEARCH_SCHEMA: a ranked recipe table, code patterns (correct imports + current trainer args), SOTA landscape, references, and citationGraphFidelity.

A read-only Explore critic returns a CRITIC_SCHEMA object. It tool-checks that the cited papers, datasets, and models actually exist and that code patterns use real current APIs. mustReblock=true only for hard problems (a cited resource doesn't exist, a hallucinated import); "is every result truly attributable?" is a soft warn. If blocked, one corrective re-research pass runs.

A parallel([model, dataset]) of Explore agents (RESOURCE_SCHEMA). If the user named a model/dataset, it confirms existence and inspects; if not, it evaluates candidates and recommends. It also recommends a hardware flavor sized to the model footprint (R6).

An audit agent (DATA_AUDIT_SCHEMA) inspects schema/columns, splits, distributions, and sample rows, then validates the format against the method using the profile's schemaRules (e.g. DPO needs prompt/chosen/rejected). A format critic confirms compatibility; if it can be fixed by column remapping it passes with a mapping, otherwise mustReblock → STOP (dataset_format_incompatible, an R8 ask-the-user stop).

The implementer adapts the profile's shipped reference script (e.g. train_sft_example.py) and writes train.py + eval.py into WORK_DIR, returning an IMPL_SCHEMA whose trainScriptContent is the full inline source (by-value transport). The script must wire trackio metrics + structured alerts, a concrete push_to_hub destination, a sized timeout, the chosen flavor, OOM knobs (batch/accum/grad-checkpoint), and SMOKE knobs (SMOKE=1 → 1 step, tiny slice, CPU+fp32, optional tiny proxy model).

An Explore critic reads the files and flags hallucinated imports, wrong trainer args, local-path leakage, missing monitoring, a username/... placeholder destination, short timeout, OOM-readiness, and source-build smells (R12). Any error-severity issue sets mustReblock and a code-fix pass edits the files before anything executes.

Runs attemptWithRetry (≤3): execute train.py with SMOKE=1 on CPU+fp32 for one train step + one eval step (uv run resolves PEP-723 deps). A high loss is not a failure; an exception/import/arg/schema error is. If the real model can't load on CPU, a tiny proxy model is used (the real load is first exercised on GPU preflight). Each failure → analyze → minimal-fix → retry.

Gated by canSpend(estimatedPreflightUSD) and assertSubmitInvariant() before submitting. Then attemptWithRetry (≤MAX_JOB_RETRIES) submits a tiny GPU job by value on a small flavor, polls logs to a terminal state, and validates the GPU-only code paths CPU could not: CUDA, mixed precision, the real model load, small-scale OOM. recordSpend() accrues the cost.

A single CHECKLIST_SCHEMA agent verifies the spec §5.6 pre-flight checklist: reference impl cited, dataset format verified, GPU smoke ok and the real model loaded on GPU, concrete persistence destination, sized timeout, monitoring wired, by-value transport. If !allSatisfied → STOP (preflight_checklist_unsatisfied).

Cost-gated and invariant-checked again, then attemptWithRetry (≤MAX_JOB_RETRIES) submits one full job by value, polls logs to confirm healthy (a step advancing / first metric), then monitors trackio alerts (trackio list alerts --json --since <cursor>). An ERROR-level alert (divergence/NaN/OOM) is treated as a failure to analyze and correct — the alert-driven retry the spec's §5.7 calls for. The dashboard URL is recorded as an artifact.

An EVAL_SCHEMA agent evaluates the trained model and confirms it actually works (not merely produced), verifies it is persisted on the Hub, and reports the metric+value, eval URL, model URL, and dashboard URL. (For eval-only / data-only tasks, this is reached directly from the early exit.)

finalize() runs a read-only, tool-backed Explore agent (COMPLETION_SCHEMA) that re-checks the spec §9 completion criteria — verify, do not trust prior claims: research preceded implementation, resources verified, smoke-tested, checklist satisfied, alerts emitted, result persisted & evaluated, no scope change vs. the frozen baseline, every artifact a direct URL. Sets result.conforms.

Scope-change guard (R3/R4/R7). A pure function checks whether a proposed fix touches a protected key. Even if a subagent doesn't flag scopeChange, this catches it:

const PROTECTED_KEYS = ['method','model','dataset','max_seq_length', /* …variants… */]
function isScopeChange(configChanges) {
  return Object.keys(configChanges).some(k => PROTECTED_KEYS.includes(k.toLowerCase()))
}
// in the retry loop:
const scope = analysis.scopeChange || isScopeChange(analysis.configChanges)
if (scope) return { stopped: true, stoppedReason: 'scope_change' } // STOP, ask the user

Billable-submit invariant (§5.6). Run before every paid submit; refuses to launch unless the code is present by value, the timeout is sized, monitoring is wired, and no local path leaked:

function assertSubmitInvariant(impl) {
  // missing unless: trainScriptContent (inline, >=50 chars), timeoutHours,
  // monitoringWired, and NO /Users/ or ./ml-run/ path in the script source
  return { ok: missing.length === 0, missing }
}

One helper drives CPU smoke, GPU preflight, and the full job. It never retries unchanged (R10): every retry is preceded by a diagnosis and a bounded, scope-safe fix.

for (let attempt = 1; attempt <= maxAttempts; attempt++) {
  const res = await submit(attempt, ctx)
  if (res.status === 'success') return res          // done
  if (attempt === maxAttempts) return { stopped:true, stoppedReason:'retries_exhausted' }
  const analysis = await analyze(attempt, res, ctx)   // FAILURE_ANALYSIS
  if (scopeChange || analysis.unrecoverable) return { stopped:true }  // STOP
  await applyFix(analysis, ctx)                       // edit train.py
  ctx.oomLadderRung = Math.max(ctx.oomLadderRung, analysis.oomLadderRung)  // R7 ladder carries
}

The carried ctx remembers the OOM ladder rung (so R7 escalates 1→2→3 across attempts) and the alert poll cursor (--since).

Each is a read-only Explore agent returning a CRITIC_SCHEMA with a mustReblock flag. They turn "verify, never assume" into a tool-backed second opinion at the riskiest moments:

• Verify Research (Phase 2) — do cited papers/datasets/models exist; are APIs real?
• Format critic (Phase 4) — is the dataset shape compatible with the method, or remappable?
• Code review (Phase 6) — hallucinated imports, local-path leak, missing persistence/monitoring.
• Final verification (finalize) — tool-backed conformance against the spec §9 criteria.

One table lets a single pipeline serve every task type without per-type branching. Each profile names the skill, the per-method reference script, the dataset inspector, the eval skill, and the schema rules:

llm: {
  skill: 'huggingface-skills:huggingface-llm-trainer',
  scriptByMethod: { sft:'train_sft_example.py', dpo:'train_dpo_example.py', grpo:'train_grpo_example.py' },
  evalSkill: 'huggingface-skills:huggingface-community-evals',
  schemaRules: { sft:'messages | text | prompt+completion', dpo:'prompt, chosen, rejected', grpo:'prompt' },
}

Profiles eval (evalOnly) and data (dataOnly) trigger the early exits; vision, embedding, and other reuse the same flow.

The analysis prompt injects a closed list of failure categories, each with the only allowed minimal fix — so diagnosis is constrained, not open-ended. A few:

• oom → the R7 ladder in order (preserve effective batch → grad-checkpoint → bigger memory); never reduce seqlen or switch method.
• wrong_trainer_arg → re-check current docs (R11) and correct the kwarg; set recheckedDocs.
• dataset_schema_mismatch → remap columns; if fields genuinely missing → scope-change STOP (R8).
• timeout → raise the timeout (R5); shrinking data/epochs/seqlen to fit is a forbidden scope change.
• divergence_nan → corrective config change (lr×0.1 or per the alert) — iteration, not a crash.

Implementation and analysis prompts are prefixed with a HARD_RULES block that restates R1–R14 and the principles in imperative form (research-grounded, verify-never-assume, no silent scope change, no silent substitution, persistence, sized timeout/hardware, the OOM ladder, prefer prebuilt, secrets-from-env, direct URLs), plus a SKILLS_NOTE telling the agent to drive the HF skills and prefer their shipped reference scripts over writing ML code from memory.

Two helpers bound HF spend independently of the harness token budget. Each billable submit is gated:

function canSpend(estUSD){ return (spentUSD + estUSD) <= COST_CAP_USD }
// before GPU preflight and before the full job:
if (!canSpend(est)) return { stoppedReason: 'cost_cap_full_job' }  // fully prepared; only spend is gated
recordSpend(actualOrEstimatedUSD)  // accrue after submit

Every subagent is forced to return a validated object, so the orchestrator branches on fields rather than parsing prose: