Phase 1A frozen benchmark protocol

Set ID

8
test_set_v1_phase1a frozen on 2026-05-17.

Model

3B
simplefold-3b through native MLX direct CLI mode.

Run outcome

8 / 8
All frozen targets now have SimpleFold-3B scores.

Boundary

review
Biological sign-off is still required before final biological claims.

Purpose

Phase 1A converts the exploratory build into a closed experimental unit. It does not claim a new protein model, a SimpleFold paper reproduction, or cross-model superiority. It asks whether a single M3 Max workstation can run and audit SimpleFold-3B across this frozen set under one documented protocol.

Frozen Target Set

Target	Bin	Length	Role	Pre-freeze evidence
`13CZ:A`	mid	161	repeat-control anchor	technical smoke observed
`13EE:A`	mid	161	benchmark slice	benchmark slice observed
`13FT:A`	mid	228	benchmark slice	benchmark slice observed
`11BC:C`	hard	116	alignment-coverage probe	benchmark slice observed
`12OY:A`	hard	304	membrane/complex context	prospective
`9H8N:A`	long	604	membrane/transport stress	prospective
`9K9X:A`	long	565	long enzyme context	prospective
`9K9Y:A`	long	565	long enzyme/ligand context	prospective

The follow-up biological review packet marks the set as acceptable for operational viability while preserving caveats for biological interpretation. In particular, 11BC:C is coverage-limited and 12OY:A is a membrane/complex stress case.

Required Metrics

Every target card must report prediction status, failure tag when applicable, pLDDT mean, wall-clock seconds, approximate peak RSS GB, sequence-aligned lDDT, TM-score, RMSD, lDDT chain-matching mode, sequence identity, aligned-residue count, route, seed, sample count, checkpoint/cache evidence, and progress samples when available.

Run Outcome

The prospective target execution was completed on 2026-05-17. All four after-freeze targets produced predictions and scores, and the complete 8-target summary is now published.

Target	Stratum	Wall clock	Peak RSS	lDDT	TM-score
`12OY:A`	prospective	308.066 s	33.2323 GB	0.8135	0.8724
`9H8N:A`	prospective	739.974 s	34.6408 GB	0.8420	0.8491
`9K9X:A`	prospective	713.290 s	34.0239 GB	0.8815	0.9467
`9K9Y:A`	prospective	753.399 s	34.1274 GB	0.9157	0.9826

Decision Rules

Rule	Reason
No frozen target can be removed after a bad result unless logged as a deviation.	Prevents opportunistic target selection.
Failed inference counts as a result.	Apple Silicon viability includes failure modes.
Timeout counts as route/model evidence.	Separates orchestration limits from biological claims.
lDDT must show aligned-residue coverage.	Complex-derived references can hide partial comparisons.
Pre-freeze evidence and prospective evidence are reported separately.	Preserves chronology honestly.

Run Template

cd pilot/local-runner

ORCHESTRATOR_URL=https://eigen-reprod-atlas-pilot.raul-chio-leon.workers.dev \
MCP_CITE_VERIFY_URL=https://eigen-reprod-atlas-cite-verify.raul-chio-leon.workers.dev \
MCP_PDF_PARSE_URL=https://eigen-reprod-atlas-pdf-parse.raul-chio-leon.workers.dev \
MCP_STRUCTURE_PRED_URL=https://eigen-reprod-atlas-structure-pred.raul-chio-leon.workers.dev \
MCP_PDB_FETCH_URL=https://eigen-reprod-atlas-pdb-fetch.raul-chio-leon.workers.dev \
MCP_SCORING_URL=https://eigen-reprod-atlas-scoring.raul-chio-leon.workers.dev \
DIRECT_M3MAX_URL=http://127.0.0.1:8001 \
DIRECT_PREDICTION_MODE=simplefold_cli \
SIMPLEFOLD_CLI_BIN=../m3max-server/.venv/bin/simplefold \
SIMPLEFOLD_CLI_TIMEOUT_MS=7200000 \
RESULTS_DIR=./results-phase1a \
bun run pilot \
  --doi 10.48550/arXiv.2509.18480 \
  --url https://arxiv.org/pdf/2509.18480 \
  --recipe-fixture ../fixtures/phase1a/simplefold3b_phase1a_12oya_v1.json \
  --model simplefold-3b \
  --run-id phase1a-2026-05-17-simplefold-3b-12oya \
  --benchmark

Data

Source document: docs/33_phase1a_frozen_benchmark_protocol.md