Purpose
evidence
Failures are part of the viability result, not noise to hide.
Current entries
16
Timeouts, scoring assumptions, checkpoint setup, CLI observability, auxiliary caches, repeat variability, metadata gaps, alignment coverage, runtime-card packaging, release packaging, PDF toolchain friction, and archive-upload correction.
Boundary
bounded
These findings bound the completed Phase 1A result; they do not imply broad biological generality.
Entries
| Date | Run | Difficulty | Diagnosis | Mitigation |
|---|---|---|---|---|
| 2026-05-13 | phase1-rehearsal-2026-05-13-simplefold-100m-3targets | Public route returned HTTP 524 on long target 13BB:A. | Route orchestration timeout, not model failure. | Added orchestration_timeout; long target completed through direct local prediction. |
| 2026-05-13 | phase1-rehearsal-2026-05-13-simplefold-100m-3targets | lDDT failed under residue-ID alignment on real PDB numbering. | Predicted single-chain residues may be numbered from 1 while reference PDB residues are not. | Added single-chain sequence-aligned scoring. |
| 2026-05-14 | phase1-rehearsal-2026-05-14-simplefold-3b-short-9pfe | First 3B run outlived runner client while downloading checkpoint. | Install-time checkpoint acquisition exceeded client request window. | Cached checkpoint, recorded SHA256, separated install evidence from cached inference. |
| 2026-05-14 | phase1-rehearsal-2026-05-14-simplefold-3b-mid-long | Local HTTP route timed out and left 3B subprocesses running. | Synchronous HTTP remained unsuitable for longer 3B jobs. | Stopped orphaned subprocesses and added DIRECT_PREDICTION_MODE=simplefold_cli. |
| 2026-05-14 | phase1-rehearsal-2026-05-14-simplefold-3b-mid-13cz-cli | CLI mode emitted no structured progress until completion. | CLI execution avoids HTTP timeout but had an observability gap. | Added runner-side progress logging and approximate RSS peak capture; rerun needed to populate old cards. |
| 2026-05-14 | phase1-rehearsal-2026-05-14-simplefold-3b-mid-13cz-cli | CLI created large auxiliary cache files inside a run-specific temp output. | Per-run setup may affect wall-clock time and disk use. | Track auxiliary cache separately from checkpoint download and pure inference. |
| 2026-05-14 | phase1-rehearsal-2026-05-14-simplefold-3b-mid-13cz-cli-telemetry | Repeat CLI smoke showed close lDDT/TM-score but different pLDDT and wall-clock. | Structure score was stable enough for smoke evidence, but confidence and timing need controls. | Add repeated-run controls and cache-state notes before throughput or confidence claims. |
| 2026-05-14 | phase1-rehearsal-2026-05-14-simplefold-3b-mid-13cz-cli-execmeta | Cards needed structured execution metadata. | Prose notes are not enough for reproducible route/cache interpretation. | Added execution metadata with route, seed, checkpoint files, auxiliary cache files, and progress samples. |
| 2026-05-14 | phase1-science-slice-2026-05-14-simplefold-3b-13ee, 13ft | Initial benchmark cards showed high TM-score but artificially low lDDT. | Residue-number matching compared model-numbered predictions against PDB-numbered references. | Re-scored cards with single-chain sequence-aligned lDDT. |
| 2026-05-14 | phase1-science-slice-2026-05-14-simplefold-3b-11bc | Hard-bin target used peak RSS 41.4060 GB and aligned only 63 reference residues. | Complex-derived/partially observed references require coverage-aware lDDT interpretation. | Added lDDT alignment metadata and an aligned-residue reporting rule. |
| 2026-05-17 | phase1a-2026-05-17-simplefold-3b-12oya, 9h8na, 9k9xa, 9k9ya | The first two prospective targets have primary cards but no terminal tee logs; later targets include logs. | Review-packaging gap, not a scientific-result failure. | Treat cards/scores/summary files as primary evidence; logs and screenshots are supplemental aids. |
| 2026-05-17 | phase1a-2026-05-17-simplefold-3b-9h8na, 9k9xa, 9k9ya | Long-bin targets completed without OOM at 34.0239-34.6408 GB peak RSS. | For this model/route/hardware/set, long-bin inference was runtime-bounded rather than memory-bounded. | Report per-target runtime and RSS; keep claims limited to observed 565-604 aa targets. |
| 2026-05-17 | simplefold3b_phase1a_environment | Benchmark interpretation depended on an implicit local setup unless hardware, versions, source commits, and checkpoints were promoted to first-class evidence. | Reproducibility requires a runtime card, not only target-level result cards. | Added public model/runtime card, environment JSON, and browser screenshot. |
| 2026-05-18 | phase1a_repeat_controls | Most Phase 1A targets were initially single-run observations. | Acceptable for completion/viability, but weak for runtime and confidence interpretation. | Added n=3 controls for 11BC:C and 9K9Y:A, with 13CZ:A as the existing route/cache anchor. |
| 2026-05-18 | v1-workshop.pdf | PDF generation exposed native toolchain assumptions: WeasyPrint lacked Pango/GObject and Pandoc/Tectonic lacked SVG conversion. | Venue formatting is a packaging/toolchain issue, not a scientific-results issue. | Generated the current review PDF from the Pandoc HTML using Chrome headless; kept HTML and Markdown as canonical manuscript artifacts. |
| 2026-05-18 | zenodo-20275055 | The corrected archive upload hit local TLS certificate validation failure, then Zenodo HTTP 415 on non-octet-stream uploads. | Archive publication has its own reproducibility/tooling constraints; DOI metadata alone is not enough if the archived PDF is stale. | Retried with an explicit certificate bundle, deleted partial draft files, uploaded all release files as application/octet-stream, and published DOI 10.5281/zenodo.20275055. |
Paper Implications
- Inference route must be specified as part of Methods.
- Long-running local inference should not be evaluated through public synchronous HTTP.
- Scoring methods must account for real PDB residue numbering.
- lDDT must be reported with aligned-residue coverage for complex-derived references.
- Checkpoint acquisition, auxiliary cache materialization, cached inference, scoring, and peak memory must be measured separately.
- Failure tags are meaningful only when route, install, and scorer failure modes are separated.
- Repeated-run controls are required before making throughput or confidence-calibration claims.
- Logs and screenshots are supplemental evidence; final numeric claims trace to result cards and generated CSV/JSON summaries.
- Model/runtime cards are part of the evidence chain, not administrative metadata.
- Long-bin SimpleFold-3B inference on the M3 Max appears feasible for the observed 565-604 aa Phase 1A targets, but the statement is route/model/set-specific.
- PDF rendering can fail on native dependencies even when manuscript content is ready; preserve HTML and source Markdown as canonical until final venue formatting.
- Archive publication can fail on TLS/tooling and upload content-type details; verify the final archived files, not only the DOI metadata page.