ETTR Phase 2 Data and Compute Plan

Status: active operational plan, 2026-07-28. This plan authorizes measured Phase 2 preparation and training only after its named gates pass. The protected step-300k checkpoint remains a read-only trust root.

Decision

Shohin needs two distinct training streams.

1. General pretraining stream. High-quality language, math, code, science, and procedural documents train the 125.08M-parameter trunk's language, knowledge, and representation substrate.
2. ETTR-native stream. Exact WORLD/COMMAND/QUERY episodes, terminal packets, generic transactions, interventions, and invariant views train the 67.70M new compiler/reactor/query-reader parameters to use the architecture.

The streams are complementary. General text cannot identify the ETTR state machine, while ETTR episodes are too narrow to supply broad language and world knowledge.

Current Inventory

General data

The historically mounted, evaluation-decontaminated corpus contains 62,426,256,278 manifest tokens:

Two future candidates add 30,000,001,936 tokens:

The combined available inventory would be 92,426,258,214 tokens if both candidates were approved. All six full-corpus structural scans completed on 2026-07-28. Every scanned shard had zero byte-fallback fraction. DCLM's entropy range was 10.500--10.590 bits/token and OpenMath's was 7.995--8.003 bits/token, with tight token-frequency distributions. Those measurements establish token-stream integrity, not semantic quality.

Deterministic manual decoding of five DCLM shards found coherent long-form prose but also low-value forum conversation, dated news, and awkward/SEO-like text. Five OpenMath shards contained useful worked mathematics but also at least one visibly unreliable geometry derivation and generated prose that can sound authoritative while self-correcting or remaining incomplete. Therefore neither candidate is admitted wholesale. DCLM requires document-level quality stratification; OpenMath requires answer verification and rejection of unverifiable or inconsistent solutions before either approval record is written.

The 4.60B-token FineWeb-Edu sample-10BT artifact is an undersized diagnostic pilot and is not a 25B replacement. The quality-first target remains at least 100--120B admitted unique general tokens, with cross-source near-dedup and equal-token source ablations before a long run.

The active admission policy is docs/research/PHASE2_DATA_SELECTION_STANDARD.md, and the machine-readable candidate registry is pipeline/pretrain_sources.json. The next corpus expansion prioritizes selected FineWeb-Edu, English FinePDFs-Edu, license-resolved Common Pile components, and a measured Stack-Edu versus Stack-v3 code challenger. Raw FineWeb, raw FinePDFs, wholesale DCLM, wholesale OpenMath, and synthetic FinePhrase are not core-stream substitutes. FinePhrase's structured FAQ, math, table, and tutorial formats are a zero-weight late-pretraining challenger after the April 2026 controlled study, with hallucination, truncation, source-duplication, and template-diversity gates. The proposed source mixture is an ablation schedule, not permission to train; each selected payload still requires the complete hash-bound admission receipt.

ETTR-native data

The frozen target is 62,500 semantic cores. Each core expands into 64 architecture-native rows and 528 charged positions:

As of this plan, 207/216 main receiver-qualification reports and all 108/108 sealed-confirmation reports are durable. Nine long main cells are running under 48-hour recovery job 753823. Selector 753824, task manifest 753825, materialization arrays 753826/753827, aggregate audits 753828/753829, and separation audit 753830 are dependency-held.

No partial or pre-audit ETTR payload may be consumed by training.

Immutable training release

The optimizer does not read materializer output directly. After the main and confirmation aggregate audits pass, the release builder must:

1. Reopen and hash every train/development source shard.
2. Fully rematerialize every semantic core with the pinned tokenizer.
3. Divide each 64-row core into four 16-row batches only on complete causal rectangle boundaries, with all row references remapped and revalidated.
4. Build the target-bound packet/query sufficiency index from those exact 16-row payloads.
5. Publish no-replace continuation, stream, shard, tokenizer, audit, separation, and packet-index receipts under one release SHA-256.
6. Keep train/development reserves and sealed confirmation data outside the optimizer-visible release.

The continuation manifest's legacy qualification_payload_sha256 field binds the main audit's qualification-freeze receipt. Its legacy hybrid_payload_sha256 field binds the independent main/confirmation separation-report digest. These mappings are compatibility names, not permission to substitute another payload.

The distributed loader accepts the externally pinned release-file SHA-256, rehashes every source shard before launch, rematerializes only receipt-listed batches, and permits restart only at a global optimizer-update boundary. Each rank receives a disjoint global position, while gradients are averaged before clipping and optimizer update. Checkpoints bind the protected base, complete ETTR model, optimizer, schedule, RNG, epoch/global-position cursor, world size, accumulation, release hash, and between-episode lifecycle.

Training Sequence

1. Resource and transport qualification. Use the 10-H100 backfill allocation for H100 memory sweeps, a bounded multi-node DDP/NCCL canary, and ETTR composite-objective throughput measurements. Do not infer quality from a transport canary.
2. Architecture bootstrap. Run a bounded ETTR-native pilot with the base frozen. Require finite losses and gradients, exact checkpoint resume, and held-out causal-control improvement before increasing the budget.
3. Joint continuation pilot. Interleave general LM updates and ETTR-native updates while training the complete 192.78M-parameter system. Freeze validation sets and compare at least two stream ratios at equal presented positions.
4. Scale only the winning schedule. Continue to the long run only if general NLL is retained and ETTR treatment improves against query-only, state-reset, deranged-binding, and dense-state controls.
5. Post-training follows later. Instruction following and user-visible reasoning traces are a separate phase and cannot substitute for native ETTR causal qualification.

Initial stream ratios are experiment arms, not assumptions. Start with an ETTR-only architecture bootstrap, then compare 95/5 and 85/15 general/ETTR charged-position ratios. Repetition, unique positions, and per-source epochs must be reported separately.

ETTR Learning Gate

train/eval_ettr_v3.py is the mandatory development evaluator for every ETTR-native pilot. It reconstructs the raw architecture with the exact architecture seed and, for a trained arm, validates the immutable run contract, optimizer configuration, complete parameter receipt, exact-resume checkpoint, distributed cursor, release, tokenizer, packet-sufficiency index, and protected base. It then evaluates raw and trained models on the same deterministic development microbatches.

The report includes all twelve objective losses, complete supervision counts, WORLD and COMMAND query-binding margin rates, per-batch identities, paired checkpoint-minus-raw deltas, 95% confidence intervals, and parameter receipts. The strict bootstrap learning signal requires:

• the checkpoint parameter receipt differs from raw initialization;
• the upper 95% confidence bound for paired total-loss change is below zero;
• WORLD query-binding margin satisfaction increases; and
• COMMAND query-binding margin satisfaction increases.

Training loss alone, transport throughput, or a changed parameter hash is not an ETTR capability result. Run the initial ladder at approximately 100, 500, and 2,000 optimizer updates. Stop or revise the objective when the fixed development gate fails; do not spend the 10- or 20-H100 allocation on a blind 300,000-update continuation.

train/jobs/run_reserved_ettr_v3_pilot.sh is the reservation launcher for this ladder. It verifies the immutable source archive, exact release, concrete healthy node list, protected checkpoint, update range, and optional exact resume state. It launches one torchrun worker per selected H100 through the existing reservation, then uses one H100 to evaluate the final checkpoint against raw initialization before publishing a result hash inventory.

Before the first release-visible optimizer, train/jobs/run_reserved_ettr_distributed_canary.sh must run train/canary_ettr_distributed_h100.py on the intended multi-node geometry. The canary sees no release or training shard. It executes two B16 architecture-only updates from deterministic rank-distinct synthetic batches, uses the production fixed-bucket gradient averager, and requires exact post-update parameter equality across every rank while preserving the protected checkpoint byte-for-byte.

Newton Capacity

Both jobs request two H100s, four CPUs, and 32 GiB per node. Newton advertises HDR InfiniBand. A same-switch-only constraint delayed both test-only start estimates, so the live requests remain unconstrained and topology is measured instead of assumed.

The 20-GPU shape is promoted only if the measured canary is healthy and efficient. Otherwise the allocation is partitioned into independent 2-, 4-, or 10-GPU training/ablation lanes. Reserving 20 GPUs does not require forcing one inefficient 20-way model replica.

Hard Launch Gates

• exact private source commit installed in a clean shared Newton checkout;
• all requested GPUs pass real BF16 allocation and InfiniBand device checks;
• multi-node rendezvous and NCCL complete without rank loss or hang;
• the synthetic ETTR distributed canary produces identical post-update parameter hashes across every intended rank;
• general shard manifests, scans, approvals, and source weights are frozen;
• ETTR selection, both materializations, both aggregate audits, and separation audit pass;
• production record-to-ETTRContinuationBatch loader re-hashes every batch;
• exact-resume checkpoint save/load passes after a real optimizer update;
• raw and checkpoint arms pass the paired immutable-development evaluator;
• output directory is fresh, isolated, no-replace, and never aliases the protected checkpoint or historical flagship output;
• evaluation and rollback checkpoints are scheduled before the long run.
• the reservation launcher completes its paired development evaluation before another update rung is authorized.

Active Jobs

• Newton DCLM scan: 719553
• Newton OpenMath scan: 719554
• Newton historical-stream scans: 719561--719564
• Stokes ETTR long-cell recovery and downstream chain: 753823--753830
• Stokes read-only Phase 2 source probes: FineWeb-Edu 753857, FinePDFs-Edu English 753858, Common Pile/Comma 753859, peS2o 753862, and Stack-Edu Python metadata 753863
• Stokes current-source challenger probes: Stack v3 753874 and FinePhrase FAQ/math/table/tutorial 753875--753878
• Stokes score-4+ FineWeb-Edu candidate builder: 754154
• Serial fresh-source challenger profiles after 754154: Dolma 3 components 754163--754166, PleIAs Common Corpus 754167, and Essential-Web 754168
• Newton four-H100 allocation 719591: evc40 independently reproduces a CUDA initialization failure. Clean three-rank DDP transport canaries on evc28, evc30, and evc45 completed at approximately 298k tokens/s for BS=16,ACC=1 and 436,907 steady tokens/s for the production-shaped BS=32,ACC=4 update. The latter uses 524,288 charged positions per update, approximately 59,273 MiB per H100, and had no skip, OOM, NCCL error, or DDP hang. These are initialization-only hardware measurements, not production or capability checkpoints.

The exact inverse ETTR record materializer is implemented as rematerialize_record. It reconstructs all 64 rows from one frozen SemanticCoreRecord, verifies source/view/token receipts, and requires the resulting tensor batch to reproduce the stored materialization SHA-256. All three ontology families and hostile source/target mutations are covered. The remaining training critical path is a streaming sharded dataset wrapper and the distributed ETTR trainer/launcher. The bounded generic DDP transport launcher is train/jobs/run_reserved_multinode_ddp_canary.sh.