Capability Diagnosis: 2026-07-12

Executive conclusion

Shohin is healthy as a training run but is not yet an intelligent general reasoner. At raw step 166,250 it behaves primarily as a text-completion model: it can emit fragments of familiar templates but does not reliably execute arithmetic, preserve equation invariants, apply transformations, or follow a concise interaction contract. The first v2 SFT pilot made its reasoning look more coherent and did teach a few narrow routines, but it did not produce transferable problem-solving.

This is not a GPU-utilization or loss-stability failure. It is a curriculum, coverage, and interface-contract failure. Continuing the current live pretrain is safe, but simply continuing the same mix cannot be treated as the complete route to the stated reasoning goal.

The live pretrain remains protected. All diagnosis, curation, and SFT work is isolated from its checkpoint/output directory.

What was measured

Corrected benchmark status

The original shared decoder stopped at any blank line. Many SFT targets place a blank line before The answer is ..., so those old scores are diagnostic only. train/eval_suite.py now stops only after a complete explicit final-answer line.

The CUDA-only corrected public board (686277) completed for sft_v2_120k/sft_ep1.pt:

Earlier results must not be compared directly to corrected-decoder results. The corrected board is enough to reject v2 as a promotion candidate. Its corrected held-out RG result is 90/800 = 11.25%, which is above the raw 120k baseline (29/800 = 3.625%) but remains highly concentrated: chain sums 20/25, string insertion 19/25, basic arithmetic and decimal-chain sums 13/25 each, and products 9/25. It is zero on most transformation, logic, cipher, geometry, and search-like families. The in-training control is only 98/800 = 12.25%, so the roughly one-point gap does not support an exact-trace-memorization explanation. The pilot did learn a few routines that transfer within its limited family coverage; it did not learn a general algorithmic substrate, which is why the public board still rejects the recipe.

The first raw-base board attempt (686314) is invalid as a board: it loaded the rotating ckpt_0168000.pt, completed only GSM8K maj@4 at 1/100, then the source checkpoint was deleted by normal rotation before the other four metrics loaded. This is an evaluation lifecycle defect, not a model result. The evaluator now pins its source before decoding. The corrected raw board (686315) ran from a reflink-pinned best_step168750.pt and completed cleanly: GSM8K maj@4 5/100, GSM8K pass@1 2/100, MATH-500 2/100, HumanEval 7/164, and MBPP 0/100. This is the valid raw baseline for the v4 SFT experiment, not the rotated-checkpoint partial result.

Direct interaction, not only benchmarks

I ran the raw 166,250 checkpoint and the v2 SFT checkpoint on twelve fresh, hand-authored prompts using the same Question: ... Answer: contract used for SFT. The complete verbatim transcript is artifacts/eval_history/interactive_v1_686293.json.

The SFT model therefore has isolated wins (state tracking and a minimal code predicate) and partial correct intermediates, but no robust rule execution. This is direct evidence, not an inference from a loss curve.

Fresh compositional interview: raw 170k still fails every nontrivial task

The seven-case replay could have been dismissed as too close to the earlier diagnostic families, so a second eight-case composition-focused interview was run locally against the preserved raw 170k checkpoint. It uses distinct wording, numbers, and two executable Python contracts: a word problem, base-7 conversion, three-operation counter update, negative sort/deduplicate, string splice, set constraint, count_evens, and sum_positive. Every case receives five turns: initial answer, independent review, a true supplied intermediate, requested state= representation, and reuse of that state. The full unabridged transcript is artifacts/eval_history/generalization_interview_raw170k_20260712_mps.json (md5 d9ad30fad6c00958ad9d6908ca14c38a).

The sole success was the elementary set constraint. The failure traces identify missing execution rather than a stopping-only issue: 17 * 23 becomes 351, then the model does not subtract even when given the correct product 391; the base-7 task uses decimal-style powers; and the counter task jumps from 12 into unrelated competitive-programming boilerplate. It emits no requested state line. For code, it substitutes count_even for count_evens and adds a print-driven template, while sum_positive is malformed/recursive rather than a function with the requested predicate. This independently reproduces the prior direct audit: raw Shohin has no demonstrated general algorithm execution, self-repair, or compact-context continuation at this stage.

V4 has procedural signal, but is not a broad promotion

The corrected V4 r3 held-out procedural evaluator (686337) scored 209/800 = 26.125%, compared with the corrected V2 held-out evaluator's 90/800 = 11.25% on the same rg_v2/rg_eval.jsonl sample seed and complete-answer decoder. Its strongest gains are simple equations (1 -> 17 / 25), number sorting (0 -> 16), isomorphic strings (0 -> 16), AIW word problems (0 -> 14), and decimal-chain sums (13 -> 20). Basic arithmetic and LCM each changed by only one item downward.

This is a real reason to retain V4 as a diagnostic/generator candidate, but not proof that the V4 data alone caused the gain: its base checkpoint is raw 168.75k, whereas V2 used raw 120k. More importantly, V4's public board is still weak (GSM8K majority@4 5/100, MATH-500 1/100, HumanEval 2/164, MBPP 0/100), so it is not a broad promotion candidate. Its adaptive direct interaction (686338) is also only 1/6 initial, 1/6 review, and 0/6 scaffolded. The remaining matched capability matrix will separate prompt-format effects from this local procedural transfer signal.

That matched matrix and the raw-versus-V4 transcript are now complete. On the same 48 cases and seed, raw 168.75k scored Q/A 4/48, direct 5/48, CoT 0/48, and one-shot 7/48; V4 scored 4/48, 4/48, 4/48, and 10/48. The V4 CoT gain is four arithmetic cases, while the one-shot gain is mainly syllogisms; native Q/A and direct instruction remain at or below the raw checkpoint. The complete transcript (interactive_raw_vs_v4_168750_686343.json) shows why the aggregate is not enough: it turns saffron into nroffas, emits 19 * 17 = 303, copies base-6 254 as decimal 254, counts one a in bananas, and fails to carry r = 14 into 3r. V4's extra structure is sometimes useful for sampled procedural tasks, but it is not a dependable execution trace.

V5 proves missing primitives are trainable, not that reasoning is solved

The source-balanced V5 ablation retained broad V4 sources and added a 30% share of 210,000 solver-verified primitive examples. Its 3,500-row held-out bank uses separate prompts, random seeds, and numeric ranges. On the fixed 700-case sample, raw scored 0/700 and V5 scored 272/700 = 38.86%. The family breakdown is decisive: syllogism 100/100, string insertion 88/100, correction 47/100, state update 19/100, arithmetic 12/100, base conversion 4/100, and sort/deduplicate 2/100.

Its direct interview reached 2/8 initial, 2/8 review, 1/8 scaffold, and 3/8 compact-state reuse, against raw 170k's 1/8, 0/8, 1/8, and 0/8. The three reuse successes are state update, sort/deduplication, and precedence correction: all close to explicit V5 curriculum families. It still emits 43 * 17 = 701, fails base-8 conversion and string splice, and cannot write a valid minimal Python predicate. V5 therefore establishes that explicit compact-state supervision can install a few execution moves; it does not establish autonomous compaction, broad self-correction, or general reasoning. Its prompt matrix and public board remain the promotion gates.

On the fixed 48-case matrix, V5 moves native Q/A 4/48 -> 17/48 and explicit CoT 0/48 -> 11/48, driven by arithmetic, sorting, and state updates. Plain direct instruction is only 5/48 -> 6/48 and one-shot falls 7/48 -> 5/48. The result is a useful warning against reporting only the best prompt: V5 has learned some execution under the supervised Q/A or CoT contract, but has not become a reliable instruction-following solver.

The completed public board makes the promotion decision unambiguous. V5 (686401) scored GSM8K majority@4 10/100, greedy GSM8K 9/100, MATH-500 3/100, HumanEval 2/164, and MBPP 0/100. Relative to the pinned raw 168.75k board (5/100, 2/100, 2/100, 7/164, and 0/100), that is a narrow arithmetic-format gain alongside a severe code regression. V5 is rejected as a broad SFT recipe.

A separate fresh seven-case transcript probe (686425) was run after the board rather than inferred from aggregate metrics. Raw 168.75k scored 1/7 initial, 0/7 review, 1/7 after a verified fact, and 0/7 state reuse. V5 scored 3/7, 3/7, 2/7, and 3/7. Its exact wins are the trained arithmetic, sorting, and logic patterns; it still fails base conversion, sequential state updates, string insertion, and syntax-valid Python. V5 normally does not emit the requested state= representation. Its reuse wins are final-answer matches after a new prompt, not evidence that it produced or faithfully continued from a compact state. The hash-matched transcript is artifacts/eval_history/manual_capability_raw168750_vs_sft_v5_20260712_JOBID.json (md5 28dd0b15de2af16a10a2012f630072a1).

V6 learns the constructed contracts, not independent compaction

V6 was designed to remove V5's response-format ambiguity. It uses the same raw 168.75k base and a frozen r2 contract curriculum with explicit Q/A, direct, chain-of-thought, review, scaffold, compact, and reuse targets. The result is substantial on the disjoint generator-held-out set: raw is 20/245 = 8.16% and V6 is 142/245 = 57.96%. The largest apparent gains are review 28/35, scaffold 34/35, and reuse 34/35.

Those numbers do not generalize to the independently phrased deep interview. There V6 is 4/8 initial, 1/8 review, 1/8 scaffold, and 0/8 compact reuse. It can initially solve the arithmetic and state cases, then changes a correct answer to 701 or 62 when asked to review; it turns correct compact prompts into invalid values such as 43 * 17 = 761 and faithfully repeats that wrong state. This is exactly why the project treats generated-contract performance as an atomic gate, not evidence of latent reasoning. The hash-matched artifacts are sft_v6_contracts_168750_r2_contracts_p5.json (md5 5ba5ad4ad6b7562e0daa35e2639cfcf1) and sft_v6_contracts_168750_r2_deep_interaction_686415.json (md5 879403fe4f46cd889ffd618ed037cc2d).

V7 passes a typed-state generator but fails independent interaction

V7 made the state representation explicit and solver-checkable. Its corpus has 315,000 train rows and 10,500 held-out rows across write, repair, and reuse contracts; source-prompt overlap, normalized duplicates, malformed rows, and 13-gram held-out overlap were all zero. On the 420 held-out rendered prompts it achieved 307/420 = 73.10% answer accuracy and 169/280 = 60.36% exact typed-state accuracy. Repair is strong (128/140 answers, 132/140 states) and reuse is 140/140 answers, while write is only 39/140 answers and 37/140 states. Those are meaningful diagnostics of the representation task, not a claim that the model can create or use a general scratch state.

The independent eight-case interview confirms that limitation. V7 scored 1/8 initial, 1/8 after review, 1/8 with a verified intermediate fact, and 0/8 after compact-state reuse. It states 43 * 17 = 751, repeats the error after review, and when explicitly supplied the correct product returns 731 without performing the requested subtraction. It treats base-8 725 as decimal 725, emits unrelated or malformed state fragments for state tracking, sorting, and string insertion, and produces syntactically invalid Python. The only initial/review success is the precedence-correction case; the only scaffolded success is the simple negative syllogism. Therefore V7 is rejected as a general-reasoning or latent-compaction candidate. Preserve it solely as evidence that a small model can learn a generated typed-state contract.

The full local/Newton hash-matched artifacts are sft_v7_state_protocol_168750_p20.json (md5 1f9fe0b2993d1a9dafc98cd2d7943887) and sft_v7_state_protocol_168750_deep_interaction_686484.json (md5 c4963fae52d5ac9c38614e77f93f98c8).

A second, differently phrased seven-case operator interview was run locally on MPS against raw 170k and V7, preserving all five turns per case. Raw is 1/7 initial, 0/7 review, 1/7 verified fact, 0/7 state reuse. V7 is 2/7 initial, 2/7 review, 0/7 verified fact, 1/7 state reuse. Its only initial/review wins are the product-and-subtract and sequential-state templates; the reuse win is the same arithmetic calculation, not a valid emitted compact state. V7 still fails base conversion, sorting/deduplication, string insertion, the negative syllogism, and syntax-valid Python, and it loses both arithmetic/state cases when a correct intermediate fact is supplied. This independent result rules out the interpretation that the earlier 73.10% score reflects a general solver. The hash-matched artifact is manual_capability_raw170k_vs_sft_v7_20260712_local_mps.json (md5 a6d8c25cb3482cd37026bbc85306008f).

Controlled prompt matrix at 168k

The first twelve hand-authored prompts established the failure qualitatively. A second reproducible audit (train/capability_matrix.py, job 686306) then used 48 fresh generated tasks across arithmetic, base conversion, state updates, sorting/deduplication, string insertion, and syllogisms. It tested the raw 168k checkpoint and v2 SFT under four prompt contracts. The complete transcripts and per-family scores are in artifacts/eval_history/capability_matrix_v1_686306.json.

The raw model only solved four negative syllogisms under its native Q/A format. The v2 model gained three arithmetic cases only in that exact format; it scored zero on all eight base-conversion, state-update, sorting, and string tasks in the same condition. A request to think step by step did not unlock latent computation. For example, raw 168k correctly wrote 18 + 9 = 27, 27 * 5 = 135, and 135 - 14 = 121, then continued into a different question and emitted a final 1. V2 instead applied the wrong precedence (18 + 9*5 = 63). This separates a weak output/stopping contract from the deeper missing algorithmic competence.

Multi-turn correction and scaffold test at 168.75k

I also interacted directly with the same preserved raw checkpoint through a six-case, three-turn audit (interactive_adaptive_168750_686316.json). Each case received an initial question, an explicit independent-review request using its prior answer, and a fresh version with one verified intermediate fact. The scores were 1/6 initial, 1/6 review, and 1/6 scaffold. The only exact success in all three conditions was the simple negative syllogism.

The failures identify the missing operation rather than merely a bad stopping token. For 27 * 14 + 9, it asserted 27 * 14 = 398; on review it repeated the same result; with the verified product 378 it repeated the fact but did not add 9. It maps base-7 356 to 1000, turns a state update into repeated additions, returns a generic [1,2,3,4,5,6] for an unrelated sort/deduplicate task, and collapses a string insertion into empty code fences or pq. Thus review and provided state do not activate an unexpressed solver. The model needs training on state transitions, transformations, answer contracts, and correction moves; prompt engineering alone is not a credible remedy.

Compact-state interview at 170k

To test the specific latent-reasoning/compaction claim rather than infer it from the earlier six cases, job 686370 ran a second pinned raw checkpoint interview against best_step170000.pt. It used eight fresh cases across arithmetic, base-8 conversion, state transitions, sort/deduplication, string splice, logic, counterexample correction, and a minimal Python contract. Each case was tested as an initial answer, after independent review, with a verified intermediate fact, and after the model had been asked to create then reuse a compact state= representation.

The canonical, syntax-checked result is 1/8 initial, 0/8 review, 1/8 scaffolded, and 0/8 compact-state reuse. The sole success is the simple logic constraint. The model repeats 43 x 17 = 651, treats base-8 725 as 8 or 1000, uses wrong operator order for the state transition, and emits generic code/search templates for list and string tasks. Its apparent initial code success in the first instrumentation pass was rejected: the generated is_even body was not syntactically valid Python. The scorer was tightened to parse the function AST without executing model-produced code, then the interview was rerun from the same pinned checkpoint. The transcript is artifacts/eval_history/deep_interaction_raw170k_r2_686370.json (md5 1979bcc79cb18830cb3080a7cab85e82).

This is direct negative evidence for the desired feature: the current model does not create a usable internal summary, cannot continue faithfully from one it generated, and does not repair simple errors when prompted. It does not rule out training an explicit compact-state curriculum later; it rules out claiming that the capability already exists.

Training state and corpus replay

At step 168,300 the run has processed 88.24B nominal tokens, or 705.4 tokens per the 125.1M parameter count printed by the actual SFT loader. The codebase/runbook headline should use 125.1M trained parameters unless a new checkpoint proves otherwise; calling it 135M does not make the model stronger, and hides a roughly 10% target mismatch. Mean training loss has been essentially flat across the extension: 1.659 (60k-80k), 1.655 (80k-100k), 1.663 (100k-120k), 1.645 (120k-140k), 1.635 (140k-160k), and 1.640 so far after 160k. This is a healthy optimization trace, but it is not evidence of capability growth.

The live loader uses equal directory round-robin, not corpus-proportional sampling. At step 166,250 it has drawn about 21.79B tokens from each enabled directory:

The next pretraining handoff must not retain FineMath-4 as another directory: it is a subset of FineMath-3 and therefore adds replay, not coverage. The future-only relaunch script now uses OpenWebMath, FineMath-3, OpenMath, code, FineWeb-Edu, and the scanned 25B DCLM replacement. Its effective BS32 mix is approximately 24.8% math, 25.1% code, and 50.1% educational English. This is a forward correction only; it does not alter the active run or claim that historic narrow-math replay was undone.

The active source mix is therefore exactly 75% math-oriented text and 25% raw code. It has no substantial general educational English, logic/deduction, or instruction-following pretraining source. That differs materially from the stated strategy of a language floor plus a reasoning-tilted mix. At 168,300, the equal four-way loader has supplied about 22.06B tokens to each source: FineMath4 has been replayed 11.03 times while FineMath3 has received only 0.88 pass.

There was an additional handoff defect. Checkpoints stored model and optimizer state but not the asynchronous loader cursor. Every resumed Slurm job constructed ShardLoader with the same DSEED=777, so it could restart the shuffled stream from its beginning. The exact replay fraction cannot be recovered after the fact, but the logs confirm multiple completed handoffs with that fixed seed; treating their nominal token count as fully new data would be unjustified. This is now fixed forward-only: checkpoints record a data-stream generation and every resume uses a deterministic new stream seed. It prevents repeated stream prefixes but does not pretend to serialize an exact prefetched byte cursor. The active job is not modified; the first next handoff from an older checkpoint becomes generation 1 and therefore cannot reuse its seed-0 ordering.

Why the first SFT did not repair it

The v2 pilot was cleanly isolated and trained as intended: 349,317 examples, 85.34M packed tokens, 64.29M answer-supervised tokens, and one 2,605-step epoch. The loss fell from roughly 0.99 to 0.46-0.55. That demonstrates the model learned to imitate the answers in the mix; it does not demonstrate broad reasoning.

The actual content makes the result unsurprising:

• 240,297 retained rows are OpenMath-derived, so arithmetic-style derivation is overwhelmingly represented.
• 83,611 procedural traces came from only six earlier hand-built families.
• The mix has only 444 code rows, and just 50 examples contain a code fence.
• This older frozen mix predates source-balanced sampling: all 349,449 rows have no training_group, so its 2,605 packed sequences were shuffled in their natural source proportions rather than deliberately sampling code, procedural, math, and teacher supervision.
• Logic, strings, stateful algorithms, and error correction are too sparse to support the 32-family held-out RG battery.
• The answer-only loss is correct for SFT, but one epoch cannot install missing algorithms that neither the base nor the data has represented broadly.

The completion-mask implementation itself was checked on 1,360 deterministic samples from the v2 mix: every tokenized prompt was an exact prefix of its full prompt-plus-answer tokenization. The poor result is not caused by a shifted label boundary or accidental prompt-token supervision.

There is also a code-specific contract error. train/sft.py teaches every example as Question: {problem}\nAnswer: {code}, while train/eval_code.py asks HumanEval for a raw Python continuation and MBPP with a separate [BEGIN] prompt. A 125M model is highly sensitive to this mismatch. The code result remains genuinely weak (the direct audit found a correct trivial predicate but the public board is low), but it is additionally penalized by training and evaluation on different formats. Future code SFT must include a verified raw-completion form matching the evaluation contract, alongside instruction-form code examples.

The first v4 code-completion pilot uncovered a second, more subtle contract bug before any candidate artifact was accepted. For 461 of 3,542 completion-form code rows, the tokenizer's IDs for the separately encoded prompt were not a prefix of IDs for prompt + completion, most often at CRLF plus indentation. The old packer computed a prompt-length mask from the former but trained the latter, shifting labels at exactly the code boundary that matters. The pilot was canceled and preserved as invalid. train/sft.py now independently encodes the prompt and continuation and concatenates those IDs, which matches autoregressive inference exactly; test_sft_prompt_boundaries.py covers a normal Q/A boundary and the CRLF Python case. The clean v4 rerun starts from the same raw checkpoint only after that regression test passes locally and on Newton.

Latent reasoning / context compaction status

There is no trained latent-reasoning or self-compaction capability in the current flagship. GPTConfig.n_loop exists as an experimental weight-shared repeat of the block stack, but the live checkpoint has n_loop=1. It has never been trained or validated with recurrence. The model has a conventional KV cache for inference; that speeds token decoding but does not compact context or let the model summarize its own reasoning state.

Switching n_loop on at inference would be an untrained architecture change, not extended thinking. The correct path is a separate Mame-scale proxy ablation with recurrence enabled during training, a fixed test-time loop budget, and gates against an equally trained n_loop=1 control. It must not be injected into the live run.

That first mechanical ablation is now complete. Identical 31.5M Mame runs over 800 steps with the same data seed were both stable and each had one recovered grad-norm skip. n_loop=1 finished in 886 seconds at 472.7k tok/s with final logged loss 2.4899; n_loop=2 finished in 1,466 seconds at 286.0k tok/s with final logged loss 2.4890. This proves the implementation can train recurrently, but not a capability benefit: the 1.65x wall-time cost has no measurable short-run loss advantage. Keep recurrence off the flagship until a longer paired capability evaluation earns that cost.

Root causes, ranked

1. Missing reasoning substrate and uneven replay in pretraining. The active equal-domain mix is math/raw-code only, with severe replay imbalance and, before the forward-only handoff fix, potentially repeated stream prefixes. Stable loss here is not evidence of broad skill acquisition.
2. SFT coverage is narrow and format-heavy. It teaches concise derivation style more strongly than reusable algorithms. The direct transcript shows plausible prose without reliable state transitions or invariants.
3. Code is underrepresented and prompt-misaligned. Four hundred forty-four code rows cannot move HumanEval/MBPP, and the SFT/eval prompt mismatch wastes what little code supervision exists.
4. Early evaluation was partially invalid. The blank-line stop depressed older SFT scores. This was fixed before drawing the current conclusion, but it delayed a clear diagnosis.
5. The advertised latent-reasoning feature is only dormant scaffolding. It has no trained behavior and no context-compression objective.
6. Capacity is constrained. The model is 125.1M parameters, not a general frontier model. The viable route is a focused math/code/logic specialist with exceptional data and calibrated decoding, not an unsupported claim of broad general intelligence at this stage.
7. The model has not learned a usable compact-state protocol. A later turn can be correct without using the earlier model text; the transcript must validate both the emitted state contract and faithful continuation before any score is described as latent reasoning.

Rejected explanations

• GPU underutilization: rejected. The live H100 holds 99-100% utilization; BS32 is validated and provides a modest throughput increase.
• Training divergence: rejected. Loss and gnorm are stable; isolated guard skips recover immediately.
• Incorrect SFT output routing: rejected. The pilot initialized from the intended 120k checkpoint and wrote only to train/sft_v2_120k/.
• A single bad prompt or parser: rejected. The raw and SFT failures recur across fresh direct prompts, corrected GSM8K, and the broad procedural gate.
• A broken SFT label mask: rejected. The prompt-prefix/token-mask audit found no sampled boundary mismatch.
• A magical prompt or hidden latent mode: rejected. One-shot prompts gave the raw model only 5/48 and asking for chain of thought gave it 0/48; v2 remained tied to its Q/A template.

Throughput reality check

The H100 is not idling: the live BS32/ACC8 run is holding about 154.2k tokens/s with 99-100% reported GPU utilization. At that rate it processes about 13.32B tokens/day. A claim of 30T tokens in ten days would require 34.72M tokens/s, 225x this run's already-saturated rate, and would take about 6.17 years here. That claim is therefore not a comparable single-GPU pretraining result; it is not evidence that a missing graph-fusion flag explains the capability gap. The measured BS32 change gained about 4%, and the whole-update CUDA graph canary only about 1.8%, which is why they are not the central remediation path.

Remediation plan and promotion gates

1. Keep the protected pretrain running. Do not rewrite its live shard list. At a natural checkpoint handoff, use explicit domain weights to prevent FineMath4 replay from dominating and add the already-tokenized 5.0B-token openmath_pt source.
2. Restore a language/instruction floor. fineweb_edu_probe.sbatch is validating a decontaminated educational-English source before any tokenization. A full source is admitted only after schema, contamination, manifest, and quality checks; it remains future-relaunch-only.
3. Replace the six-family procedural SFT component. rg_v4 already contains 374,659 answer-checked, deduplicated traces across 25 families. Build a new frozen mix from it, not from writer files or the older six-family set.
4. Scale and audit code before a new SFT. The first APPS scan retained 234 verified rows from 5,000 candidates, so a 75,000-candidate isolated scan is running rather than pretending the pilot met its 3,000-row goal. CodeContests is a separate train-only, execution-verified source. Neither enters a mix until its final quality report is clean.
5. Use measured source balancing for v4 SFT. Do not let code remain a sub-1% residue merely because the math corpus is larger, but do not turn a small verified set into a memorization loop. The frozen v4 mix has 62,926 packed sequences: math 34,848, procedural 24,847, code 1,225, teacher 2,006. Its pilot uses 40/47/8/5 math/procedural/code/teacher, which gives code about 4.1 replays and teacher 1.6 per epoch rather than the unearned 7.7/3.1 of the prior 40/35/15/10 proposal. It includes raw-code-completion templates that better match code evaluation and is frozen with quality/contamination reports.
6. Gate every candidate on four comparisons. Require: corrected public board, balanced held-out RG, a fixed direct-interaction transcript, and code execution under a prompt format represented in training. Promote only if it improves the relevant axes without a material regression on the others.
7. Treat latent reasoning as an ablation, not a promise. Train and compare n_loop=1 versus n_loop=2 at Mame scale first. No live architecture change is justified until that measured proxy wins.

Current execution gates

• train/eval_nll.py now provides a fixed-input, token-weighted NLL/perplexity monitor with independent named domains. It excludes training-only auxiliary loss and writes immutable per-checkpoint reports. This is intentionally not a reasoning score: use it to detect broad-language/math/code likelihood changes around the future curriculum handoff, while direct transcripts and public boards remain the promotion evidence. Monitor text must be frozen outside artifacts/evals and outside training shard paths; it must never be silently added to the live decontamination glob or to a training corpus.
• The first language-trend baseline is raw 170k on a frozen WikiText-103 test monitor: NLL 3.9648849 / PPL 52.7142 across 301,056 scored tokens. The monitor input is SHA-256-bound in the result, and the local/Newton artifact is artifacts/monitor_history/raw170_wikitext_mps_hashbound.json (md5 fa9f0ea310287d710d9300c8cb0781ab). This does not prove web-disjointness and cannot be substituted for the reasoning board; it gives the future language-balanced handoff a reproducible English-likelihood trend line.
• A second fixed CodeContests test-split monitor separates code continuation from executable problem solving. Raw 170k reaches NLL 1.3537146 / PPL 3.8718 over 145,408 code tokens, yet still scores 7/164 HumanEval and 0/100 MBPP. Treat this as directional rather than causal: raw code_python uses CodeParrot-Clean, and source-level overlap with CodeContests has not been ruled out from the tokenized shards. It is still insufficient to claim code capability because likelihood does not measure task parsing, solution selection, or instruction-aligned executable emission. Scale execution-verified code and preserve the represented completion format, but require transfer to code-execution benchmarks before claiming progress.
• The independent raw/V7 interactions rule out a hidden general solver: neither verified intermediate facts nor model-produced compact states unlock reliable arithmetic, transformations, or code. More state-template SFT is prohibited until an independent transfer gate changes that conclusion.
• A fresh local-MPS seven-case, five-turn replay against the preserved raw 170k checkpoint independently reproduced this result: 1/7 initial, 0/7 review, 1/7 verified-fact use, and 0/7 compact-state reuse. The hash-matched local and Newton transcript is manual_capability_raw170k_refresh_20260712_mps.json (md5 9cd3216365b5292851e298bee4a1aeef).
• The first clean cross-family verifier derivative contains 1,201 positive and 1,201 negative examples but packs to only 737 sequences. That is about 92 updates across two epochs, so it is rejected as underpowered. A 10,000-question train-only expansion is generating 16 student candidates per question; only a clean, materially larger packed derivative may proceed to isolated verifier SFT.
• DCLM and FineWeb-Edu are being rebuilt with direct current-eval n-gram decontamination. Their 25B replacements are future-only and require a full decoded-shard scan after successful manifests. Neither pilot nor a partial directory can enter the flagship.
• Competitive-programming TACO examples remain a separate algorithmic-code source. Every supplied test case must pass before an isolated transfer ablation; it must not be counted as HumanEval-style completion supervision.

The immediate success criterion is not a prettier loss or longer derivation. It is repeatable improvement on fresh answer-checked tasks with direct transcripts that show correct state updates, transformations, and final answers.

Raw 190k Decode-Depth Diagnostic

A fresh seven-case, five-turn interview was run locally on MPS against the preserved 190k checkpoint. At the standard 128-token budget it scored 1/7 initial, 0/7 independent review, 1/7 with a verified fact, and 0/7 after compact-state reuse. The full artifact is artifacts/eval_history/manual_capability_raw190k_20260713_mps.json (md5 86214f2d4b096a67950cb1885c4109fd).

One state-transition response emitted the locally correct sequence 14 + 9 = 23, 23 * 3 = 69, 69 - 20 = 49, then looped between 49 and 69 instead of committing a final answer. The same frozen interview with a 32-token cap changes only this narrow behavior: 2/7 initial and 1/7 review, with the state-transition trace cut immediately after 49; verified-fact use remains 1/7 and state reuse remains 0/7. That artifact is artifacts/eval_history/manual_capability_raw190k_20260713_mps_max32.json (md5 6f37c4fcf44351773981c83c12c68811).

This is evidence of a termination and completion-contract defect layered on top of weak computation, not evidence of thinking: the same model still gives 29 x 16 = 496, treats base-6 425 as decimal 0.425, cannot apply a supplied correct product, fails state reuse, and falls into unrelated templates on string, sorting, and code tasks. Future work may improve explicit answer commitment, but a short decoder cap cannot count as a reasoning solution unless it also transfers to held-out fact, review, and state-reuse conditions.

A matched 48-case QA matrix with a new seed tests whether this is broad. The 32-token decode scores 8/48 = 16.7% and the standard 128-token decode scores 7/48 = 14.6%. The sole changed item is one state-update case; both budgets are 7/8 syllogisms and 0/8 arithmetic, base conversion, sorting, and string insertion. The records are capability_matrix_raw190k_seed20260713_qa_max32_mps.json (md5 89889f3616cae656c415bf277244c67f) and capability_matrix_raw190k_seed20260713_qa_max128_mps.json (md5 83ea629a248655bc4b7ceec3ecb8ec66). This rejects decoder budget or answer-commitment tuning as a broad capability intervention at raw 190k.

Raw 260k Continuation-Mode Confirmation

The old seven-case direct-interaction score understated one narrow capability because the raw pretrained model often emits a worked solution and then continues into another textbook example instead of placing only the final integer in the answer slot. This was tested without changing the checkpoint or using training data.

An exploratory four-case, five-format probe first found three brittle positive paths: one complete sequential update, one modular worked continuation, and one correct subtraction from a supplied product. That artifact is artifacts/eval_history/raw260k_continuation_modes_20260715_mps.json, SHA-256 f462391f3351a8491955587c036e7579559deb6d52e3c44827a236f245d41290.

A score-blind confirmation then generated 20 fresh cases from fixed seed 2026071501, five each for multiply-subtract, base conversion, sequential state update, and modular update. The immutable case manifest SHA-256 is 3bae0add841e403d01251ae6e6ff110f3c6a07324b28de1b671a59f012071f7c; the transcript artifact SHA-256 is f333c8f54383c411813551bc2001077b88e49514923b76c3cfe0331e9fd6bb47. After a parser defect around numbered Question 2: headers was found, the responses were not regenerated. A separate hash-bound assessor read the same immutable transcript and wrote SHA-256 058aa9dafdc741efc181e6377db5d46b233875504b4b4b6d92837a0db71ea62b.

Strict final-answer results over the first answer segment are:

The sequential-state result is genuine narrow procedural evidence: every worked-continuation case contains both correct intermediate states and the correct final answer, while all five direct cases contain the correct answer and four stop the first answer segment correctly. It is not broad arithmetic. The model gets all modular sums but only two remainders under demonstrations, gets only two of five multiplication intermediates, and treats every tested non-decimal numeral as positional-looking decimal text rather than applying the base.

The operational conclusion changes in one precise way. Shohin is not a blank template generator: pretrain-native worked-example context can elicit a stable local add/multiply/subtract transition routine. However, the routine does not transport across arithmetic families or response formats. Future training should test whether surface-format conjugacy can stabilize an already present procedure, but one successful family cannot be promoted as latent thought or general reasoning.