10,800 records
A distinct synthetic morphological-position dataset with 12 structural positions, 30 contexts, 24 training contexts, 6 held-out contexts, and 30 instances per position/context configuration.
VARZIN V2 — Recoverable but Not Composable: Representation Recovery and Systematic Composition in a Frozen Language Model
Frozen Qwen2.5-1.5B-Instruct strongly encodes the tested synthetic morphological-position variable in a linearly decodable form across held-out contexts, but the preregistered downstream composition test produced 0/58 passes. This is evidence for a separation in this experiment, not a universal theorem that decodability never implies composability.
Evidence status
Strong representation-recovery result paired with a clean preregistered negative composition result. Publication status: Preprint / results-locked research report; not treated as peer-reviewed validation.
Research question and current reading.
Research question
Does strong recovery of a structural variable from a frozen model imply that the model can use that representation as a substrate for systematic compositional generalization?
Frozen Qwen2.5-1.5B-Instruct strongly encodes the tested synthetic morphological-position variable in a linearly decodable form across held-out contexts, but the preregistered downstream composition test produced 0/58 passes. This is evidence for a separation in this experiment, not a universal theorem that decodability never implies composability.
What was actually tested.
- Phase 0: construct, leakage-audit, and cryptographically freeze a 10,800-record synthetic dataset with 12 positions and 30 contexts.
- Phase 1: evaluate surface-span, whole-sentence, and last-token representations across all 29 hidden-state indices without holdout-driven best-layer selection.
- Phase 2: test k = (i + j) mod 12 on 144 ordered pairs (120 train, 24 fixed holdout), 30 realizations per pair, and 10 preregistered seeds.
- Compare TRUE against SHUFFLED, WRONG_OP, and RANDOM conditions; require both paired exact sign-flip p < 0.01 and ≥0.15 balanced-accuracy margin against every control.
Strongest supported findings.
- Phase 1 surface-span mean BA across 29 indices = 0.9666; median = 1.000; 25/29 indices reach BA = 1.000, while embedding index 0 is at chance (0.0833).
- Phase 2 verdict: PASS 0/58, AMBIGUOUS 0/58, FAIL 58/58.
- Closest positive TRUE-vs-control effect = +0.082 BA (MLP, layer 5 vs RANDOM), exact sign-flip p = 0.1719—well below the preregistered +0.15 / p<0.01 thresholds.
- The failure is not explained by constant-output collapse: most evaluated rows use the full 12-class output space.
How this experiment relates to Core-30 and later scaled benchmarks.
29 hidden-state indices
Surface-span mean reached mean balanced accuracy 0.9666; 25/29 indices reached 1.000 under held-out-context evaluation.
144 ordered pairs / 58 layer-composer combinations
120 training pairs and 24 fixed held-out pairs; final preregistered verdict 0/58 PASS, 58/58 FAIL.
Scale boundary: Core-30 remains a canonical reference subset. Larger generated corpora and later benchmark datasets are distinct experimental artifacts and should not be silently relabeled as a single enlarged canonical lexicon.
Negative findings are part of the result.
- Strong linear decodability did not translate into detected systematic generalization of the TRUE composition rule on unseen ordered pairs.
- The post-hoc Latin-square pattern is explicitly exploratory/hypothesis-generating, not confirmatory.
- The study does not show that Qwen lacks all algebraic structure or that language models generally cannot compose.
What this record does not establish.
- The result is tied to the frozen Qwen2.5-1.5B-Instruct protocol, synthetic structure, fixed composers, and stated decision rule.
- Historical v1.4 identity remains unresolved; the frozen V2 artifact is an independently generated/audited artifact, not claimed byte-identical to that historical selection.
- Generalization to other models, natural language, operations, and readout architectures remains open.
Boundary: a Zenodo deposit is a citable public record; it does not by itself imply peer review, independent validation, causal inference, or generalization beyond the documented experiment.
Full text, code, data, and provenance.
- Dataset, preregistrations, pair split, condition mappings, results, and audits are linked by documented SHA-256 hashes.
- Frozen dataset SHA-256: a6aa66a92133fee22a98e6628ffd286f69bd2073bc45cb9cdfdab2a48e05ab5e.
- The Zenodo record exposes the exact PDF plus “All code Phase.zip” (about 1.30 GB).
- The uploaded PDF supplied for this website audit is byte-identical to the Zenodo PDF (verified by MD5).
VARZIN_V2_Integrated_Phase0_1_2_Final.pdf
paper · 93.14 KB · md5:01cb5d0c70b078384581fdecc090c223
Zenodo REST file inventory and checksums re-verified 2026-09-14; uploaded paper copy exactly matches the public Zenodo PDF.
Canonical version-specific record.
10.5281/zenodo.22679978
Nirouyar, R. (2026). VARZIN V2 — Recoverable but Not Composable: Representation Recovery and Systematic Composition in a Frozen Language Model (Version V1.0). Zenodo.
Verified repository identity.
Nirouyar, Reza
Normalized project identity: Reza Nirouyar · ORCID 0009-0000-4690-6842.
2026-09-09
Version V1.0 · Preprint · open access.
CC BY 4.0
Reported only when present in verified Zenodo metadata.
Zenodo metadata identifies this item as an open-access preprint published on 2026-09-09 under CC BY 4.0.
Keep this deposit distinct from later interpretation.
The study separates representation recovery from composition and records that successful recovery alone is insufficient evidence for systematic generalization. This is a narrower and more defensible conclusion than treating probe success as proof of compositional competence.
Version V1.0 is the current public Zenodo record listed by VARZIN for the integrated Phase 0–2 report. Any later remediation or follow-up experiment should be cited separately from this fixed deposit.