TR-3 study models: main compressed models and matched-budget baselines
Layout: / / . Method families are tr3 (our 3-site tensor ring +
sparse residual), tt3 (3-site tensor train ablation), svd_llm, svd_llm_v2 and asvd_llm
(matched-budget baselines). Stages are heal
(KD-distilled, deployable) and unheal (raw post-compression, for measuring the healing delta).
| method | stage | model | params (B) | licence | KD result (probe) |
|---|---|---|---|---|---|
| tr3 | heal | qwen2.5-7b |
5.187 | qwen-research (non-commercial) | see dir |
| tt3 | heal | llama2-7b |
4.021 | llama2 | see dir |
| tt3 | heal | qwen2.5-3b |
2.388 | qwen-research (non-commercial) | see dir |
| tt3 | unheal | llama2-7b |
4.021 | llama2 | — |
| tt3 | unheal | qwen2.5-3b |
2.388 | qwen-research (non-commercial) | — |
| tr3 | heal | llama2-7b |
4.023 | llama2 | MMLU 43.21* |
| tr3 | heal | llama2-13b |
7.740 | llama2 | see dir |
| tr3 | heal | qwen2.5-3b |
2.078 | qwen-research (non-commercial) | see dir |
| tr3 | unheal | llama2-13b |
7.740 | llama2 | — |
| tr3 | unheal | llama2-7b |
(off-cluster archive) | llama2 | — |
| svd_llm | heal | llama2-7b |
4.022 | llama2 | MMLU 39.47 |
| svd_llm | heal | llama2-13b |
7.738 | llama2 | MMLU 46.54 |
| svd_llm | unheal | llama2-7b |
4.022 | llama2 | — |
| svd_llm | unheal | llama2-13b |
7.738 | llama2 | — |
| svd_llm_v2 ‡ | heal | llama2-7b |
4.023 | llama2 | MMLU 40.22 |
| svd_llm_v2 ‡ | heal | llama2-13b |
7.740 | llama2 | MMLU 44.96 |
| svd_llm_v2 ‡ | unheal | llama2-7b |
4.023 | llama2 | — |
| svd_llm_v2 ‡ | unheal | llama2-13b |
7.740 | llama2 | — |
| asvd_llm | heal | llama2-7b |
4.023 | llama2 | MMLU 35.09 |
| asvd_llm | heal | llama2-13b |
7.736 | llama2 | MMLU 42.46 |
| asvd_llm | unheal | llama2-7b |
4.023 | llama2 | — |
| asvd_llm | unheal | llama2-13b |
7.736 | llama2 | — |
* full-benchmark lm-eval 0.4.12 result; SVD probe columns are limit-40 healing probes here — each dir's README/provenance carries the precise protocol.
‡ Official SVD-LLM V2 code was never released; svd_llm_v2 artifacts are our faithful
implementation of the published algorithms (per-layer ranks document the allocation).
⚠️ These are NOT standard transformers checkpoints
tr3/** stores ring cores (load with each dir's tensor_formats.py); svd_llm*/** stores
low-rank factor pairs (load with each dir's lowrank_linear.py). Weight files are
uniformly named state_dict_fp16.pt. from_pretrained() will not work on this repo.
Benchmark results
All numbers below were produced by us under a single locked protocol: lm-eval-harness 0.4.12, fp16, seed 1234; MMLU 5-shot, HellaSwag 10-shot, everything else 0-shot; full benchmarks, no subsampling. Every baseline was re-run at the same parameter budget and the same 60M-token KD recipe as our model rather than quoted from its paper, so the rows within a block are directly comparable.
Perplexity is token-level over fixed non-overlapping 2048-token windows, following SVD-LLM's evaluater.py. It is not lm-eval's word-level perplexity; the two differ by the exponent N_tok/N_word (1.4156 for LLaMA-2 on WikiText-2) and must never be placed in the same column.
-- marks a cell not yet measured; it is a placeholder, not a zero or a failure.
Perplexity (lower is better)
| Model | Method | Params (B) | WikiText-2 | C4 |
|---|---|---|---|---|
| LLaMA-2-7B-chat | dense FP16 | 6.738 | 6.94 | 9.48 |
| TR-3 (ours) | 4.023 | 8.34 | 11.10 | |
| SVD-LLM | 4.022 | 11.49 | 15.09 | |
| SVD-LLM V2 ‡ | 4.023 | 11.03 | 14.73 | |
| ASVD | 4.023 | 11.22 | 14.18 | |
| TT-3 (open chain) | 4.021 | 12.52 | 15.09 | |
| LLaMA-2-13B-chat | dense FP16 | 13.016 | 6.11 | 8.52 |
| TR-3 (ours) | 7.740 | 6.98 | 9.65 | |
| SVD-LLM | 7.738 | 10.15 | 13.46 | |
| SVD-LLM V2 ‡ | 7.740 | 9.72 | 13.09 | |
| ASVD | 7.736 | 9.13 | 11.88 | |
| LLaMA-2-7B (base) | dense FP16 | 6.738 | 5.47 | 7.29 |
| TR-3 (ours) | 4.023 | 7.28 | 9.58 | |
| SVD-LLM | 4.022 | 10.17 | -- | |
| SVD-LLM V2 ‡ | 4.023 | 9.73 | -- | |
| ASVD | 4.023 | 10.04 | 12.48 | |
| LLaMA-2-13B (base) | dense FP16 | 13.016 | 4.88 | 6.74 |
| TR-3 (ours) | 7.740 | 6.18 | 8.51 | |
| ASVD | 7.746 | 7.97 | 10.44 | |
| SVD-LLM | 7.738 | 9.14 | -- | |
| SVD-LLM V2 ‡ | 7.740 | 8.74 | -- | |
| Qwen2.5-3B | dense FP16 | 3.086 | 8.56 | -- |
| TR-3 (ours) | 2.078 | 11.75 | -- | |
| TT-3 (open chain) | 2.077 | -- | -- | |
| Matrix low-rank † | 2.077 | -- | -- | |
| Qwen2.5-7B | dense FP16 | 7.616 | 7.46 | -- |
| TR-3 (ours) | 5.187 | 9.51 | -- |
Zero-shot accuracy (%)
| Model | Method | BoolQ | ARC-C | ARC-E | LAMBADA | PIQA | Winogrande | MathQA | Avg |
|---|---|---|---|---|---|---|---|---|---|
| LLaMA-2-7B-chat | dense FP16 | 80.58 | 43.26 | 68.39 | 69.82 | 77.04 | 68.35 | 28.48 | 62.27 |
| TR-3 (ours) | 77.09 | 38.48 | 65.11 | 66.52 | 75.52 | 66.30 | 27.50 | 59.50 | |
| SVD-LLM | 67.98 | 32.76 | 57.20 | 56.76 | 71.55 | 62.19 | 25.49 | 53.42 | |
| SVD-LLM V2 ‡ | 68.90 | 34.30 | 59.30 | 57.29 | 71.65 | 61.25 | 25.66 | 54.05 | |
| ASVD | 69.69 | 33.28 | 57.03 | 55.15 | 71.33 | 60.14 | 24.56 | 53.03 | |
| TT-3 (open chain) | 63.67 | 32.68 | 52.06 | 55.77 | 71.22 | 58.80 | 25.23 | 51.35 | |
| LLaMA-2-13B-chat | dense FP16 | 81.96 | 49.91 | 73.40 | 72.79 | 79.33 | 72.85 | 32.86 | 66.16 |
| TR-3 (ours) | 80.52 | 43.86 | 70.71 | 70.74 | 76.82 | 70.01 | 29.92 | 63.22 | |
| SVD-LLM | 73.64 | 35.41 | 61.03 | 60.59 | 72.74 | 64.48 | 25.93 | 56.26 | |
| SVD-LLM V2 ‡ | 75.14 | 36.35 | 61.99 | 62.47 | 73.34 | 66.14 | 27.14 | 57.51 | |
| ASVD | 74.89 | 36.95 | 64.60 | 61.60 | 74.05 | 65.19 | 26.37 | 57.66 | |
| LLaMA-2-7B (base) | dense FP16 | 79.33 | 45.05 | 73.86 | 73.65 | 78.73 | 69.46 | 28.41 | 64.07 |
| TR-3 (ours) | 70.95 | 39.16 | 65.24 | 68.02 | 75.35 | 64.48 | 25.66 | 58.41 | |
| SVD-LLM | 68.62 | 33.19 | 58.71 | 55.58 | 71.16 | 61.33 | 24.19 | 53.25 | |
| SVD-LLM V2 ‡ | 68.53 | 33.96 | 60.06 | 56.49 | 72.31 | 62.83 | 25.59 | 54.25 | |
| ASVD | 59.54 | 32.17 | 57.83 | 57.83 | 71.49 | 60.14 | 23.92 | 51.85 | |
| LLaMA-2-13B (base) | dense FP16 | 82.11 | 48.81 | 76.47 | 76.54 | 80.25 | 72.45 | 31.46 | 66.87 |
| TR-3 (ours) | 76.21 | 44.11 | 72.43 | 71.74 | 78.51 | 68.67 | 29.65 | 63.05 | |
| ASVD | 74.19 | 38.74 | 68.14 | 66.10 | 75.73 | 65.90 | 25.36 | 59.17 | |
| SVD-LLM | 70.34 | 36.01 | 64.94 | 60.94 | 73.78 | 65.82 | 24.69 | 56.64 | |
| SVD-LLM V2 ‡ | 71.13 | 38.14 | 67.30 | 61.67 | 73.83 | 65.67 | 25.66 | 57.63 | |
| Qwen2.5-3B | dense FP16 | 80.09 | 48.04 | 72.90 | 66.12 | 78.29 | 69.38 | 35.61 | 64.35 |
| TR-3 (ours) | 76.39 | 43.17 | 69.36 | 60.51 | 75.24 | 64.09 | 34.17 | 60.42 | |
| TT-3 (open chain) | 66.88 | 31.14 | 56.19 | 49.16 | 70.40 | 59.12 | 27.64 | 51.50 | |
| Matrix low-rank † | 71.35 | 36.43 | 59.55 | 50.51 | 72.14 | 61.88 | 28.41 | 54.33 | |
| Qwen2.5-7B | dense FP16 | 86.42 | 54.86 | 80.93 | 69.77 | 80.36 | 70.56 | 40.37 | 69.04 |
| TR-3 (ours) | 84.68 | 51.11 | 73.82 | 64.84 | 77.64 | 70.24 | 42.01 | 66.33 |
Few-shot / generative (%)
| Model | Method | MMLU (5s) | HellaSwag (10s) | TriviaQA (5s) | GSM8K (5s) |
|---|---|---|---|---|---|
| LLaMA-2-7B-chat | dense FP16 | 47.22 | 78.63 | 57.18 | 22.59 |
| TR-3 (ours) | 43.21 | 70.75 | 36.93 | 9.40 | |
| SVD-LLM | 39.47 | 57.65 | 16.18 | 2.05 | |
| SVD-LLM V2 ‡ | 40.26 | 58.82 | 18.62 | 0.68 | |
| ASVD | 34.65 | 59.46 | 18.02 | 2.27 | |
| TT-3 (open chain) | 34.25 | 56.38 | 14.48 | 0.76 | |
| LLaMA-2-13B-chat | dense FP16 | 53.55 | 82.07 | 65.20 | 35.25 |
| TR-3 (ours) | 49.69 | 76.46 | 47.81 | 24.11 | |
| SVD-LLM | 44.18 | 63.41 | 21.57 | 0.15 | |
| SVD-LLM V2 ‡ | 44.77 | 64.99 | 24.52 | 0.15 | |
| ASVD | 41.43 | 68.01 | 29.71 | 7.51 | |
| LLaMA-2-7B (base) | dense FP16 | 45.97 | 78.95 | 64.39 | 13.12 |
| TR-3 (ours) | 39.49 | 71.11 | 39.75 | 6.52 | |
| SVD-LLM | 32.74 | 58.36 | 16.43 | 1.59 | |
| SVD-LLM V2 ‡ | 34.33 | 59.36 | 19.34 | 1.97 | |
| ASVD | 26.58 | 61.21 | 20.69 | 1.29 | |
| LLaMA-2-13B (base) | dense FP16 | 55.54 | 82.14 | 70.69 | 23.28 |
| TR-3 (ours) | 50.46 | 77.04 | 50.96 | 14.33 | |
| ASVD | 42.44 | 70.15 | 35.50 | 5.23 | |
| SVD-LLM | 42.70 | 63.80 | 21.87 | 4.25 | |
| SVD-LLM V2 ‡ | 44.67 | 65.00 | 24.49 | 4.93 | |
| Qwen2.5-3B | dense FP16 | 66.42 | 75.26 | 45.78 | 10.31 |
| TR-3 (ours) | 56.02 | 66.39 | 27.61 | 41.32 | |
| TT-3 (open chain) | 41.97 | 51.65 | 11.86 | 9.78 | |
| Matrix low-rank † | 45.44 | 57.16 | 11.51 | 15.85 | |
| Qwen2.5-7B | dense FP16 | 74.27 | 81.37 | 53.17 | 75.74 |
| TR-3 (ours) | 65.43 | 75.20 | 40.79 | 66.34 |
† Matrix low-rank on Qwen2.5-3B is our own internal matrix-factorisation ablation, not a published method.
‡ Official SVD-LLM V2 code was never released; these are our faithful implementation of the published algorithms. Its close agreement with the official SVD-LLM v1 rows (within ~1.5 pp on every zero-shot task) is the evidence that the re-implementation is sound.
GSM8K on Qwen2.5 uses lm-eval's strict-match filter, which is fragile to output formatting on this model family. Under strict-match the dense Qwen2.5-3B teacher scores 10.31 while its compressed student scores 41.32; under flexible-extract the same teacher scores 63.46. The Qwen GSM8K column should therefore not be read as a capability ranking. LLaMA-2 rows are unaffected (strict and flexible agree to <0.2 pp on the dense teachers).