XAbys-tiny β€” pipeline-proof checkpoint (NOT the flagship)

Honest summary first: this is a 1.45M-parameter demo model trained from random initialization on 9,226 tokens of sample text. It proves the XAbys training pipeline works end to end (tokenizer β†’ data β†’ training β†’ checkpoint β†’ eval β†’ generate). It cannot reason, code, or answer questions. Its generations are gibberish. The real flagship XAbys will come from fine-tuning an open-weight base model (Qwen 3 / Llama 4 / Nemotron) on Colab with Unsloth β€” that work has not happened yet. Do not mistake this checkpoint for that model.

Model details

  • Architecture: decoder-only Transformer, written from scratch (RMSNorm, RoPE, SwiGLU, causal attention, tied embeddings)
  • Config: 2 layers, 4 heads, d_model 128, vocab 8000, block size 512
  • Parameters: 1,450,624 (train.py count)
  • Tokenizer: ByteLevel BPE, trained on the sample corpus (actual vocab 1,788)
  • Training: from random initialization, AdamW, cosine LR, 3 epochs, 51 steps, batch 4, block 128, CPU only, ~17.5 seconds
  • Data: 573 raw lines β†’ 550 kept, 9,226 tokens (train 8,765 / val 461) β€” original sample text (English, Roman Hinglish, Python, maths)

Measured results (14 Nov 2026, CPU)

  • Train loss: 8.99 (uniform over 8000 vocab) β†’ 7.79 at step 50
  • Eval loss (best checkpoint, step 40): 7.8109, perplexity 2467.39
  • Held-out perplexity (sample_math.txt): 2814.04
  • Generation exact-match: 0 / 4 correct (0.0%)
  • Sample generation, prompt "Dukaan subah khulti hai" β†’ "TheTheTheTheThe."

Files

  • config.json β€” model config
  • tokenizer.json β€” BPE tokenizer
  • xabys-tiny-model.pt β€” model weights only (5.8 MB, state dict)
  • xabys-tiny-best.pt β€” full training checkpoint (17.4 MB: model + optimizer + scheduler + config + step + best loss)

How to load

import json
import torch
from xabys.model import XAbysConfig, XAbysForCausalLM
from xabys.tokenizer import load_tokenizer

cfg = XAbysConfig(**json.load(open("config.json")))
model = XAbysForCausalLM(cfg)
model.load_state_dict(torch.load("xabys-tiny-model.pt", map_location="cpu"))
model.eval()
tok = load_tokenizer("tokenizer.json")

Full source, training scripts, tests and docs: https://github.com/hexray1 β€” see the XAbys project (workspace/xabys).

Limitations

  • 1.5M params on 9k tokens cannot learn language, reasoning or code.
  • No vision, audio, video, tools or agent capability in this checkpoint.
  • Generation repeats tokens ("AAAA.", "TheTheThe...").
  • The 7B config in the source is config-only and was never trained here.
Downloads last month
37
Inference Providers NEW
This model isn't deployed by any Inference Provider. πŸ™‹ Ask for provider support