mine:o1-nano

A 124M-parameter GPT-2-style language model, pre-trained from scratch in JAX/Flax NNX on a TPU v5e-8, then supervised fine-tuned in three rounds.

Built by MineAI Technology, Islamabad, Pakistan: a small, independent team building a sovereign foundation model from the ground up rather than fine-tuning an existing base.


📄 Technical paper	10.5281/zenodo.21993150
🌐 Platform	getmineai.net
👤 Author	Bilal Mehtab, ORCID 0009-0000-3085-8559
📜 License	Apache 2.0

Honesty note: This card states benchmark numbers and known failure modes plainly, including the ones that don't look good. Read Limitations before deciding whether this model fits your use case.

What this model is (and is not)
Model summary
Files in this repository
Quick start
Chat format and recommended decoding
Training
Evaluation
SFT progression
Limitations & known failure modes
Intended use
Bias, risks, and safety
Source code status
Reproducing the evaluation
License
Links & citation

What this model is (and is not)

It is:

Proof that a small team can run the full stack end to end: data pipeline, from-scratch pre-training on TPU, staged fine-tuning, structured evaluation, and CPU serving.
A transparent baseline for other 124M-parameter GPT-2-style models, with every measured number and failure mode published.
A research and educational reference for from-scratch small-LM training in JAX/Flax NNX.

It is not:

A state-of-the-art model. It sits behind compressed and distilled GPT-2 variants on zero-shot WikiText-103 perplexity (see Evaluation).
Ready for open-ended, free-text chat deployment.
A reasoning model. The "o1" in the name is MineAI's own model-line numbering (MineAI Technology, first model, nano size); it has no relation to OpenAI's o1 series.

Model summary


Architecture	GPT-2 Small style, decoder-only Transformer, pre-norm
Parameters	124M
Layers	12
Attention heads	12 (64 dim each)
Hidden size	768
Feed-forward size	3072 (GELU)
Context length	1024 tokens
Vocabulary	50,259 tokens (GPT-2 BPE base of 50,257 plus 2 custom special tokens)
Special tokens	`<\|user\|>`, `<\|assistant\|>`
Embeddings	Tied input/output
Framework	JAX + Flax NNX + Optax
Training hardware	Kaggle TPU v5e-8
Inference hardware	AWS t3.small (CPU), Flax NNX decode/cache mode

Each block is: masked multi-head self-attention → residual → layer norm → feed-forward (768→3072→768, GELU) → residual, repeated 12 times.

Detailed block diagram (attention + FFN internals)

Causal (masked) self-attention illustration

Files in this repository

File / folder	Approx. size	Use it for
`model.onnx`	535 MB	Full-precision (fp32) ONNX export. Reference and any ONNX runtime
`model.fp16.onnx`	335 MB	Half-precision ONNX. Smaller, near-identical outputs
`model.quant.onnx`	134 MB	Quantized ONNX. Fastest on CPU; the hardware class the live demo runs on
`ocdbt.process_0/`, `array_metadatas/`, `d/`, `_METADATA`, `_CHECKPOINT_METADATA`, `_sharding`, `manifest.ocdbt`	~1.4 GB	Orbax checkpoint for JAX / Flax NNX. Use to continue training or fine-tune
`assets/`	~1.5 MB	Architecture and evaluation figures used in this card
`evaluation_reports/`	~70 KB	Full, unedited SFT and decoding evaluation reports
`LICENSE`		Apache License 2.0

Which one should I use?

Goal	Use
Run on CPU, lowest latency	`model.quant.onnx`
Run on CPU, closest to original outputs	`model.fp16.onnx`
Reference or debugging	`model.onnx`
Continue training or fine-tune in JAX	Orbax checkpoint

Quantization can shift outputs slightly. If exact behavior matters, compare against model.fp16.onnx.

Quick start

ONNX (CPU)

pip install onnxruntime transformers numpy huggingface_hub

import numpy as np
import onnxruntime as ort
from huggingface_hub import hf_hub_download
from transformers import GPT2TokenizerFast

REPO = "MineAITechnology/mine-o1-nano"

# GPT-2 BPE plus the two custom chat markers
tok = GPT2TokenizerFast.from_pretrained("gpt2")
tok.add_special_tokens({"additional_special_tokens": ["<|user|>", "<|assistant|>"]})

path = hf_hub_download(REPO, "model.quant.onnx")
sess = ort.InferenceSession(path, providers=["CPUExecutionProvider"])

# Check the exact input/output signature of the export before writing a loop
print([(i.name, i.shape, i.type) for i in sess.get_inputs()])
print([(o.name, o.shape, o.type) for o in sess.get_outputs()])

prompt = "<|user|> Who are you? <|assistant|>"
ids = np.array([tok.encode(prompt)], dtype=np.int64)
logits = sess.run(None, {sess.get_inputs()[0].name: ids})[0]
print(repr(tok.decode([int(logits[0, -1].argmax())])))

The snippet performs one greedy step to confirm the export loads and runs. For real generation, loop over tokens with the decoding settings below. Inputs and outputs depend on how the ONNX graph was exported (with or without a KV cache or attention mask), so check the printed signature first.

JAX / Flax NNX (research)

Download the Orbax checkpoint and restore it with Orbax into a Flax NNX GPT-2-style module matching the model summary dimensions. The training and model-definition code is not published (see Source code status), so you need to recreate the module with the same shapes (12 layers, 12 heads, 768 hidden, 3072 FFN, 1024 context, 50,259 vocab, tied embeddings).

from huggingface_hub import snapshot_download

local = snapshot_download(
    "MineAITechnology/mine-o1-nano",
    allow_patterns=["ocdbt.process_0/*", "array_metadatas/*", "d/*",
                    "_METADATA", "_CHECKPOINT_METADATA", "_sharding", "manifest.ocdbt"],
)
print(local)

Chat format and recommended decoding

The model was fine-tuned with two turn markers:

<|user|> {user message} <|assistant|>

Generation continues after <|assistant|>. Stop when the model emits a new <|user|> or hits your length limit.

Recommended decoding settings:

Parameter	Value
`temperature`	0.7
`top_k`	40
`top_p`	0.9
`repetition_penalty`	1.3

Do not use greedy decoding. The evaluation reports show that repetition loops, role inversion, and refusal-then-comply contradictions in early checkpoints were decoding artifacts, not training gaps. They were resolved by these sampling settings alone.

Training

Pre-training

Trained from scratch (random initialization, no warm start from any existing checkpoint) on a Kaggle TPU v5e-8 using JAX, Flax NNX, and Optax.
Vocabulary extended from the base GPT-2 BPE vocab (50,257) to 50,259 tokens to add <|user|> / <|assistant|> turn markers.

Hyperparameter	Value
Architecture	GPT-2
Dataset	OpenWebText
Batch size	64
Embedding dim	768
Feed-forward dim	3072
Initial learning rate	5e-4
Max steps	80,000
Attention heads	12
Transformer layers	12
Sequence length	1024
Weight decay	0.1
Final train loss	3.1932
Final validation loss	3.20313
Runtime	4h 57m 45s
Platform	Kaggle (TPU v5e-8)

Supervised fine-tuning

Three rounds (SFT1 → SFT2 → SFT3), each targeting specific behavioral gaps found through structured evaluation rather than informal spot checks:

Round	Focus
SFT1	Initial chat behavior and identity (checkpoint at step 3388)
SFT2	Identity, location and creator disclosure, arithmetic
SFT3	Targeted supplementary arithmetic data, final comprehensive evaluation

Evaluation

Language modeling (perplexity)

Evaluated on WikiText-2, WikiText-103, and LAMBADA using non-overlapping stride windows matching the original GPT-2 paper's methodology (Kaggle GPU T4x2).

WikiText-103 zero-shot perplexity comparison

The chart compares zero-shot WikiText-103 perplexity only, since that is the fairest like-for-like comparison. mine:o1-nano currently sits behind compressed and distilled GPT-2 variants (TQCompressedGPT2, KnGPT-2, Krony-PT). That is an honest reflection of being an early, from-scratch checkpoint rather than a compression of an already-trained larger model.

Not shown on the chart: GPT-2 and DistilGPT2. Their commonly cited WikiText-103 numbers (16.3 and 21.1) come from a fine-tuned setup, not zero-shot, so including them would misstate the gap. GPT-2's own zero-shot number from its original paper (Section 5.1) is 37.5.

Behavioral evaluation

Behavior was evaluated with fixed test sets (Set A / Set B, a 20-question arithmetic stress test, identity and creator-disclosure probes) and one unscripted out-of-distribution probe. See the next two sections and the full reports in evaluation_reports/.

SFT progression — measured, not claimed

Left: Set A/B scores across decoding experiments (R1: greedy → R2: temperature → R3: temperature + repetition penalty + nucleus sampling) and the SFT2 continued fine-tune. Most early gains came from decoding strategy, not retraining.

Right: Basic arithmetic accuracy on a fixed 20-question test, by SFT round. Decoding changes could not move this metric; it required a targeted data round (SFT3).

Key findings

Identity and branding: stable by SFT2. The model reliably identifies itself as mine:o1-nano from MineAI Technology. An early checkpoint hallucinated being "a UC Berkeley professor".
Repetition loops, role inversion, refusal-then-comply contradictions: confirmed decoding artifacts. Resolved by temperature=0.7, top_k=40, top_p=0.9, repetition_penalty=1.3.
Basic arithmetic: 0/20 → 1/20 → 19/20 across the three SFT rounds, via a targeted supplementary dataset. This was a genuine data-coverage gap.
Location and creator disclosure: 7/7 correct in SFT3 across varied phrasings.

Limitations & known failure modes

MineAI Technology's policy is to report benchmarks and limitations honestly rather than promotionally.

mine:o1-nano is NOT production-ready for open-ended, free-text deployment. It suits narrow, scripted use cases: an FAQ-style assistant with known prompt formats, or a widget with suggested prompts rather than open free-text chat.

On the scripted test suite it scored strongly (19/20 math, 7/7 identity, clean Set A/B). On the unscripted real-world probe using casual phrasing it hadn't seen in training, 4 of 10 exchanges failed:

Failure mode	Example
Sensitive-topic mishandling	"i got breakup with my girlfriend" → incoherent, non-empathetic, garbled response. No training data covers emotionally sensitive topics.
Math boundary errors	"what is 90+10" → answered 120 (should be 100), despite 19/20 accuracy in the core trained range. Suggests memorized number pairs rather than generalized addition.
Follow-up brittleness	"is he founder or CEO?" → fell back to a memorized identity string instead of answering.
Casual-phrasing deflection	"i want to know about ur owner & company?" → deflected, despite this being well-covered training territory in standard phrasing.

What is and isn't ready

✅ Ready	❌ Not ready
Identity, company and creator disclosure (even with novel phrasing)	Sensitive or emotional topic handling (no safety behavior trained)
Greetings and casual small talk	Arithmetic generalization at range boundaries or with casual phrasing
Scripted arithmetic within trained ranges and formats	Natural conversational follow-ups outside the trained prompt structure

As with any 124M-parameter model, these are known limitation classes at this scale and not specific to this training pipeline. They are stated plainly rather than around.

Intended use

Suitable

Narrow, scripted conversational interfaces (structured FAQ, guided-prompt widgets)
Research and educational reference for from-scratch small-LM training in JAX/Flax NNX
Baseline or comparison point for other 124M GPT-2-style models
Edge or CPU-only experiments where a tiny model is acceptable

Not recommended

Open-ended free-text chat deployment
Any use involving emotionally sensitive user input
Arbitrary arithmetic or precise calculation
Factual question answering where correctness matters
Production systems without a human fallback path

Bias, risks, and safety

Training data: pre-trained on OpenWebText, a web-scraped corpus. It inherits the biases, stereotypes, and factual errors common to web text, and the model can reproduce them.
No safety training: SFT covered identity, disclosure, and arithmetic. No refusal, safety, or crisis-handling behavior was trained. Do not expose this model to users in sensitive contexts.
Hallucination: at 124M parameters the model produces fluent but frequently incorrect statements. Do not use its output as a source of facts.
Identity claims are trained, not grounded: its statements about MineAI Technology are memorized from SFT data. It has no retrieval or live knowledge.
Recommended mitigation: keep a human in the loop, restrict to scripted prompts, and add input and output filtering appropriate to your use case.

Source code status

This repository releases the trained weights (Orbax checkpoint and ONNX exports), the model card, and the evaluation reports.

The training pipeline, data-processing code, and infrastructure are not included in this release. The architecture is standard GPT-2 Small and fully specified in the model summary, so the model can be re-instantiated from the checkpoint without them.

Reproducing the evaluation

Perplexity was measured with non-overlapping stride windows following the GPT-2 paper's protocol, on WikiText-2, WikiText-103, and LAMBADA, on Kaggle GPU T4x2. The full method, decoding configurations, prompt sets, and raw results for every SFT round are in evaluation_reports/:

Report	Contents
`SFT1_Evaluation_Report.pdf`	Initial SFT checkpoint (step 3388), Set A and B
`SFT_Decoding_Report2.pdf`	Greedy vs. temperature sampling
`SFT_Decoding_Report3.docx`	Adding repetition penalty and nucleus sampling
`SFT2_Main_Report.docx`	Continued fine-tune: identity, location and creator disclosure, arithmetic
`SFT3_Final_Report.docx`	Final report and real-world out-of-distribution probe

License

Released under the Apache License 2.0. You may use, modify, and distribute the weights commercially, provided you keep the license and attribution notices and state significant changes. Apache 2.0 also includes an express patent grant.

The GPT-2 BPE tokenizer vocabulary originates from OpenAI's GPT-2 release (MIT licensed). The pre-training dataset was OpenWebText; check its terms if data provenance matters for your use case.

Links & citation

Live demo: nano.getmineai.net
Platform: getmineai.net
Technical paper: Zenodo DOI 10.5281/zenodo.21993150
Author: Bilal Mehtab, ORCID 0009-0000-3085-8559

If you use this model or reference these results, please cite the Zenodo record:

@misc{mehtab2026mineo1nano,
  author    = {Mehtab, Bilal},
  title     = {mine:o1-nano: A 124M-parameter GPT-2-style language model pre-trained from scratch in JAX/Flax NNX},
  year      = {2026},
  publisher = {Zenodo},
  doi       = {10.5281/zenodo.21993150},
  url       = {https://doi.org/10.5281/zenodo.21993150}
}

MineAI Technology, Islamabad, Pakistan

Downloads last month: 89

MineAITechnology
/

mine-o1-nano