babylm_english_2026

A GPT-2-style causal language model trained on English, built for the BabyLM 2026 shared task. This repo contains 6 model checkpoints comparing two training strategies — random data ordering vs. curriculum learning — each run with 3 different random seeds, enabling statistically robust comparison.

This model serves as the English baseline in a cross-lingual study alongside babylm_hindi_2026 and babylm_telugu_2026, which are trained on machine-translated versions of the same corpus.

---

Repo Structure

babylm_english_2026/
├── random_seed1/        # Random data order, seed 1
├── random_seed2/        # Random data order, seed 2
├── random_seed3/        # Random data order, seed 3
├── curriculum_seed1/    # Curriculum learning, seed 1
├── curriculum_seed2/    # Curriculum learning, seed 2
└── curriculum_seed3/    # Curriculum learning, seed 3

Each subfolder is a self-contained GPT-2 checkpoint (Safetensors format) loadable independently via from_pretrained.

---

Experimental Design

The core research question is: does curriculum learning improve sample-efficient pretraining for English, and does the effect transfer to Hindi and Telugu?

Condition	Data Order	Seeds
random_seed*	Shuffled uniformly at random	1, 2, 3
curriculum_seed*	Simple → complex, ordered by sentence length (bytes per line)	1, 2, 3

The curriculum strategy orders training examples from shortest to longest sentence (by byte count), so the model encounters simpler linguistic structures before more complex ones. This mirrors cognitively plausible language acquisition and follows the byte-ranked curriculum approach explored in prior BabyLM submissions.

Running 3 seeds per condition allows variance estimation and guards against seed-specific artefacts — a known issue in low-data training regimes.

---

Model Details

Property	Value
Architecture	GPT-2 (decoder-only Transformer)
Language	English (en)
License	MIT
Training framework	Hugging Face Transformers
Model format	Safetensors

---

Training Data

Trained directly on the original BabyLM-community/BabyLM-2026-Strict corpus — no translation step, unlike the Hindi and Telugu counterparts.

Split	Rows	Size
Train	11.6M	338.9 MB
Val	—	via BabyLM-dev
Test	—	via BabyLM-Test

• Total tokens: ~100M
• License: MIT
• Content: Developmentally plausible English text (child-directed speech, educational content, subtitles, and similar low-complexity sources)

---

How to Use

Load any individual checkpoint by pointing from_pretrained at the subfolder:

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

# Choose any of the 6 checkpoints
checkpoint = "pulipakav-1/babylm_english_2026/curriculum_seed1"

tokenizer = AutoTokenizer.from_pretrained(checkpoint)
model = AutoModelForCausalLM.from_pretrained(checkpoint)

prompt = "Once upon a time, a child learned to speak"
inputs = tokenizer(prompt, return_tensors="pt")

with torch.no_grad():
    outputs = model.generate(
        **inputs,
        max_new_tokens=50,
        do_sample=True,
        temperature=0.8,
        top_p=0.95,
    )

print(tokenizer.decode(outputs[0], skip_special_tokens=True))

Comparing all runs

from transformers import AutoTokenizer, AutoModelForCausalLM
import torch

repo = "pulipakav-1/babylm_english_2026"
checkpoints = [
    "random_seed1", "random_seed2", "random_seed3",
    "curriculum_seed1", "curriculum_seed2", "curriculum_seed3",
]

text = "Children learn language by listening to the people around them."

for ckpt in checkpoints:
    path = f"{repo}/{ckpt}"
    tokenizer = AutoTokenizer.from_pretrained(path)
    model = AutoModelForCausalLM.from_pretrained(path)
    inputs = tokenizer(text, return_tensors="pt")
    with torch.no_grad():
        loss = model(**inputs, labels=inputs["input_ids"]).loss
    print(f"{ckpt:25s}  perplexity: {torch.exp(loss).item():.2f}")

---

Evaluation

⚠️ Results will be updated as evaluations are completed.

Planned metrics, reported as mean ± std across the 3 seeds per condition:

Metric	random (mean ± std)	curriculum (mean ± std)
Test perplexity	—	—
BLiMP accuracy	—	—

---

Cross-lingual Comparison

This model is the English baseline in a three-language study. The same GPT-2 architecture and curriculum strategy are applied across all three languages, with the only variable being the training data:

Model	Language	Training data
babylm_english_2026	English	Original BabyLM 2026 Strict corpus
babylm_hindi_2026	Hindi	IndicTrans2 translation of the same corpus
babylm_telugu_2026	Telugu	IndicTrans2 translation of the same corpus

---

Intended Use

• English baseline for cross-lingual curriculum learning experiments
• Research into data-efficient language modeling under BabyLM constraints
• Cognitive science research on language acquisition
• Reproducibility and benchmarking for BabyLM 2026

Out-of-scope: Production use, sensitive applications, or tasks requiring high factual accuracy. Outputs are unfiltered.

---

Limitations

• Limited to ~100M tokens — the model may struggle with rare vocabulary and complex syntax
• Curriculum ordering is by byte length, a surface-level proxy for linguistic complexity
• No RLHF or alignment fine-tuning

---

Citation

@misc{babylm_english_2026,
  author    = {pulipakav-1},
  title     = {babylm\_english\_2026: GPT-2 Language Models for English with Random and Curriculum Training},
  year      = {2026},
  publisher = {Hugging Face},
  url       = {https://huggingface.co/pulipakav-1/babylm_english_2026}
}

@misc{babylm2026,
  title  = {BabyLM Turns 4: Call for Papers for the 2026 BabyLM Workshop},
  author = {Choshen, Leshem and Cotterell, Ryan and Gul, Mustafa Omer and Jumelet, Jaap and Linzen, Tal and Mueller, Aaron and Salhan, Suchir and Shah, Raj Sanjay and Warstadt, Alex and Wilcox, Ethan Gotlieb},
  year   = {2026},
  url    = {https://arxiv.org/abs/2602.20092}
}

---

Contact

Maintained by @pulipakav-1.