braindecode/duin-pretrained

Du-IN — braindecode weights

braindecode-native re-host of the official Du-IN pretrained checkpoints (Zheng et al., NeurIPS 2024), converted so they load directly with braindecode.models.DuIN.

Du-IN decodes speech (61 Chinese words) from sEEG. Each subject's signal (10 selected bipolar channels, 1000 Hz) goes through a subject-specific spatial projection, a patch-wise convolutional tokenizer (one token per 100 ms) and an 8-layer Transformer encoder. The encoder is pretrained per subject by masked modelling of discrete codes learned by a VQ-VAE (mae); the VQ-VAE encoder itself (vqvae) is the other released stage.

Provenance

Original code	https://github.com/liulab-repository/Du-IN (MIT)
Original weights	https://huggingface.co/datasets/liulab-repository/Du-IN (pretrains/duin/<subject>/<stage>/model/checkpoint-399.pth, CC BY 4.0)
Paper	Zheng et al. (2024), Du-IN: Discrete units-guided mask modeling for decoding speech from intracranial neural signals, NeurIPS 2024, arXiv:2405.11459

These weights are a format conversion only of the authors' checkpoints, with no re-training. For each of the 24 checkpoints:

• the 183 encoder tensors are carried over unchanged (182 bit-for-bit, plus the subject layer W (160, 1), reshaped to spatial_projection.weight (16, 10));
• the ported encoder reproduces the upstream duin_cls encoder output on the same input to < 1e-5 (max abs diff 7.2e-6 over all 24);
• DuIN.from_pretrained on the uploaded file gives the same state dict and outputs as loading the original .pth.

The pretraining-only parts (mask_emb, VQ codebook, MAE token head) are dropped. The classification head (head_hidden, final_layer, 61 outputs) is randomly initialised and must be fine-tuned. Per-file source sha256 are in conversion_report.json.

Files

path	content
config.json	shared DuIN config (n_chans=10, n_times=3000, sfreq=1000, n_outputs=61)
subj-XXX/mae/model.safetensors	MAE-pretrained encoder of subject XXX (001 to 012), the paper's Du-IN
subj-XXX/vqvae/model.safetensors	VQ-VAE-stage encoder of subject XXX (the paper's "Du-IN (vqvae)")
model.safetensors	default = subj-001/mae

Every checkpoint is subject-specific: it was trained on that subject's 10 channels and only fits that montage, in the channel order of the authors' dataset. They are not a cross-subject foundation model.

Training data

The data that shaped these weights. The ids are the datasets: field of this card, which braindecode reads into model.card_data.datasets when you call from_pretrained (MOABB class names for MOABB datasets, OpenNeuro / NEMAR accessions, Hub ids otherwise). nm/on ids are datasets on NEMAR (on = NEMAR's copy of the OpenNeuro dataset with the same number). Rows marked private: are data the authors did not release; they are not in datasets:.

datasets: id	Dataset	Stage	Modality
private:Du-IN-Xuanwu-sEEG-other-tasks	Unreleased sEEG recordings of the same 12 Du-IN subjects from other cognitive tasks (~12 h per subject; Xuanwu Hospital, Beijing)	pretraining	iEEG
nm000365	Du-IN Chinese 61-word reading sEEG dataset (12 subjects, Xuanwu Hospital; preprocessed ~3 h task recordings per subject), 'seeg.he2023xuanwu'	pretraining	iEEG

• Per-subject: each of the 24 checkpoints is trained only on ONE subject's sEEG (10 selected bipolar channels). vqvae stage = VQ-VAE; mae stage = masked code modelling initialised with that subject's VQ-VAE codex. Paper main setting uses all ~15 h of the subject (12 h other tasks + 3 h word-reading task); whether the released checkpoints include the 3 h task recordings is not stated.
• Incomplete list. Same dataset family for all 24 checkpoints, per subject (subj-001..012). The ~12 h of non-task recordings are unreleased; the authors do not say whether the released checkpoint-399 files were trained with or without the 3 h word-reading recordings (paper Table 3: 'mae w/ DD' 15 h vs 'mae w/o DD' 12 h). A dataset missing here is therefore not proof that these weights never saw it.
• Sources: arXiv:2405.11459 Section 4.1; arXiv:2405.11459 Section 4.2; https://github.com/liulab-repository/Du-IN README, Stage 0; liulab-repository/Du-IN pretrains/duin/001/mae/save/params (pickled DotDict; value of key 'dataset').

Usage

from braindecode.models import DuIN

# MAE-pretrained Du-IN of subject 002; the 61-word head is re-initialised.
model = DuIN.from_pretrained(
    "braindecode/duin-pretrained",
    filename="subj-002/mae/model.safetensors",
)
# Pass n_outputs=... for another task (the head is re-created).

# input: (batch, 10, 3000) — 3 s at 1000 Hz, z-scored per channel
logits = model(x)

The reference head ends with a sigmoid and is trained with F.cross_entropy(torch.sigmoid(logits), target); braindecode returns logits. Use that loss to match the reference training.

Replication (braindecode PR #1298)

Fine-tuning on the 61-word task with the authors' run_cls.py protocol (200 epochs, AdamW, sigmoid cross-entropy, 6 seeds × 12 subjects, paper-time data revision 056a87b7, mean ± SE over subjects). There are two independent runs:

• A: the authors' run_cls.train() loop run with both the original model and the port, starting from the same weights (Gaudi2 / CPU);
• B: a standalone braindecode re-implementation of the same loop with the port, on CUDA, with the run order drawn at random per seed (see below).

	paper (Table 2)	A: authors' code	A: port	B: port
Du-IN (mae)	62.70 ± 4.69	60.52 ± 4.58	60.16 ± 4.62	59.37 ± 4.69
Du-IN (vqvae)	58.24 ± 4.83	55.10 ± 4.51	55.00 ± 4.51	54.02 ± 4.57
scratch	56.29 ± 5.20	54.06 ± 5.12	53.73 ± 5.13	53.77 ± 5.10

The port matches the authors' code to within 0.4 points. Training the port on plain logits (cross_entropy(logits)) instead of the sigmoid loss gives 55.6 for mae in A and 54.5 in B.

Split caveat. The authors' split concatenates a subject's runs in os.listdir order and puts the first 80 % of each word's trials in the training set. Validation and test therefore come almost entirely from the last run listed, and that run depends on the filesystem. If it is the chronologically last session (run6), accuracy drops sharply. On subject 002 (scratch, seed 42), holding out run6 gives 43–45 %, while holding out any other run gives 71–78 %. A sorted run order underestimates the paper by about 8 points on average.

License

Weights: CC BY 4.0, as released by the authors; cite the original work. braindecode code: BSD-3-Clause; original Du-IN code: MIT.

Citation

@inproceedings{zheng2024duin,
  title={Du-IN: Discrete units-guided mask modeling for decoding speech from intracranial neural signals},
  author={Zheng, Hui and Wang, Hai-Teng and Jiang, Wei-Bang and Chen, Zhong-Tao and He, Li and Lin, Pei-Yang and Wei, Peng-Hu and Zhao, Guo-Guang and Liu, Yun-Zhe},
  booktitle={Advances in Neural Information Processing Systems},
  volume={37},
  year={2024}
}

Cite Braindecode

If you use this braindecode port in scientific work, cite the software on Zenodo, doi:10.5281/zenodo.17699192, and the original model and datasets cited above.

@software{braindecode,
  author = {Aristimunha, Bruno and
            Guetschel, Pierre and
            Wimpff, Martin and
            Gemein, Lukas and
            Rommel, Cedric and
            Banville, Hubert and
            Sliwowski, Maciej and
            Wilson, Daniel and
            Brandt, Simon and
            Gnassounou, Théo and
            Paillard, Joseph and
            {Junqueira Lopes}, Bruna and
            Sedlar, Sara and
            Moreau, Thomas and
            Chevallier, Sylvain and
            Gramfort, Alexandre and
            Schirrmeister, Robin Tibor},
  title = {Braindecode: toolbox for decoding raw electrophysiological brain data
           with deep learning models},
  url = {https://github.com/braindecode/braindecode},
  doi = {10.5281/zenodo.17699192},
  publisher = {Zenodo},
  license = {BSD-3-Clause},
}