athnlp2025_tokenization / requirements.txt
gpantaz
Add application file
79660f1
Raw History Blame Contribute Delete
137 Bytes
gradio>=4.38.1
transformers>4.40.0
sentencepiece
tiktoken
icetk
torch
nltk
boto3
protobuf==4.25.3
ai2-olmo
ipadic
fugashi
datasets
loguru