SparseLLMs

The organization is founded by THUNLP and ModelBest with the help of IPADS, aimed at promoting the development of Sparse Large Language Models (SparseLLMs). By utilizing the sparsity of LLMs, we can significantly reduce the computational cost of inference. Currently, the organization is mainly focused on the ReLU-activated LLMs, which are converted from existing LLMs through fine-tuning.

The LLaMA series within it is provided by THUNLP and ModelBest. Additionally, IPADS contributed the Falcon model, and participation from other institutions is also welcomed.