Xiaohongshu’s Hi Lab Open-Sources Text-Generating AI Model dots.llm1

Introduction:

In a significant move for the open-source AI community, Xiaohongshu’s Hi Lab has released Dots.LLM1, a mid-sized Mixture of Experts (MoE) text large language model. This release provides researchers and developers with a valuable tool for exploring and advancing the field of large language models. With its impressive performance and efficient training techniques, Dots.LLM1 is poised to make a significant contribution to the development of AI technologies.

What is Dots.LLM1?

Dots.LLM1 is a Mixture of Experts (MoE) text large language model developed and open-sourced by Xiaohongshu’s Hi Lab. The model boasts 142 billion parameters, with 14 billion parameters activated at any given time. This architecture allows for efficient scaling and specialization, enabling the model to handle a wide range of tasks effectively.

Key Features and Capabilities:

Dots.LLM1 offers a diverse set of capabilities, making it a versatile tool for various applications:

Multilingual Text Generation: The model excels at generating high-quality text in both Chinese and English, suitable for applications like writing assistance and content creation.
Complex Instruction Following: Dots.LLM1 can understand and execute intricate instructions, enabling it to perform specific tasks such as data organization and code generation.
Knowledge-Based Question Answering: The model provides accurate answers to knowledge-based questions, assisting users in accessing the information they need.
Mathematical and Code Reasoning: Dots.LLM1 possesses the ability to solve mathematical problems and generate simple code, showcasing its reasoning capabilities.
Multi-Turn Dialogue: The model supports engaging in multi-turn conversations, understanding context and providing appropriate responses for natural and fluid interactions.

Technical Details and Training:

Dots.LLM1 was pre-trained on a massive dataset of 11.2 trillion high-quality tokens. To optimize training efficiency, the Hi Lab team employed advanced techniques such as Interleaved 1F1B pipeline parallelism and Grouped GEMM optimization. These techniques significantly reduced training time and resource consumption.

The model’s architecture is based on the Mixture of Experts (MoE) approach, which allows for specialized learning and efficient scaling. This architecture enables Dots.LLM1 to achieve strong performance across a variety of tasks while maintaining a manageable computational footprint.

Performance and Benchmarking:

Through a carefully designed data processing pipeline and two-stage supervised fine-tuning, Dots.LLM1 demonstrates exceptional performance in general-purpose Chinese and English scenarios, as well as in specialized tasks such as mathematics and code. According to Hi Lab, the model exhibits strong competitiveness compared to other models like Qwen2.5-72B.

Open Source Contribution:

Hi Lab has released the checkpoints for each 1T token of the pre-training phase, as well as the Instruct model. This generous contribution provides the large model community with a rich foundation for research and development, fostering innovation and progress in the field.

Conclusion:

The release of Dots.LLM1 by Xiaohongshu’s Hi Lab marks a significant step forward for the open-source AI community. Its impressive performance, efficient training techniques, and comprehensive feature set make it a valuable resource for researchers and developers. As the AI landscape continues to evolve, Dots.LLM1 is poised to play a key role in shaping the future of large language models and their applications.

References: