In an era of information overload, recommendation systems have become an indispensable part of our lives. Recently, a collaborative effort between the University of Science and Technology of China (USTC) and Huawei has yielded a groundbreaking generative recommendation large model, successfully deployed on the domestically produced Ascend NPU for the first time. This development marks a significant step forward in the field of personalized recommendations, offering new possibilities for various applications.
The research team’s findings, recently shared publicly, delve into the evolution of recommendation paradigms, highlighting the potential of generative recommendation as the future trend, driven by scaling laws.
The Rise of Generative Recommendation Large Models
The development of recommendation systems has been steadily moving away from manual feature engineering and model structure design. Early systems relied on hand-crafted features and simple models due to limited computational resources. With the advent of deep learning, researchers focused on designing complex models to better capture user preferences and leverage GPU parallel computing. However, as deep learning capabilities reached their limits, feature engineering regained attention.
Inspired by the success of scaling laws in large language models, researchers in the recommendation field have begun exploring the potential of large models. Scaling laws describe the power-law relationship between model performance and key indicators such as parameter size, dataset size, and training resources. By increasing model depth and width, and combining them with massive datasets, recommendation effectiveness can be significantly improved. This approach has led to the emergence of recommendation large models.
Meta’s pioneering work in generative recommendation, exemplified by the HSTU framework, has demonstrated the potential of this approach, expanding recommendation parameters to the trillion level and achieving remarkable results. This success has validated the existence of scaling laws in the recommendation domain, sparking a surge in research on generative recommendation large models.
The USTC-Huawei team believes that generative recommendation large models are poised to disrupt the current recommendation system paradigm. Understanding which models truly possess scalability, identifying the reasons behind their successful application of scaling laws, and leveraging these principles to enhance recommendation effectiveness have become critical research areas.
Scalability Analysis of Generative Recommendation Large Models Based on Different Architectures
To evaluate the scalability of generative recommendation large models across different architectures, the team compared four Transformer-based architectures: HSTU, Llama, GPT, and SASRec. The analysis was conducted on three public datasets, examining performance under varying numbers of attention modules.
The results revealed that when model parameters were small, the architectures exhibited similar performance, with the optimal architecture varying depending on the dataset. However, as parameters increased, HSTU and Llama demonstrated significant performance improvements, while GPT and SASRec showed limited scalability. Despite GPT’s strong performance in other domains, it fell short of expectations in recommendation tasks. The team attributes this to the lack of recommendation-specific design in the GPT and SASRec architectures.
Key Findings and Future Directions
The USTC-Huawei collaboration has provided valuable insights into the development and deployment of generative recommendation large models. The successful deployment on the Ascend NPU demonstrates the feasibility of using domestically produced hardware for advanced recommendation systems. The research also highlights the importance of architecture design in achieving scalability and the potential of generative recommendation to revolutionize the field.
Looking ahead, the team envisions further research in the following areas:
- Exploring novel architectures: Investigating new architectures specifically designed for recommendation tasks to further enhance scalability and performance.
- Understanding the underlying mechanisms: Delving deeper into the mechanisms that drive scaling laws in recommendation models to optimize model design and training strategies.
- Addressing real-world challenges: Applying generative recommendation large models to complex real-world scenarios and addressing challenges such as data sparsity, cold start, and user privacy.
The release of this generative recommendation large model and the accompanying research findings mark a significant milestone in the evolution of recommendation systems. As research continues and technology advances, we can expect to see even more sophisticated and personalized recommendation experiences in the future.
References
(To be added with specific citations for HSTU, Llama, GPT, SASRec, and relevant academic papers, adhering to a consistent citation format such as APA or MLA.)
Views: 0