The NeurIPS (Neural Information Processing Systems) conference, one of the most prestigious gatherings in the field of artificial intelligence and machine learning, is known for its rigorous peer-review process. However, this year, a particularly egregious reviewer comment has ignited a firestorm of criticism and reignited the debate surrounding the quality and integrity of the review process, especially in the face of surging submission numbers and the increasing reliance on AI-assisted reviewing.
The controversy centers around a review of a paper submitted to NeurIPS 2025. The reviewer, whose identity was initially unknown but later revealed to be evaluating the work of researchers using the Adam optimization algorithm, a widely used and fundamental technique in deep learning, wrote: Both architectures use the Adam optimizer. Who/what is ‘Adam’? I think this is a very serious typo that the authors should have removed before submission.
This comment, shared on X (formerly Twitter) by Yiping Lu, an assistant professor of Industrial Engineering and Management Sciences at Northwestern University and an alumnus of Peking University, quickly went viral, garnering hundreds of thousands of views and sparking widespread outrage within the AI community. The implication that a reviewer for a top-tier AI conference was unfamiliar with the Adam optimizer, a cornerstone of modern deep learning, was seen as deeply concerning and indicative of a broader problem with the quality control mechanisms of the conference.
The Viral Tweet and Community Backlash
Lu’s tweet, simply showcasing the reviewer’s comment, acted as a catalyst for a wave of criticism directed at the NeurIPS review process. The comment was perceived as not only ignorant but also disrespectful to the authors, suggesting a lack of basic understanding of the field. The absurdity of the situation was not lost on the AI community, with many expressing disbelief and frustration.
Dan Roy, a professor at the University of Toronto, weighed in on the controversy, stating bluntly, NeurIPS reviewing is complete garbage. This sentiment resonated with many researchers who have experienced similarly baffling or unhelpful reviews in the past. The incident served as a stark reminder of the potential for flawed reviews to unfairly impact the acceptance or rejection of valuable research.
The Pressure Cooker: Submission Volume vs. Review Quality
The Adam controversy has brought to the forefront a long-standing concern within the AI community: the growing disparity between the ever-increasing volume of submissions to top conferences like NeurIPS and the ability to maintain a consistently high standard of review quality.
NeurIPS, like other leading AI conferences such as ICML (International Conference on Machine Learning) and ICLR (International Conference on Learning Representations), has witnessed an exponential surge in submissions in recent years. This year, NeurIPS reportedly received close to 30,000 submissions, a staggering number that places immense pressure on the review process.
The sheer volume of papers makes it increasingly difficult to find qualified reviewers who have the time and expertise to thoroughly evaluate each submission. Reviewers, often academics themselves, are already burdened with their own research, teaching, and administrative responsibilities. The added burden of reviewing multiple papers for conferences can lead to rushed and superficial reviews, as evidenced by the Adam incident.
The Rise of AI-Assisted Reviewing: A Double-Edged Sword
In response to the overwhelming workload, many conferences, including NeurIPS, have begun to explore the use of AI-assisted reviewing tools. These tools can help with tasks such as identifying potential reviewers, checking for plagiarism, and even providing initial feedback on the technical soundness of the paper.
However, the increasing reliance on AI in the review process has also raised concerns about bias, fairness, and the potential for errors. While AI can assist with certain aspects of the review process, it cannot replace the critical thinking and nuanced judgment of human reviewers.
Xuandong Zhao, a postdoctoral researcher at UC Berkeley, noted that the prevalence of AI-assisted reviewing has increased significantly in recent years. Two years ago, maybe one in ten reviews felt like they were partially written by AI. Now? It seems like nine out of ten reviews have been edited by AI, including grammar corrections, Zhao observed.
While AI can improve the clarity and grammar of reviews, it can also lead to a homogenization of feedback and a lack of in-depth analysis. Furthermore, if the AI models are trained on biased data, they can perpetuate existing biases in the review process, potentially disadvantaging certain authors or research areas.
The Identity of the Reviewer and the Aftermath
Following the widespread outrage, the identity of the reviewer responsible for the Adam comment was eventually revealed. While the name has been circulating within the AI community, it is important to note that publicly shaming individuals is not a productive solution. The focus should be on addressing the systemic issues that contribute to the problem of low-quality reviews.
The incident has prompted a wave of introspection within the NeurIPS organization and the broader AI community. There is a growing recognition that the review process needs to be reformed to ensure fairness, accuracy, and thoroughness.
Potential Solutions and Future Directions
Several potential solutions have been proposed to address the challenges facing the NeurIPS review process and other similar conferences:
- Increasing the number of reviewers: Expanding the pool of qualified reviewers is crucial to reducing the workload on individual reviewers and ensuring that each paper receives adequate attention. This could involve actively recruiting reviewers from industry, government labs, and other research institutions.
- Improving reviewer training and guidelines: Providing reviewers with better training and clear guidelines on how to conduct thorough and constructive reviews can help to improve the quality of feedback. This could include workshops, online resources, and examples of good and bad reviews.
- Implementing a more robust reviewer selection process: Developing more sophisticated methods for matching reviewers to papers based on their expertise and experience can help to ensure that reviewers are well-equipped to evaluate the submissions they are assigned.
- Exploring alternative review models: Experimenting with alternative review models, such as open review or meta-review, could help to improve transparency and accountability in the review process. Open review involves making reviews publicly available, while meta-review involves having senior reviewers evaluate the quality of the reviews themselves.
- Developing better AI-assisted reviewing tools: Focusing on developing AI tools that can augment, rather than replace, human reviewers can help to improve efficiency without sacrificing quality. This could involve using AI to identify potential errors, summarize key findings, and provide suggestions for improvement, while leaving the critical evaluation and judgment to human reviewers.
- Rethinking the conference format: Considering alternative conference formats, such as smaller, more focused workshops or online conferences, could help to reduce the pressure on the review process and allow for more in-depth discussions of research findings.
- Promoting a culture of constructive criticism: Fostering a culture of constructive criticism within the AI community can help to encourage reviewers to provide helpful and insightful feedback, even when they have concerns about the quality of the work.
The Broader Implications: Trust and Integrity in AI Research
The Adam incident is not just an isolated case of a single reviewer making a mistake. It is a symptom of a larger problem: the increasing pressure on the AI research community to publish and the potential for this pressure to compromise the quality and integrity of the research process.
The incident highlights the importance of maintaining rigorous standards of peer review to ensure that only high-quality, reliable research is published and disseminated. The credibility of the AI field depends on the ability to trust the research findings that are presented at conferences and in journals.
Conclusion: A Call for Reform and Vigilance
The Adam controversy has served as a wake-up call for the NeurIPS organization and the broader AI community. It is a reminder that the review process is a critical component of the scientific enterprise and that it must be continuously improved to ensure fairness, accuracy, and thoroughness.
While AI-assisted reviewing tools can play a valuable role in improving efficiency, they should not be seen as a substitute for human judgment and critical thinking. The focus should be on developing AI tools that augment, rather than replace, human reviewers.
Ultimately, maintaining the integrity of the AI research field requires a collective effort from researchers, reviewers, conference organizers, and funding agencies. By working together to improve the review process, promote a culture of constructive criticism, and uphold the highest standards of scientific rigor, we can ensure that the AI field continues to advance in a responsible and trustworthy manner. The future of AI depends on our ability to maintain trust in the research that underpins it. The Adam incident should serve as a catalyst for meaningful change and a renewed commitment to quality in AI research. The community must remain vigilant and proactive in addressing the challenges facing the review process to ensure that future generations of AI researchers can rely on a fair, accurate, and rigorous system of peer review.
Views: 0