I am one of the Editors-in-Chief of the Transactions on Machine Learning Research (TMLR). I reached out to authors of 10 paper submissions to TMLR, originally slated for desk rejection, asking for a meeting to discuss their paper. The author-attendees of the meetings included undergraduate students, master’s students, PhD students, faculty, and independent researchers. Most, but not all, of these papers were solo authored.

Of the ten submissions:

• Authors of one paper withdrew their submission.
• Authors of one paper said they were unavailable due to other commitments.
• Authors of one paper scheduled a meeting but did not show up.
• Authors of three papers were unable to answer basic questions about the paper.
• Authors of three papers answered questions about high-level ideas in the paper but had difficulty when asked further questions on technical details.
• Authors of one paper answered all of my questions (although I identified a major flaw in that paper).

Overall takeaways:

1. Most journals and conferences require authors to ensure correctness and integrity of their papers and take responsibility for them. This is particularly relevant given potential AI generated or heavily AI-assisted submissions. When authors could not answer questions about the technical parts, and sometimes even basic questions about the paper, it is difficult to see how they could have verified the paper’s contents.

2. This is concerning in today’s research ecosystem where credit is largely assigned via authorship of a paper. The value of publication as a signal of researcher contribution is much weaker when authors cannot explain or defend their paper.

3. In many journals and conferences, authors of submitted/published papers are asked to become reviewers and asked to review others’ papers. If authors do not understand their own papers, it raises questions about them reviewing others’ papers effectively.

4. Amidst the surge in submissions, we need to manage the load on our volunteer reviewers and AEs. This exercise provides us with additional confidence that our desk rejection process is working well. In addition, having submission quotas aligned with recent submission patterns and emphasis on clear writing have been very helpful.

5. This exercise took a lot of my time — 20 to 25 hours in total across two weeks for 8 papers. Due to the time investment required, the process I followed seems hard to scale, especially given the large numbers of submissions. (Separately, our group has been exploring approaches along these lines to make such evaluations more scalable, for better credit assignment and alignment with journal/conference policies: https://www.cs.cmu.edu/~nihars/preprints/greCAPTCHA.pdf.)

More details about the process:

I was on my rotation as one of the Editors-in-Chief in the period of August 14 to 28, 2026, and conducted the following exercise in that period.

All reviewers, Action Editors and Editors-in-Chief for TMLR are unpaid volunteers. Hence we have several aspects to our process that can ensure a manageable review load, especially under the recent surge in submissions. One of them is desk rejections, where like many other scientific journals, TMLR desk rejects papers at the Editor-in-Chief level or at the Action Editor level if it is envisaged to be unlikely to meet some acceptance criterion. The fraction of desk rejected papers at TMLR used to be about 6% in 2023 but is now at about 53%. I informally sampled 10 papers from those slated for desk rejection. Instead of desk rejecting them, I sent the authors of these 10 papers a message via OpenReview (the platform used for the review process) that I would like to speak with them:

“Hi, Thank you for submitting this paper to TMLR. One of the Editors-in-Chief would like to speak with you to understand the paper better before sending it out for review. If that is ok with you, please email some times you will be available to tmlr-editors@jmlr.org”

The objective of this exercise was threefold:
(i) To obtain more clarity behind the papers.
(ii) To evaluate our desk rejection process.
(iii) To assess whether the authors could provide sufficient oversight of their papers, if they were AI generated or heavily AI assisted.

Most but not all of these papers were solo authored. In what follows, I use the term “authors” generically with respect to all papers irrespective of whether they were solo-authored or not.

Out of the ten papers I contacted, the authors of one paper withdrew their paper after the message. The author of another paper said they were too busy with other commitments at that time. I scheduled Zoom meetings with the authors of the remaining eight papers.

I read through each paper as carefully as I reasonably could. I did not necessarily go through every result in every paper (which would have been unmanageable with eight papers in a span of two weeks, alongside regular Editor-in-Chief responsibilities), but studied the premise and a subset of the main results. I also used LLMs as an aid to understand the paper better and also learn some concepts used in the papers that I did not know prior to this exercise.

The authors of one of the remaining eight papers did not show up for the scheduled meeting. I had meetings with authors of the other seven papers. The author-attendees in the meetings included undergraduate students, master’s students, PhD students, faculty, and independent researchers.

In the meetings, I explained the context approximately as follows:
“Thank you for joining this meeting and for your submission to TMLR. I will first describe the context of this meeting. Papers submitted to TMLR first go through an editorial evaluation where if Editors-in-Chief or Action Editors find the paper unclear or having other issues, then the paper may be desk rejected. Your paper was also heading for a desk rejection, but we are doing a small trial where we are speaking with the authors to get more information.

After our meeting, I will convey our discussion to my fellow Editors-in-Chief and possibly an Action Editor. We will then decide the next steps for your paper, which would be one of three possibilities. One, it continues the desk rejection route. Two, it is desk rejected with an invitation to resubmit based on this discussion. Three, the paper is sent to reviewers as is. We will get back to you by next week on OpenReview.”

Broadly, I asked two types of questions:
(1) Basic questions about the problem setting, notation, and results claimed in the paper; and
(2) More detailed questions about particular technical expressions, theoretical results, and design choices made in the experiments.

The meetings lasted about 30 minutes each.

The authors of three of the papers were unable to answer basic questions about their paper. All three of these papers were solo authored. Two of the authors appeared to have almost no substantive understanding of the contents of their own papers. Another author could not identify where some key results claimed in their abstract were presented or supported.

The authors of the remaining four papers were able to answer basic questions about their problem setup, notation, etc. However, the authors of three of these papers had difficulty when asked deeper questions about technical aspects or design choices.

The authors of one paper were able to answer all my questions about the paper. However, my examination of the paper uncovered a major error in one of the paper’s main claims, which the authors subsequently acknowledged.

Here are two additional anecdotes. Authors of two papers, who were unable to answer basic questions in the meeting, subsequently wrote me answers to my questions after our meeting. Pangram classified both these emails as “100% AI.” In a separate meeting, the author of one paper tried to describe the methods they used for analysis, but inadvertently ended up describing an entire p-hacking workflow.

Subsequent to these conversations, we made the following decisions on the ten papers. For the paper where the author was able to answer all questions about the paper, we desk rejected but allowed a resubmission after correcting the error or reducing their claim appropriately, and issuing various clarifications for parts that were not clear. We desk rejected the remaining nine papers without an option to resubmit.

All in all, authors are ultimately responsible for ensuring the accuracy and integrity of the papers they submit under their name. If they cannot explain the basic claims, methods, or technical details of those papers, then this is a serious problem. Journals and conferences should think carefully about the objectives of their review processes and quickly adapt via various initiatives and experiments; we are doing several of these already at TMLR (here, here, and here) and will continue to do so.

1. Peer reviews generally put least emphasis on the criterion of clear writing.

Relative importance given by reviewers in ICLR to different review criteria in their overall recommendations [Kitch et al. 2026]. The criterion of presentation receives the lowest importance.

From our experience, this is true for TMLR as well.

2. Journals and conferences including TMLR are seeing a large number of submissions, including potentially AI generated or heavily AI written ones.

3. AI writing is currently often convoluted, has unnecessary jargon, and not easy to understand.

In light of these, TMLR will put more emphasis on the clarity of writing. TMLR’s criteria for acceptance have been whether the claims made in the submission are supported by accurate, convincing and clear evidence and whether some individuals in TMLR’s audience would be interested in the findings of this paper.

We are changing the latter criterion to explicitly include clarity of communication: “Would some individuals in TMLR’s audience be interested in the findings of this paper, and does the paper communicate those findings clearly to TMLR’s audience?” A submission should clearly communicate what it has found, why those findings may be of interest to some researchers in the area, and what readers can learn from the work. This is not a requirement for elegant prose, flawless English, or any particular writing style. The writing and organization should make the paper’s main contributions, findings, and takeaways understandable to its intended audience. Problems of presentation should be evaluated in terms of whether they materially impede a reader’s ability to understand the work and its relevance. A submission may contain potentially useful findings but present them so unclearly that (human) readers cannot determine the problem setting or experimental setup, in which case it will not meet this criterion. If a paper is entirely AI generated with little or no human involvement, then at least with current AI systems, it is unlikely to meet this criterion.

TMLR will begin an experimental deployment of AI review in which each submission will receive one AI-generated review alongside the usual human reviews. Our approach towards the deployment is as follows (based on Part 1 of this NeurIPS’25 talk):

1) Soundness assessment (only): Use the AI reviews to only evaluate soundness, that is, whether the claims made in the paper are supported by accurate, convincing and clear evidence. The AI reviews will not evaluate any subjective criteria. This is also well aligned with TMLR’s primary acceptance criterion.

2) Evaluations: Conduct evaluations to select an AI reviewer, and ensure that it is of a satisfactory quality. Our evaluation report is available here. Based on this, we will use the AI reviewer of CSPaper.

The AI review will be available to AEs, other reviewers, and authors along with the regular reviews. The AI review will not comment on the “audience” criterion nor will it make a final acceptance/rejection recommendation. The authors will have an opportunity to provide their response to the AI review. We will ensure that the AI review is concise. There will not be a further response from the AI review. The AE would then use the regular reviews, the AI review and the authors’ response to make their decision. The AI review is only advisory and not binding and AEs will make the final decision.

This is an experimental deployment meaning that we will continue to elicit feedback from authors, other reviewers and AEs on the accuracy and usefulness of the AI review. This feedback will help guide use of AI reviewing after that. We are currently integrating it with OpenReview (thanks to Celeste Martinez) and expect it to be live in August 2026. The AI reviews will be available for submissions after launch, and not for papers already under review then.

Fixed versus coauthorship-dependent quotas

Many conferences, journals, and funding agencies have already adopted author quotas on submissions. But they usually impose a fixed quota, e.g., nobody can submit more than 10 papers. This fixed quota rule treats a solo-authored submission and a large collaborative submission identically for each author, even though the reviewing demand per author is lower for a collaborative submission. As we discuss below, this fixed quota approach is too coarse and does not reflect contemporary submission patterns.

As seen in Figure 1, there has been a dramatic rise in single-author submissions in 2026. We have seen many individuals submit four to six papers within one or two weeks. Under these submission patterns, a community-run peer-review process is not sustainable. For instance, if we set a fixed annual quota of 10 submissions per author, then each of these high-volume solo authors could still submit as many as 10 papers. Given the submission patterns we are already observing, our review process cannot sustain that. We therefore set the annual quota for authors submitting only solo-authored papers at two.

But if a fixed quota of two is applied to every author, then advisors with more than two students wanting to submit — a reasonable and common situation — will not be able to do so. These competing requirements thus motivate a quota rule based on the number of coauthors.

How the quota rules work

TMLR will therefore use the harmonic family of quota rules where the cost to submit a paper depends on the number of authors. Under these rules, each researcher has an annual budget. Each submission uses part of every author’s annual budget. The amount charged to each author decreases as the number of coauthors increases. A submission is permitted only if every author has enough remaining budget.

A coauthorship-based rule could create a perverse incentive to add spurious authors in order to increase the number of permitted submissions. This incentive is especially strong in a per-capita rule that would divide the cost of a submission equally among its authors. The harmonic rules are designed to curb this incentive by reducing each author’s cost more gradually as the number of coauthors increases. Since adding authors to a submission reduces the credit to any individual author of the submission, the harmonic rule ensures that an author cannot increase their total credit across all permitted submissions by adding spurious coauthors. The generalized harmonic family allows submission quotas to grow faster with the number of coauthors and makes the rule easier to interpret, but this added flexibility comes at the cost of relaxing that guarantee.

TMLR’s parameters

TMLR will use the generalized harmonic rule with the following parameters:

• N_1 = 2: an author submitting only single-author papers may submit up to 2 papers per calendar year.
• N_9 = 9: an author whose submissions all have 9 authors may submit up to 9 papers per calendar year.

Authors who are active members of the TMLR Action Editor or reviewer pools will receive double the annual budget: N_1 = 4 and N_9 = 18.

Implementation details

Authors can evaluate their quotas here: https://www.cs.cmu.edu/~nihars/quota/author.html?rule=generalized&N1=2&A=9&NA=9

Active AEs and reviewers can use this link: https://www.cs.cmu.edu/~nihars/quota/author.html?rule=generalized&N1=4&A=9&NA=18

An author can run this script to gather all their TMLR submissions from the current year and print the number of authors for each submission. This information can be entered in the pages above to determine how many more submissions an author may make this year.

The quota will count submissions beginning January 1, 2026. Enforcement will begin for papers submitted on or after July 1, 2026. Papers submitted before July 1 will count toward the 2026 annual budget, but will not be affected retroactively by the cap. All submissions, whether accepted, rejected, withdrawn, or desk rejected for any reason, will count toward the quota.

Why these parameter choices

• As mentioned above, no author should solely burden the reviewer pool with more than 2 solo-authored submissions per year (N_1 = 2).
• At the time of writing, 95% of submissions have at most 9 authors, making it a concrete number to focus on (A = 9).
• If a group of A authors write N_A papers together, each of them should have the chance to be first author of at least one submission (N_A ≥ A).
• Within these constraints, N_9 = 9 resulted in other quotas we considered reasonable.

These values also aligned well with our intuition on submission limits for other coauthorship sizes.

In conclusion

By design, some authors will be restricted from submitting the number of papers they would like to. We understand this will be disappointing to them. However, we are taking this initiative in an effort to be respectful to our volunteer team of reviewers and action editors, who are recently faced with substantially increased demands. This is being done in parallel to several other initiatives towards this end. This includes greater editorial oversight of low quality work and stronger enforcement of TMLR policies, which will result in desk rejection and/or prohibiting authors from submission. We aim to mitigate the pressure on the reviewing system while preserving room for genuine collaboration and continuing to support the broad, community-driven review model of TMLR.

Update (July 13, 2026): Following our initial blog post on June 17, 2026, we have had a number of discussions with members of our community. We are glad that people have found our approach sensible and necessary given current submission trends, and that they appreciate TMLR taking the lead in adopting a novel approach to this problem. The questions we received also made us realize that the initial post did not provide sufficient detail, so we have updated it accordingly.

The 2025 TMLR Outstanding Paper Committee is pleased to award the Outstanding Certification to the following paper:

• Mantis: Interleaved Multi-Image Instruction Tuning, by Dongfu Jiang, Xuan He, Huaye Zeng, Cong Wei, Max Ku, Qian Liu, Wenhu Chen*

The Mantis paper has multiple contributions. First, the authors provide a large-scale multi-image instruction tuning dataset. The authors further train a model which achieves state-of-the-art results, despite being trained on academic-level resources. Finally, they provide a human-curated benchmark that requires multi-image understanding. Conceptually, the paper demonstrates that interleaved multi-image instruction tuning can outperform massive pre-training. Furthermore, the committee recognized its substantial and immediate impact on the community, with the Mantis-Eval benchmark adopted by subsequent state-of-the-art models such as Qwen3-VL, Intern3.5-VL, and LLaVA-OneVision.

The Committee also recognizes three papers as Outstanding Certification Finalists:

• Causal Reasoning and Large Language Models: Opening a New Frontier for Causality, by Emre Kiciman, Robert Ness, Amit Sharma, Chenhao Tan
• Cognitive Architectures for Language Agents, by Theodore Sumers, Shunyu Yao, Karthik Narasimhan, Thomas Griffiths
• Gradient Scarcity in Graph Learning with Bilevel Optimization, by Hashem Ghanem, Samuel Vaiter, Nicolas Keriven

Causal Reasoning and Large Language Models was lauded for its comprehensive empirical look at causal reasoning abilities of LLMs, opening this frontier for broader investigation. Cognitive Architectures for Language Agents (CoALA) provided a conceptual framework for connecting classic ideas in computational cognitive science and LLM agent research, synthesizing a fragmented body of work into an actionable set of insights and roadmap for the future. Gradient Scarcity in Graph Learning with Bilevel Optimization gave a mathematical characterization of why gradient scarcity occurs when learning graphs, providing deep insight into the nature of the problem. All papers listed above will receive a Featured Certification, if they did not already have one.

Selection Process

The selection process follows the same general approach used in previous years. All papers published in TMLR up until April 30, 2025, were considered, excluding those considered in previous years. This pool of papers was filtered down based on two criteria: if they were awarded a Featured Certification by their Action Editor, or if they received a large number of citations (determined using Google Scholar). Since TMLR has been rapidly expanding, this still resulted in too many papers to be considered by the committee, despite having a larger number of committee members than in previous years. For each paper that fit these criteria, we asked its Action Editor for their opinion of whether the paper should be considered for the Outstanding Certification, instructing them to err on the side of being permissive, as we didn’t want their sole opinion to carry too much weight unless they were very confident. For the papers which proceeded further, the Action Editors’ opinions were used as an expert review in deliberations.

At this point, we partitioned the resulting set of papers amongst the committee, with care for conflicts of interests. Each committee member was responsible for independently handling roughly six papers, and their instructions were to gather opinions from two additional experts in the area of the paper. These external opinions are essential, as experts in the area have the most informed evaluation of how significant a result is. Each committee member was instructed to advance one paper (or, exceptionally, zero or two papers) to the shortlist.

Eleven papers advanced to the shortlist.** Over the course of roughly two weeks, the committee was instructed to form opinions on the papers in the shortlist, using their own reading, expert comments, and discussion among the committee members. After voting and some additional discussion, committee members agreed upon the outcome above.

After the process, committee members noted that many of the shortlisted papers seemed to be focused on Large Language Models. On one hand, this reflects a popular direction that the community has coalesced around, and thus it is natural that the best work may be concentrated in this area. On the other hand, it may be a bias within the process itself. It was noted that LLM papers are more likely to have a high number of citations, and based on a high number of citations, even experts may be more inclined to comment on them more favourably than they would otherwise. We appreciate the committee’s feedback and will reflect upon whether there are ways to represent more styles of papers in the selection process.

Acknowledgments

I would like to extend my sincere thanks to the entire committee. TMLR takes the Outstanding Certification selection process very seriously, with a long and elaborate process as described above. The committee was able to meet these rigorous demands and participate actively and thoughtfully across the process, including on short notice or with rapid turnarounds at times. They have my genuine gratitude.

I would also like to thank Hugo Larochelle, who served to witness the selection process.

Finally, in previous years, we have listed all expert reviewers who were kind enough to share their opinions and expertise. Given that the recent OpenReview security incident is fresh in peoples’ minds, and the fact that we did not previously notify experts that their names might be publicly acknowledged, we felt that it might be a bit tone-deaf to reveal their names right now, even without explicit association to which papers they opined on. Nonetheless, they are free to list this information publicly if they so desire. In future years, we will explicitly warn experts that their name will appear in an acknowledgment.

*Note that I (Gautam Kamath) have an institutional conflict with several of the authors, being colleagues at the University of Waterloo. However, I was only involved in defining and overseeing the process, and did not participate in the discussions or voting process. To the best of my knowledge, my CoI did not substantially influence their decisions. Indeed, support from the committee for the winning paper was overwhelming.

**At this point, Colin Raffel was excused from the committee due to a strong CoI with one of the papers advanced (by another committee member) to the shortlist.

We are excited to launch TMLR Beyond PDF!

TMLR Beyond PDF is a new, HTML-based submission format for TMLR, that supports interactive figures and videos, along with the usual LaTeX and images. This format is inspired by interactive publication formats such as Distill.pub and ICLR Blogposts.

The submission process for TMLR Beyond PDF is largely the same as standard TMLR. Submissions are made on OpenReview to ensure that the peer-review process remains double-blind. To make a Beyond PDF submission, just select the brand new “Beyond PDF submission (pageless, webpage-style content)” submission type and upload a zipped folder containing your submission files (the author kit can be found here).

Once you submit to TMLR Beyond PDFs, your submission will be rendered automatically at https://tmlr-beyond-pdf.org/under_review!

Check out https://tmlr-beyond-pdf.org/submission-instructions for step by step instructions. Also check out the TMLR website for more details on TMLR’s editorial policies, review process, and other information that applies to both conventional and Beyond PDF submissions.

Special thanks to the OpenReview team, in particular Celeste Martinez and Melisa Bok, for their help!

We also extend our thanks to the committee led by Pablo Samuel Castro that made recommendations regarding the extension of TMLR to the Beyond PDF format. The committee included Susan Zhang, Fabian Pedregosa, Nikhil Thorat, Thomas Dietterich, Sara Hooker, Hugo Larochelle, and Paul Vicol.

We look forward to seeing you at TMLR Beyond PDF!

How We Got Here

Since late 2022, NeurIPS, ICML, and ICLR have partnered on a joint J2C track, which allows authors of papers published in the Journal of Machine Learning Research (JMLR) to present their work at one of the three conferences. This initiative has been a fantastic way for authors to share their research in person.

In a separate pilot, TMLR teamed up with ICLR 2025 to allow a select group of TMLR publications to be presented. Based on the success of that pilot and positive discussions with the boards of all three conferences, expanding the program to include TMLR in the joint J2C track was the logical next step.

Eligibility for the Track

To be eligible for the J2C track, in addition to the original eligibility criteria for JMLR publications (detailed here), a TMLR paper must receive at least one of the following certifications:

• A Featured Certification
• An Outstanding Certification
• A J2C Certification (new)

The new J2C Certification will be awarded to papers that receive strong support from their Action Editor and reviewers, following a process similar to the ICLR pilot program. This will be a selective and prestigious recognition, as we expect only about 10% of accepted TMLR papers to receive it.

We’re also retroactively applying this certification to TMLR papers published since January 1, 2024, that would have been eligible. Authors of these papers will be notified directly.

We’d like to extend a huge thank you to the board members of all three conferences for their support, as well as Celeste Martinez Gomez from OpenReview and Paul Vicol (TMLR’s Managing Editor) for their help in making this partnership a reality.

To learn more, visit the Journal-to-Conference Track page on the NeurIPS, ICML, or ICLR website.

The TMLR Editors-in-Chief
Hugo Larochelle
Naila Murray
Gautam Kamath
Nihar B. Shah

The 2024 TMLR Outstanding Paper Committee is pleased to award Outstanding Certifications to two papers this year:

• “Inversion by Direct Iteration: An Alternative to Denoising Diffusion for Image Restoration”
• “Holistic Evaluation of Language Models” (HELM)

This marks the first time the certification has been awarded to multiple papers, reflecting the exceptional quality and impact of both works.

“Inversion by Direct Iteration: An Alternative to Denoising Diffusion for Image Restoration,” by Mauricio Delbracio and Peyman Milanfar, introduces an elegant method for image restoration that effectively addresses the “regression to the mean” problem in supervised image restoration. Unlike existing denoising diffusion models, the proposed method does not require an analytic form for the degradation process, making it applicable to any restoration task with paired examples. The work provides deep theoretical connections with residual flow ODEs, denoising diffusion, and flow matching, while maintaining a remarkably simple formulation. The work has already inspired numerous follow-on papers and received strong endorsements from both expert reviewers and Action Editors for its theoretical foundations and empirical validation. Experts particularly noted the paper’s comprehensive investigation of the proposed method and its elegant theoretical connections with existing approaches.

“Holistic Evaluation of Language Models” (HELM), by a group of 50 authors including lead authors Percy Liang, Rishi Bommasani, and Tony Lee, establishes a comprehensive framework for evaluating language models that has become foundational. The paper stands out for its careful methodology in isolating conclusions and its thorough evaluation design across multiple dimensions of model performance. Expert reviewers highlighted that HELM was one of the first evaluation platforms in the era of large language models, setting important standards through its comprehensive approach to details and thoughtful consideration of evaluation metrics. The work’s impact extends beyond its immediate contributions, and the project has grown far beyond the scope of the original paper. The HELM team maintains an active project page featuring open-source release of several leaderboards, code, and evaluation products, enabling a wide range of future research. Notably, the authors thoughtfully acknowledged the platform’s limitations, which has helped guide subsequent research in addressing these gaps.

We note that, while institutional affiliation was not a selection criterion, the two winning papers represent work from both academic and industrial research teams. This reflects TMLR’s role as a venue that attracts excellent submissions from researchers across the broader machine learning community.

The Committee also recognizes three papers as Outstanding Paper Finalists:

• “Beyond the Imitation Game: Quantifying and Extrapolating the Capabilities of Language Models” (BIG-bench)
• “DINOv2: Learning Robust Visual Features without Supervision”
• “Open Problems and Fundamental Limitations of Reinforcement Learning from Human Feedback”

These papers represent significant contributions across different areas of machine learning. BIG-bench was noted for its remarkable cross-institutional collaboration and novel task designs that highlight important areas for future model improvement. DINOv2 demonstrated substantial improvements in fine-grained visual understanding tasks and has become one of the most commonly used backbones in computer vision. The RLHF survey paper provides a comprehensive analysis of an increasingly important technology while highlighting crucial open problems and limitations. All of these works have been retroactively recognized with a Featured Certification.

Selection Process

The selection process for this year’s awards followed similar criteria to last year’s inaugural selection, while incorporating additional rigor in the evaluation process. Any papers published in TMLR up until May 3, 2024 were considered for this award, excluding those that were already considered for last year’s award. The initial pool of candidates was filtered based on two primary criteria: papers that received Featured Certifications from their Action Editors or papers that demonstrated significant citation impact (identified using bibliometric analysis).

Featured Certifications continue to serve as one of the most reliable indicators of exceptional work in TMLR, as these papers have been specifically highlighted by their Action Editors for their outstanding contributions. Citation metrics, while considered, were carefully weighted to account for various factors including publication timing and subfield-specific citation patterns.

For each candidate paper, the committee solicited detailed feedback from both the paper’s Action Editor and multiple domain experts. This expert feedback was crucial in evaluating several dimensions:

• Technical depth and novelty
• Broader impact on the field
• Reproducibility and open-source contributions
• Methodological rigor
• Potential for long-term influence in their respective fields

The committee conducted multiple rounds of review and discussion, with careful attention to managing potential conflicts of interest. Committee members with institutional affiliations to any candidate papers recused themselves from voting on those papers. Even with these recusals, the winning papers received standout support from non-conflicted committee members.

The papers that advanced to final consideration demonstrated not only technical excellence but also significant impact on their respective fields. The committee noted that several other strong candidates were considered, and the final selection represents papers that have already begun to shape how the community approaches important problems in machine learning.

We encourage the machine learning community to read both the winning papers and the finalist papers, as they represent exemplary work published in TMLR and demonstrate the journal’s commitment to publishing high-quality, impactful research across different areas of machine learning.

Acknowledgments

We would like to thank Gautam Kamath and Naila Murray for logistically overseeing the selection process, and our team of expert reviewers and Action Editors, including (in alphabetical order): Naman Agarwal, Anurag Arnab, Richard Baraniuk, Yonatan Bisk, Valentin De Bortoli, Mathilde Caron, João Carreira, Antoni Chan, Swarat Chaudhuri, Changyou Chen, Pin-Yu Chen, Sinho Chewi, Leshem Chosen, Marco Cuturi, Mostafa Dehghani, Yuntian Deng, Carl Doersch, Vincent Dumoulin, Greg Durrett, Dumitru Erhan, Aleksandra Faust, Vincent Fortuin, Scott Geng, Shixiang Shane Gu, Jia-Bin Huang, Phillip Isola, Bahjat Kawar, Abhishek Kumar, Stefan Lee, Fuxin Li, Lihong Li, Yujia Li, Anatole von Lilienfeld, Marlos C. Machado, Stephan M Mandt, Mirco Mutti, Lili Mou, Karthik Narasimhan, Gang Niu, Ivan Oseledets, Gabriel Peyre, Colin Raffel, Marcello Restelli, Fancisco Ruiz, Jonathan Scarlett, Ludwig Schmidt, Evan Shelhamer, Freda Shi, Matthew E. Taylor, Kevin Xu, Qiang Xu, Makoto Yamada, Jong Chul Ye, Hanwang Zhang.

The first TMLR Outstanding Paper Committee awards the first Outstanding Certification to “A Generalist Agent,” by authors Scott Reed, Konrad Zolna, Emilio Parisotto, Sergio Gómez Colmenarejo, Alexander Novikov, Gabriel Barth-Maron, Mai Giménez, Yury Sulsky, Jackie Kay, Jost Tobias Springenberg, Tom Eccles, Jake Bruce, Ali Razavi, Ashley Edwards, Nicolas Heess, Yutian Chen, Raia Hadsell, Oriol Vinyals, Mahyar Bordbar, and Nando de Freitas. Congratulations to the authors!

The paper introduces an agent, known as Gato, which is capable of handling multiple modalities, tasks, and embodiments, all with the same set of weights. This network is simultaneously capable of playing Atari games, captioning images, chatting, stacking blocks with a robotic arm, and more. This paper was one of the earliest efforts to explore large-scale, multimodal, multitask generalist agents and policies, and it shows that the strategy of massive multitasking used in many text-based large language models is broadly applicable.

The Outstanding Paper Committee consisted of Roman Garnett, Gautam Kamath, Brian Kingsbury, Yingzhen Li, and Zhihui Zhu, and was selected by the Editors-in-Chief of TMLR, Kyunghyun Cho, Raia Hadsell, and Hugo Larochelle*. For the sake of transparency, we outline our selection process below.

The first 393 papers in TMLR, published up until May 3, 2023, were considered for this award. As it would be impossible for the committee to read and evaluate every single paper, we filtered this down to a list of 18 papers, based on the following criteria: papers that were awarded a Featured Certification by their Action Editor (AE) (10 papers), and papers that had a relatively high number of citations (8 papers, identified using the Python package scholarly, code available upon request). We additionally allowed the award committee members to nominate further papers as candidates, although none elected to do so.

Featured Certifications are likely the most reliable way to identify high-quality work in TMLR, as these papers have been specifically highlighted by their AEs. That said, there were only a small number of papers which received this designation (10/393 = ~2.5%). Whether this small number is due to few papers meeting the bar, miscalibration by the reviewers and AEs, or lack of knowledge of this option by the reviewers and AEs, is currently not clear. We encourage reviewers and AEs of future papers to seriously consider the Featured Certification for papers they are assigned to. Note in particular that TMLR reviews do not assign scores, so we were unable to use high scores as a signal.

The other indicators are less reliable. Citation-based metrics vary significantly depending on area, are prone to manipulation, and may favour papers which were published earlier in the year. To avoid “false positives,” highly cited papers were still subject to another round of scrutiny (described below). Unfortunately, high-quality but low-cited papers are difficult to discover, aside from Featured Certifications. The Outstanding Paper Committee was unable to identify any further notable candidate papers beyond those already identified. This may be because the committee was most familiar with the papers for which we were Action Editors, and if those were sufficiently high quality, we would have at least recognized them with a Featured Certification already.

Though the Outstanding Paper Committee had a breadth of expertise, it was insufficient to evaluate every candidate in context. As such, we solicited opinions from each candidate paper’s Action Editor, as well as two experts in the paper’s area, asking whether the results would indeed be suitable for this honour (these individuals are acknowledged at the bottom of this post). Based on this feedback and the committee’s judgement, we narrowed it down to five candidates. We chose the winning paper from this shortlist with a vote and some discussion.

Besides the winning paper, the other four papers considered in our final deliberations included:

• Patches Are All You Need?, by Asher Trockman and J Zico Kolter;
• On Characterizing the Trade-off in Invariant Representation Learning, by Bashir Sadeghi, Sepehr Dehdashtian, and Vishnu Boddeti;
• CoCa: Contrastive Captioners are Image-Text Foundation Models, by Jiahui Yu, Zirui Wang, Vijay Vasudevan, Legg Yeung, Mojtaba Seyedhosseini, and Yonghui Wu;
• How to train your ViT? Data, Augmentation, and Regularization in Vision Transformers, by Andreas Peter Steiner, Alexander Kolesnikov, Xiaohua Zhai, Ross Wightman, Jakob Uszkoreit, and Lucas Beyer.

While we did not explicitly optimize for this, we note that the papers under final consideration are roughly evenly split between academic and industrial affiliations. Furthermore, the latter two papers were not initially recognized with Featured Certification, but have since achieved significant impact in the community. The last paper listed was actually the first paper published in TMLR, all the way back in May 2022! We encourage folks to check out these other papers for examples of high-quality work appearing in TMLR.

We note that this selection process was lengthy, taking place asynchronously over the course of two months. We appreciate the patience of the Editors-in-Chief. Since there was no significant impetus for a rapid decision, we opted for a thorough and low-pressure process to ensure the selection was thoughtfully executed and committee members were not burnt out.

Acknowledgements

We would like to thank Hugo Larochelle for overseeing the selection process, and our team of expert reviewers and Action Editors, including Pieter Abbeel, Naman Agarwal, Aurélien Bellet, Shai Ben-David, Laurent Charlin, Tianlong Chen, Wenhu Chen, Zhe Gan, David Ha, Hilde Kuehne, Chunyuan Li, Tongliang Liu, Karthik Narasimhan, Behnam Neyshabur, Gang Niu, Christopher Pal, George Papamakarios, Aaron Roth, Maja Rudolph, Thomas Steinke, Eugene Vorobeychik, Yunhe Wang, Mark van der Wilk, Zhiwei Steven Wu, and Yaoliang Yu.

*We note that Raia Hadsell, one of the Editors-in-Chief of the journal, is a co-author of the selected paper. However, after the selection of the committee members, Raia was excluded from all discussion and deliberation to avoid a conflict of interest, since the paper having a Featured Certification meant that it was automatically under consideration. To the best of the committee’s ability, her status as EiC had no influence on our decision.