LLM2D
返回列表
Llama-Nemotron：高效的推理模型
Llama-Nemotron: Efficient Reasoning Models
作者: Akhiad Bercovich, Itay Levy, Izik Golan, Mohammad Dabbah, Ran El-Yaniv, Omri Puny, Ido Galil, Zach Moshe, Tomer Ronen, Najeeb Nabwani, Ido Shahaf, Oren Tropp, Ehud Karpas, Ran Zilberstein, Jiaqi Zeng, Soumye Singhal, Alexander Bukharin, Yian Zhang, Tugrul Konuk, Gerald Shen, Ameya Sunil Mahabaleshwarkar, Bilal Kartal, Yoshi Suhara, Olivier Delalleau, Zijia Chen, Zhilin Wang, David Mosallanezhad, Adi Renduchintala, Haifeng Qian, Dima Rekesh, Fei Jia, Somshubra Majumdar, Vahid Noroozi, Wasi Uddin Ahmad, Sean Narenthiran, Aleksander Ficek, Mehrzad Samadi, Jocelyn Huang, Siddhartha Jain, Igor Gitman, Ivan Moshkov, Wei Du, Shubham Toshniwal, George Armstrong, Branislav Kisacanin, Matvei Novikov, Daria Gitman, Evelina Bakhturina, Jane Polak Scowcroft, John Kamalu, Dan Su, Kezhi Kong, Markus Kliegl, Rabeeh Karimi, Ying Lin, Sanjeev Satheesh, Jupinder Parmar, Pritam Gundecha, Brandon Norick, Joseph Jennings, Shrimai Prabhumoye, Syeda Nahida Akter, Mostofa Patwary, Abhinav Khattar, Deepak Narayanan, Roger Waleffe, Jimmy Zhang, Bor-Yiing Su, Guyue Huang, Terry Kong, Parth Chadha, Sahil Jain, Christine Harvey, Elad Segal, Jining Huang, Sergey Kashirsky, Robert McQueen, Izzy Putterman, George Lam, Arun Venkatesan, Sherry Wu, Vinh Nguyen, Manoj Kilaru, Andrew Wang, Anna Warno, Abhilash Somasamudramath, Sandip Bhaskar, Maka Dong, Nave Assaf, Shahar Mor, Omer Ullman Argov, Scot Junkin, Oleksandr Romanenko, Pedro Larroy, Monika Katariya, Marco Rovinelli, Viji Balas, Nicholas Edelman, Anahita Bhiwandiwalla, Muthu Subramaniam, Smita Ithape, Karthik Ramamoorthy, Yuting Wu, Suguna Varshini Velury, Omri Almog, Joyjit Daw, Denys Fridman, Erick Galinkin, Michael Evans, Shaona Ghosh, Katherine Luna, Leon Derczynski, Nikki Pope, Eileen Long, Seth Schneider, Guillermo Siman, Tomasz Grzegorzek, Pablo Ribalta, Monika Katariya, Chris Alexiuk, Joey Conway, Trisha Saar, Ann Guan, Krzysztof Pawelec, Shyamala Prayaga, Oleksii Kuchaiev, Boris Ginsburg, Oluwatobi Olabiyi, Kari Briski, Jonathan Cohen, Bryan Catanzaro, Jonah Alben, Yonatan Geifman, Eric Chung
发布日期: 5/15/2025
arXiv ID: oai:arXiv.org:2505.00949v3

摘要

arXiv:2505.00949v3 宣告类型: replace-cross 摘要: 我们介绍了Llama-Nemotron系列模型，这是一个开放的异构推理模型家族，具有出色的推理能力、推理效率以及适用于企业使用的开放许可。该家族包括三种规模的模型——Nano（8B）、Super（49B）和Ultra（253B），并在与最先进的推理模型如DeepSeek-R1竞争时表现出色，同时提供了更好的推理吞吐量和内存效率。在本报告中，我们讨论了这些模型的训练过程，其中包括使用Llama 3模型的神经架构搜索以加速推理、知识蒸馏以及持续的预训练，然后是一个以推理为重点的后训练阶段，分为两个主要部分：监督微调和大规模强化学习。Llama-Nemotron模型是支持动态推理切换的第一个开源模型，允许用户在推理过程中在标准聊天模式和推理模式之间切换。为了进一步支持开放研究并促进模型开发，我们提供了以下资源：1. 我们在商业上较为宽松的NVIDIA Open Model License Agreement下发布了Llama-Nemotron推理模型——LN-Nano、LN-Super和LN-Ultra。2. 我们发布了完整的后训练数据集：Llama-Nemotron-Post-Training-Dataset。3. 我们还发布了我们的训练代码库：NeMo、NeMo-Aligner和Megatron-LM。
查看原文下载 PDF