LLM2D
返回列表
Llama-Nemotron：高效的推理模型
Llama-Nemotron: Efficient Reasoning Models
作者: Akhiad Bercovich, Itay Levy, Izik Golan, Mohammad Dabbah, Ran El-Yaniv, Omri Puny, Ido Galil, Zach Moshe, Tomer Ronen, Najeeb Nabwani, Ido Shahaf, Oren Tropp, Ehud Karpas, Ran Zilberstein, Jiaqi Zeng, Soumye Singhal, Alexander Bukharin, Yian Zhang, Tugrul Konuk, Gerald Shen, Ameya Sunil Mahabaleshwarkar, Bilal Kartal, Yoshi Suhara, Olivier Delalleau, Zijia Chen, Zhilin Wang, David Mosallanezhad, Adi Renduchintala, Haifeng Qian, Dima Rekesh, Fei Jia, Somshubra Majumdar, Vahid Noroozi, Wasi Uddin Ahmad, Sean Narenthiran, Aleksander Ficek, Mehrzad Samadi, Jocelyn Huang, Siddhartha Jain, Igor Gitman, Ivan Moshkov, Wei Du, Shubham Toshniwal, George Armstrong, Branislav Kisacanin, Matvei Novikov, Daria Gitman, Evelina Bakhturina, Jane Polak Scowcroft, John Kamalu, Dan Su, Kezhi Kong, Markus Kliegl, Rabeeh Karimi, Ying Lin, Sanjeev Satheesh, Jupinder Parmar, Pritam Gundecha, Brandon Norick, Joseph Jennings, Shrimai Prabhumoye, Syeda Nahida Akter, Mostofa Patwary, Abhinav Khattar, Deepak Narayanan, Roger Waleffe, Jimmy Zhang, Bor-Yiing Su, Guyue Huang, Terry Kong, Parth Chadha, Sahil Jain, Christine Harvey, Elad Segal, Jining Huang, Sergey Kashirsky, Robert McQueen, Izzy Putterman, George Lam, Arun Venkatesan, Sherry Wu, Vinh Nguyen, Manoj Kilaru, Andrew Wang, Anna Warno, Abhilash Somasamudramath, Sandip Bhaskar, Maka Dong, Nave Assaf, Shahar Mor, Omer Ullman Argov, Scot Junkin, Oleksandr Romanenko, Pedro Larroy, Monika Katariya, Marco Rovinelli, Viji Balas, Nicholas Edelman, Anahita Bhiwandiwalla, Muthu Subramaniam, Smita Ithape, Karthik Ramamoorthy, Yuting Wu, Suguna Varshini Velury, Omri Almog, Joyjit Daw, Denys Fridman, Erick Galinkin, Michael Evans, Katherine Luna, Leon Derczynski, Nikki Pope, Eileen Long, Seth Schneider, Guillermo Siman, Tomasz Grzegorzek, Pablo Ribalta, Monika Katariya, Joey Conway, Trisha Saar, Ann Guan, Krzysztof Pawelec, Shyamala Prayaga, Oleksii Kuchaiev, Boris Ginsburg, Oluwatobi Olabiyi, Kari Briski, Jonathan Cohen, Bryan Catanzaro, Jonah Alben, Yonatan Geifman, Eric Chung
发布日期: 5/5/2025
arXiv ID: oai:arXiv.org:2505.00949v1

摘要

arXiv:2505.00949v1 宣告类型: cross 摘要: 我们介绍了Llama-Nemotron系列模型，这是一个具有异构推理能力的开放模型家族，能够提供卓越的推理能力、推断效率，并且拥有适用于企业使用的开放许可。该家族包含三种规模的模型——Nano（8B）、Super（49B）和Ultra（253B），并在推理性能和内存效率方面优于当前最先进的推理模型DeepSeek-R1，同时还能提供竞争力的表现。在本报告中，我们讨论了这些模型的训练方法，这涉及从Llama 3模型使用神经架构搜索以加速推理、进行知识蒸馏以及继续预训练，随后是一个以推理为中心的后训练阶段，包含两个主要部分：监督微调和大规模强化学习。Llama-Nemotron模型是首次支持动态推理切换的开源模型，在推断过程中允许用户在标准聊天模式和推理模式之间切换。为支持开放研究并促进模型开发，我们提供了以下资源：1. 我们在具有商业宽容性的NVIDIA Open Model License Agreement下发布了Llama-Nemotron推理模型——LN-Nano、LN-Super和LN-Ultra。2. 我们发布了完整的后训练数据集：Llama-Nemotron-Post-Training-Dataset。3. 我们还发布了我们的训练代码库：NeMo、NeMo-Aligner和Megatron-LM。
查看原文下载 PDF