LLM2D
返回列表
一步音频：智能语音交互中的统一理解和生成
Step-Audio: Unified Understanding and Generation in Intelligent Speech Interaction
作者: Ailin Huang, Boyong Wu, Bruce Wang, Chao Yan, Chen Hu, Chengli Feng, Fei Tian, Feiyu Shen, Jingbei Li, Mingrui Chen, Peng Liu, Ruihang Miao, Wang You, Xi Chen, Xuerui Yang, Yechang Huang, Yuxiang Zhang, Zheng Gong, Zixin Zhang, Hongyu Zhou, Jianjian Sun, Brian Li, Chengting Feng, Changyi Wan, Hanpeng Hu, Jianchang Wu, Jiangjie Zhen, Ranchen Ming, Song Yuan, Xuelin Zhang, Yu Zhou, Bingxin Li, Buyun Ma, Hongyuan Wang, Kang An, Wei Ji, Wen Li, Xuan Wen, Xiangwen Kong, Yuankai Ma, Yuanwei Liang, Yun Mou, Bahtiyar Ahmidi, Bin Wang, Bo Li, Changxin Miao, Chen Xu, Chenrun Wang, Dapeng Shi, Deshan Sun, Dingyuan Hu, Dula Sai, Enle Liu, Guanzhe Huang, Gulin Yan, Heng Wang, Haonan Jia, Haoyang Zhang, Jiahao Gong, Junjing Guo, Jiashuai Liu, Jiahong Liu, Jie Feng, Jie Wu, Jiaoren Wu, Jie Yang, Jinguo Wang, Jingyang Zhang, Junzhe Lin, Kaixiang Li, Lei Xia, Li Zhou, Liang Zhao, Longlong Gu, Mei Chen, Menglin Wu, Ming Li, Mingxiao Li, Mingliang Li, Mingyao Liang, Na Wang, Nie Hao, Qiling Wu, Qinyuan Tan, Ran Sun, Shuai Shuai, Shaoliang Pang, Shiliang Yang, Shuli Gao, Shanshan Yuan, Siqi Liu, Shihong Deng, Shilei Jiang, Sitong Liu, Tiancheng Cao, Tianyu Wang, Wenjin Deng, Wuxun Xie, Weipeng Ming, Wenqing He, Wen Sun, Xin Han, Xin Huang, Xiaomin Deng, Xiaojia Liu, Xin Wu, Xu Zhao, Yanan Wei, Yanbo Yu, Yang Cao, Yangguang Li, Yangzhen Ma, Yanming Xu, Yaoyu Wang, Yaqiang Shi, Yilei Wang, Yizhuang Zhou, Yinmin Zhong, Yang Zhang, Yaoben Wei, Yu Luo, Yuanwei Lu, Yuhe Yin, Yuchu Luo, Yuanhao Ding, Yuting Yan, Yaqi Dai, Yuxiang Yang, Zhe Xie, Zheng Ge, Zheng Sun, Zhewei Huang, Zhichao Chang, Zhisheng Guan, Zidong Yang, Zili Zhang, Binxing Jiao, Daxin Jiang, Heung-Yeung Shum, Jiansheng Chen, Jing Li, Shuchang Zhou, Xiangyu Zhang, Xinhao Zhang, Yibo Zhu
发布日期: 2/18/2025
arXiv ID: oai:arXiv.org:2502.11946v2

摘要

arXiv:2502.11946v2 Announce Type: cross 摘要：实时语音交互作为一种人类与机器协作的基本接口，具有巨大的潜力。然而，当前的开源模型面临着诸如语音数据采集成本高、动态控制薄弱和智能有限等问题。为了解决这些挑战，本文介绍了Step-Audio，这是第一个生产级别的开源解决方案。主要贡献包括：1）一个包含130亿参数的统一语音-文本多模态模型，实现了统一的理解和生成能力，Step-Audio-Chat版本已开源；2）生成语音数据引擎，建立了可负担的语音克隆框架，并通过蒸馏生成了开源的轻量级Step-Audio-TTS-3B模型；3）基于指令的精细控制系统，能够动态调整方言、情绪、唱歌和说唱；4）增强的认知架构，增加了工具调用和角色扮演能力，有效管理复杂任务。基于我们新的StepEval-Audio-360评估基准，Step-Audio在人类评估中达到了最先进的性能，尤其是在指令遵循方面。在LLaMA Question等开源基准上，显示了9.3%的平均性能提升，展示了我们对推进开源多模态语言技术发展的承诺。我们的代码和模型可在https://github.com/stepfun-ai/Step-Audio获取。
查看原文下载 PDF