Literature Database Entry

chen2026fedsubmuon

Shaolong Chen, Youming Tao, Shuzhen Chen, Falko Dressler, Qingqing Ye and Di Wang, "FedSubMuon: Communication-Efficient Federated LLM Fine-Tuning via Structured Subspace Muon," Proceedings of Conference on Empirical Methods in Natural Language Processing (EMNLP 2026), Budapest, Hungary, October 2026. (to appear)

Abstract

Federated fine-tuning adapts large language models (LLMs) to decentralized client data, but its scalability in cross-device training is often limited by the high communication cost. Muon is an optimizer that improves optimization performance by orthogonalizing momentum for matrix-valued parameters. Existing federated Muon methods demonstrate the benefit of matrix-aware optimization in federated learning, but still require transmitting full layer-size updates and optimizer state. A natural way to reduce communication is to directly apply Muon to LoRA factors, but this changes the optimized object and weakens Muon's matrix-aware update geometry. We propose FedSubMuon, a communication-efficient federated Muon fine-tuning method that optimizes compact coefficient matrices within shared structured subspaces. This design keeps Muon on a single matrix-valued trainable object, while reducing the client upload to compact coefficient matrices. We further introduce FedSubMuon-GT, an accuracy-oriented extension that uses projected gradients to adapt tracked subspace bases toward task-relevant gradient directions. Experiments on instruction tuning and mathematical reasoning show that FedSubMuon-GT achieves the best overall accuracy on four of five dataset–model pairs, while FedSubMuon performs best under all matched communication budgets. On Dolly-15K, the closest communication baseline requires 5.5x and 1.4x more total communication on Llama-1B and Qwen-4B, respectively.

Quick access

BibTeX BibTeX

Contact

Shaolong Chen
Youming Tao
Shuzhen Chen
Falko Dressler
Qingqing Ye
Di Wang

BibTeX reference

@inproceedings{chen2026fedsubmuon,
    author = {Chen, Shaolong and Tao, Youming and Chen, Shuzhen and Dressler, Falko and Ye, Qingqing and Wang, Di},
    note = {to appear},
    title = {{FedSubMuon: Communication-Efficient Federated LLM Fine-Tuning via Structured Subspace Muon}},
    publisher = {Association for Computational Linguistics (ACL)},
    address = {Budapest, Hungary},
    booktitle = {Conference on Empirical Methods in Natural Language Processing (EMNLP 2026)},
    month = {10},
    year = {2026},
   }
   
   

Copyright notice

Links to final or draft versions of papers are presented here to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. In most cases, these works may not be reposted or distributed for commercial purposes without the explicit permission of the copyright holder.

The following applies to all papers listed above that have IEEE copyrights: Personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works must be obtained from the IEEE.

The following applies to all papers listed above that are in submission to IEEE conference/workshop proceedings or journals: This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible.

The following applies to all papers listed above that have ACM copyrights: ACM COPYRIGHT NOTICE. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from Publications Dept., ACM, Inc., fax +1 (212) 869-0481, or permissions@acm.org.

The following applies to all SpringerLink papers listed above that have Springer Science+Business Media copyrights: The original publication is available at www.springerlink.com.

This page was automatically generated using BibDB and bib2web.

Last modified: 2026-08-31