Literature Database Entry

shen2026fed-grow

Shikun Shen, Yifei Zou, Yuan Yuan, Hanlin Gu, Peng Li, Xiuzhen Cheng, Falko Dressler and Dongxiao Yu, "Fed-Grow: Federating to Grow Transformers for Resource-Constrained Users without Model Sharing," IEEE Transactions on Parallel and Distributed Systems, February 2026. (online first)

Abstract

The growing resource demands of large-scale transformer models pose significant challenges for resource-constrained users, particularly in distributed environments. To address this issue, we propose a federated learning framework called Fed-Grow, which enables multiple participants to collaboratively learn a lightweight scaling operation that transfers knowledge from pretrained small models to a large transformer model. In Fed-Grow, we introduce the Dual-LiGO (Dual Linear Growth Operator) architecture, consisting of Local-LiGO and Global-LiGO components. Local-LiGO addresses model heterogeneity by adapting each participant’s pre-trained model to a common intermediate form, while Global-LiGO facilitates knowledge sharing across participants without sharing local models or raw data, ensuring privacy preservation. This federated approach offers a scalable solution for growing large transformers in a distributed manner, where only the Global-LiGO is shared, significantly reducing communication overhead while maintaining comparable model performance under the same communication constraints. Experimental results demonstrate that Fed-Grow outperforms state-of-the-art methods in terms of accuracy and precision, while reducing the number of trainable parameters by 59.25% and communication costs by 73.01%. These improvements allow for higher efficiency in training large models in distributed environments, without sacrificing performance. To the best of our knowledge, Fed-Grow is the first method that enables cooperative transformer scaling in a distributed setting, making it a practical solution for resource-constrained users.

Quick access

Original Version DOI (at publishers web site)
Authors' Version PDF (PDF on this web site)
BibTeX BibTeX

Contact

Shikun Shen
Yifei Zou
Yuan Yuan
Hanlin Gu
Peng Li
Xiuzhen Cheng
Falko Dressler
Dongxiao Yu

BibTeX reference

@article{shen2026fed-grow,
    author = {Shen, Shikun and Zou, Yifei and Yuan, Yuan and Gu, Hanlin and Li, Peng and Cheng, Xiuzhen and Dressler, Falko and Yu, Dongxiao},
    doi = {10.1109/TPDS.2026.3666309},
    note = {to appear},
    title = {{Fed-Grow: Federating to Grow Transformers for Resource-Constrained Users without Model Sharing}},
    journal = {IEEE Transactions on Parallel and Distributed Systems},
    publisher = {IEEE},
    month = {2},
    year = {2026},
   }
   
   

Copyright notice

Links to final or draft versions of papers are presented here to ensure timely dissemination of scholarly and technical work. Copyright and all rights therein are retained by authors or by other copyright holders. All persons copying this information are expected to adhere to the terms and constraints invoked by each author's copyright. In most cases, these works may not be reposted or distributed for commercial purposes without the explicit permission of the copyright holder.

The following applies to all papers listed above that have IEEE copyrights: Personal use of this material is permitted. However, permission to reprint/republish this material for advertising or promotional purposes or for creating new collective works for resale or redistribution to servers or lists, or to reuse any copyrighted component of this work in other works must be obtained from the IEEE.

The following applies to all papers listed above that are in submission to IEEE conference/workshop proceedings or journals: This work has been submitted to the IEEE for possible publication. Copyright may be transferred without notice, after which this version may no longer be accessible.

The following applies to all papers listed above that have ACM copyrights: ACM COPYRIGHT NOTICE. Permission to make digital or hard copies of part or all of this work for personal or classroom use is granted without fee provided that copies are not made or distributed for profit or commercial advantage and that copies bear this notice and the full citation on the first page. Copyrights for components of this work owned by others than ACM must be honored. Abstracting with credit is permitted. To copy otherwise, to republish, to post on servers, or to redistribute to lists, requires prior specific permission and/or a fee. Request permissions from Publications Dept., ACM, Inc., fax +1 (212) 869-0481, or permissions@acm.org.

The following applies to all SpringerLink papers listed above that have Springer Science+Business Media copyrights: The original publication is available at www.springerlink.com.

This page was automatically generated using BibDB and bib2web.

Last modified: 2026-03-05