Improving the participation of resource-constrained devices in federated learning

Federated Learning (FL) has achieved huge successes in training machine learning models on remote devices. However, FL traditionally relies on devices with equal and sufficient computing capabilities. Resource-constrained devices, often termed stragglers, struggle to contribute their knowledge to FL due to limited computational resources. This issue significantly renders FL performance sub-optimal, especially when applied at large scales, as stragglers can offer unique data perspectives that other participants cannot provide. Therefore, it is important to alleviate the straggler issue by enabling the participation of resource-constrained devices that would otherwise be declared stragglers.

This research addresses the straggler issue by advancing efficient machine learning approaches to improve participation of resource-constrained devices in FL. Notably, three efficient machine learning approaches are explored in this thesis: 1) partial model training; 2) reduced-size models; 3) active data selection. Identified research gaps limiting the efficacy of existing FL works are addressed through proposed methods: 1) few-shot fine-tuning; 2) attention transfer and metadata training; 3) clustering-based and entropy-based data selection. Novel FL algorithms and worthy insights are delivered through this research, including: 1) few-shot fine-tuning not only reduces workloads on stragglers but also accelerates FL convergence, reducing energy consumption on resource-constrained devices. 2) attention transfer and metadata training enhance knowledge transfer from custom-size client models to the global model, improving the efficacy for addressing the straggler issue. 3) entropy-based data selection improves both learning efficiency and generalisation ability, achieving better FL performance with less resource consumption on small devices.

This research opens the field for other cross-strategy approaches to address the bottleneck of training large models on resource-constrained devices. In light of the emerging large language models, many current devices that generate user data are deemed stragglers if efficient machine learning approaches for FL will not be considered moving forward.