OnDev-LCT: On-Device Lightweight Convolutional Transformers towards federated learning.

Federated learning (FL) has emerged as a promising approach to collaboratively train machine learning models across multiple edge devices while preserving privacy. The success of FL hinges on the efficiency of participating models and their ability to handle the unique challenges of distributed learning. While several variants of Vision Transformer (ViT) have shown great potential as alternatives to modern convolutional neural networks (CNNs) for centralized training, the unprecedented size and higher computational demands hinder their deployment on resource-constrained edge devices, challenging their widespread application in FL. Since client devices in FL typically have limited computing resources and communication bandwidth, models intended for such devices must strike a balance between model size, computational efficiency, and the ability to adapt to the diverse and non-IID data distributions encountered in FL. To address these challenges, we propose OnDev-LCT: Lightweight Convolutional Transformers for On-Device vision tasks with limited training data and resources. Our models incorporate image-specific inductive biases through the LCT tokenizer by leveraging efficient depthwise separable convolutions in residual linear bottleneck blocks to extract local features, while the multi-head self-attention (MHSA) mechanism in the LCT encoder implicitly facilitates capturing global representations of images. Extensive experiments on benchmark image datasets indicate that our models outperform existing lightweight vision models while having fewer parameters and lower computational demands, making them suitable for FL scenarios with data heterogeneity and communication bottlenecks.

Contrastive encoder pre-training-based clustered federated learning for heterogeneous data.

Federated learning (FL) is a promising approach that enables distributed clients to collaboratively train a global model while preserving their data privacy. However, FL often suffers from data heterogeneity problems, which can significantly affect its performance. To address this, clustered federated learning (CFL) has been proposed to construct personalized models for different client clusters. One effective client clustering strategy is to allow clients to choose their own local models from a model pool based on their performance. However, without pre-trained model parameters, such a strategy is prone to clustering failure, in which all clients choose the same model. Unfortunately, collecting a large amount of labeled data for pre-training can be costly and impractical in distributed environments. To overcome this challenge, we leverage self-supervised contrastive learning to exploit unlabeled data for the pre-training of FL systems. Together, self-supervised pre-training and client clustering can be crucial components for tackling the data heterogeneity issues of FL. Leveraging these two crucial strategies, we propose contrastive pre-training-based clustered federated learning (CP-CFL) to improve the model convergence and overall performance of FL systems. In this work, we demonstrate the effectiveness of CP-CFL through extensive experiments in heterogeneous FL settings, and present various interesting observations.

Losing a jewel-Rapid declines in Myanmar's intact forests from 2002-2014.

New and rapid political and economic changes in Myanmar are increasing the pressures on the country's forests. Yet, little is known about the past and current condition of these forests and how fast they are declining. We mapped forest cover in Myanmar through a consortium of international organizations and environmental non-governmental groups, using freely-available public domain data and open source software tools. We used Landsat satellite imagery to assess the condition and spatial distribution of Myanmar's intact and degraded forests with special focus on changes in intact forest between 2002 and 2014. We found that forests cover 42,365,729 ha or 63% of Myanmar, making it one of the most forested countries in the region. However, severe logging, expanding plantations, and degradation pose increasing threats. Only 38% of the country's forests can be considered intact with canopy cover >80%. Between 2002 and 2014, intact forests declined at a rate of 0.94% annually, totaling more than 2 million ha forest loss. Losses can be extremely high locally and we identified 9 townships as forest conversion hotspots. We also delineated 13 large (>100,000 ha) and contiguous intact forest landscapes, which are dispersed across Myanmar. The Northern Forest Complex supports four of these landscapes, totaling over 6.1 million ha of intact forest, followed by the Southern Forest Complex with three landscapes, comprising 1.5 million ha. These remaining contiguous forest landscape should have high priority for protection. Our project demonstrates how open source data and software can be used to develop and share critical information on forests when such data are not readily available elsewhere. We provide all data, code, and outputs freely via the internet at (for scripts: https://bitbucket.org/rsbiodiv/; for the data: http://geonode.themimu.info/layers/geonode%3Amyan_lvl2_smoothed_dec2015_resamp).