Federated and Decentralized Learning

Instead of collecting everyone's private data onto one central supercomputer, send the model to edge devices, train locally, and only exchange mathematical updates.

Federated learning coordinates model updates through a central hub, while decentralized learning uses peer-to-peer consensus across network graphs.

Why Does This Exist?

Traditional deep learning pools all training data onto a central cluster or cloud datacenter. In modern domains—such as next-word prediction on mobile smartphones, medical diagnosis across competing hospitals, or financial fraud detection across commercial banks—centralizing raw data is legally prohibited by privacy statutes (such as GDPR, CCPA, and HIPAA) or commercially unacceptable due to trade secrets.

Moreover, streaming gigabytes of raw sensory data from millions of edge devices creates unsustainable network bandwidth bottlenecks and consumes excessive device battery.

Federated Learning (FL) reverses the computation paradigm: it brings the code to the data rather than the data to the code. Edge devices compute local gradient steps on their private datasets and upload only lightweight parameter deltas to an aggregation coordinator. Fully Decentralized Learning takes this a step further by removing the central coordinator entirely, diffusing knowledge across peer-to-peer mesh networks via gossip consensus algorithms.

Think of It Like This

A league of master chefs refining a secret sauce

Imagine ten independent restaurant chefs across different countries trying to perfect a universal pasta sauce recipe. None of the chefs are willing to share their private supplier lists, proprietary spice quantities, or confidential family heritage with a central food corporation.

Instead, an organizer sends the baseline recipe card to all ten kitchens. Each chef cooks the recipe locally, tests it with their local customers, and writes down small recommended adjustments: "add two pinches of basil, reduce salt by one gram."

The chefs mail only their adjustment cards to an impartial clerk who averages the recommendations and publishes Revision 2. In a fully decentralized kitchen league, there is no clerk at all: chefs simply phone their two nearest neighboring restaurants after every evening service and average their notes directly.

How It Actually Works

Hub-and-Spoke Aggregation vs Peer-to-Peer Consensus

In a standard centralized Federated Learning (Hub-and-Spoke) system, $K$ clients participate under the direction of a central aggregation server. Client $k$ possesses private dataset $\mathcal{D}_k$ containing $n_k$ samples, with total dataset size $n = \sum_{k=1}^K n_k$ .

The objective is to minimize the global empirical risk:

\min_w f(w) = \sum_{k=1}^K \frac{n_k}{n} F_k(w) \quad \text{where} \quad F_k(w) = \frac{1}{n_k} \sum_{i \in \mathcal{D}_k} \ell(w; x_i, y_i)

The lifecycle of each federated round $t$ proceeds through four phases:

Selection: The server samples a random fraction $C \in (0, 1]$ of clients ( $S_t \subseteq \{1, \dots, K\}$ ).
Broadcast: The server transmits current global parameters $w_t$ to each chosen client.
Local Training: Each client initializes local model $w_k^0 = w_t$ and runs $E$ epochs of SGD over local minibatches of size $B$ : $w_k^{j+1} = w_k^j - \eta \nabla F_k(w_k^j)$ yielding total local update $\Delta w_k = w_k^E - w_t$ .
Aggregation: Selected clients transmit updates back to the server (often protected by Secure Aggregation protocols or Differential Privacy noise). The server computes the weighted average: $w_{t+1} = w_t + \sum_{k \in S_t} \frac{n_k}{\sum_{j \in S_t} n_j} \Delta w_k$

In Fully Decentralized Learning (Peer-to-Peer), there is no central server. Clients are nodes in a communication graph $\mathcal{G} = (\mathcal{V}, \mathcal{E})$ . Each node $i$ maintains its own local parameters $x_i$ and communicates solely with its direct neighbors $\mathcal{N}_i = \{j \mid (i, j) \in \mathcal{E}\}$ .

Decentralized consensus is governed by a doubly-stochastic mixing matrix $W \in \mathbb{R}^{K \times K}$ satisfying $\sum_j W_{ij} = 1$ , $\sum_i W_{ij} = 1$ , and $W_{ij} = 0$ if $(i, j) \notin \mathcal{E}$ . At each iteration $t$ :

x_i^{t+1} = \sum_{j \in \mathcal{N}_i \cup \{i\}} W_{ij} x_j^t - \eta \nabla F_i(x_i^t)

Decentralized algorithms (such as D-PSGD and EXTRA) eliminate single-point server failures, avoid high-concurrency bandwidth bottlenecks at central datacenters, and scale robustly across dynamic peer topologies.

Worked Example

Trace one round of Federated Averaging across 3 hospital nodes with unequal patient counts:

Total parameters: 2 weights $w = [w_1, w_2]$ .
Global initialization: $w_0 = [10.0, 5.0]$ .
Node sample counts: Hospital 1 has $n_1 = 100$ , Hospital 2 has $n_2 = 300$ , Hospital 3 has $n_3 = 600$ . Total $n = 1000$ .
Client weights: $p_1 = 0.10, p_2 = 0.30, p_3 = 0.60$ .

Each hospital runs local SGD on its private records, arriving at:

Hospital 1: $w_1^{\text{local}} = [10.5, 4.0] \implies \Delta w_1 = [+0.5, -1.0]$
Hospital 2: $w_2^{\text{local}} = [9.0, 6.0] \implies \Delta w_2 = [-1.0, +1.0]$
Hospital 3: $w_3^{\text{local}} = [11.0, 5.5] \implies \Delta w_3 = [+1.0, +0.5]$

The server aggregates the updates using sample weighting:

\Delta w_{\text{global}} = 0.10 \begin{bmatrix} +0.5 \\ -1.0 \end{bmatrix} + 0.30 \begin{bmatrix} -1.0 \\ +1.0 \end{bmatrix} + 0.60 \begin{bmatrix} +1.0 \\ +0.5 \end{bmatrix}

\Delta w_{\text{global}} = \begin{bmatrix} 0.05 \\ -0.10 \end{bmatrix} + \begin{bmatrix} -0.30 \\ 0.30 \end{bmatrix} + \begin{bmatrix} 0.60 \\ 0.30 \end{bmatrix} = \begin{bmatrix} 0.35 \\ 0.50 \end{bmatrix}

The updated global model for round 1 is:

w_1 = w_0 + \Delta w_{\text{global}} = \begin{bmatrix} 10.0 + 0.35 \\ 5.0 + 0.50 \end{bmatrix} = \begin{bmatrix} 10.35 \\ 5.50 \end{bmatrix}

Hospital 3 had the largest dataset ( $60\%$ ), so its local trajectory heavily steers the global consensus.

Code

1import torch2import torch.nn as nn3from typing import List, Dict4
5class SimpleNet(nn.Module):6    def __init__(self) -> None:7        super().__init__()8        self.fc = nn.Linear(2, 1, bias=False)9        self.fc.weight.data = torch.tensor([[10.0, 5.0]])10
11    def forward(self, x: torch.Tensor) -> torch.Tensor:12        return self.fc(x)13
14def federated_averaging(15    global_model: nn.Module,16    client_weights: List[Dict[str, torch.Tensor]],17    client_sample_sizes: List[int]18) -> None:19    """Aggregate client weights weighted by sample sizes."""20    total_samples = sum(client_sample_sizes)21    fractions = [n / total_samples for n in client_sample_sizes]22
23    # Initialize zeroed accumulator24    aggregated_dict = {25        key: torch.zeros_like(param)26        for key, param in global_model.state_dict().items()27    }28
29    # Weighted sum of client state dicts30    for client_state, frac in zip(client_weights, fractions):31        for key in aggregated_dict:32            aggregated_dict[key] += frac * client_state[key]33
34    # Load aggregated weights into global model35    global_model.load_state_dict(aggregated_dict)36
37# Setup 3 client models with local results38client_states = [39    {"fc.weight": torch.tensor([[10.5, 4.0]])},40    {"fc.weight": torch.tensor([[9.0, 6.0]])},41    {"fc.weight": torch.tensor([[11.0, 5.5]])}42]43sample_counts = [100, 300, 600]44
45server_model = SimpleNet()46federated_averaging(server_model, client_states, sample_counts)47
48print("Aggregated Global Weights:")49print(server_model.fc.weight.data)50# -> Aggregated Global Weights:51# -> tensor([[10.3500,  5.5000]])

Watch Out For

Data leakage via gradient inversion attacks

While federated learning avoids transmitting raw data, transmitting raw gradients $\nabla w_k$ is not inherently private. Malicious servers or sniffing adversaries can run gradient inversion (such as Deep Leakage from Gradients / DLG), optimizing synthetic inputs until their calculated gradients match the client's uploaded vector, reconstructing private images and text verbatim.

To prevent reconstruction attacks, always combine federated aggregation with Differential Privacy (DP-SGD)—clipping gradient norms and adding calibrated Gaussian noise—or utilize cryptographic Secure Multi-Party Computation (SMPC) where the server can only decrypt the sum of client updates, never any individual client's gradient.

The Quick Version

Keeps raw training data decentralized on edge devices, transmitting only model parameter updates or gradients.
Centralized federated learning uses a hub-and-spoke coordinator with sample-weighted aggregation rounds.
Decentralized peer-to-peer learning eliminates server bottlenecks by exchanging model states across graph neighbors using doubly-stochastic mixing matrices.