Multi-Task Learning

Why Does This Exist?

A self-driving perception system needs to detect other vehicles, segment the drivable road surface, and estimate depth from the same camera frame, all at once, on hardware with a strict compute budget. Training three completely separate networks — one per task — triples the parameter count, triples the inference cost, and, less obviously, throws away something useful: a network that's good at detecting vehicle edges has probably learned features that would also help segment road boundaries, since both tasks depend on similar low-level structure in the same images.

Multi-task learning trains one shared network on all three tasks simultaneously, so whatever the vehicle-detection gradient teaches the shared layers is available to the segmentation and depth tasks too, and vice versa — for one inference cost instead of three.

Think of It Like This

How It Actually Works

Hard parameter sharing: one trunk, several heads

The most common architecture is direct: a shared trunk of layers processes the input the same way regardless of which task is being trained on that step, and each task has its own small head attached to the trunk's output. During training, a batch belonging to task A computes task A's loss, backpropagates through task A's head and then through the shared trunk — updating the trunk based on task A's gradient. The next batch might belong to task B, and updates the same trunk again, from a different gradient. Over many steps, the trunk ends up shaped by every task's signal combined.

Why sharing helps: each task acts as a regularizer on the others

A single-task network trained on limited data can overfit to quirks specific to that dataset. A shared trunk trained across several tasks simultaneously has less freedom to fit any one task's quirks, because it also has to remain useful for the other tasks sharing it — a form of regularization that comes for free from the multi-task setup itself, not from any explicit penalty term. This is also why an auxiliary task — a secondary task added purely to help the main one, with nobody caring about its own output quality — is a standard trick: it exists only to shape the shared trunk into something more broadly useful.

Negative transfer: when tasks fight instead of help

Sharing only helps when the tasks are related enough that a representation good for one is at least somewhat good for the others. When two tasks pull the shared trunk in genuinely conflicting directions, one task's gradient step can actively undo progress the other task just made — negative transfer. This is visible directly in gradient geometry: the cosine similarity between two tasks' gradients on the shared trunk tells you whether they're pulling together or against each other. A cosine near 1 means they reinforce; a cosine near $-1$ means the shared update, on average, works against at least one of them.

Task weighting: not every task deserves an equal vote

Simply summing every task's loss implicitly gives more influence to whichever task's loss happens to have a larger scale or noisier gradient — an accident of units, not a deliberate choice. In practice, each task's contribution to the combined loss is usually weighted, either by hand-tuned constants or by a scheme that adapts the weights during training based on how each task's loss is progressing, so that one loud task doesn't drown out a quieter but equally important one.

Show Me the Code

Two tasks' gradients on a shared trunk, one pair aligned and one pair conflicting, and what that does to the shared update.

1import numpy as np2
3
4def cosine(a: np.ndarray, b: np.ndarray) -> float:5    return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))6
7
8grad_task_a = np.array([1.0, 0.5, -0.2])9grad_task_b_aligned = np.array([0.9, 0.6, -0.1])10grad_task_b_conflict = np.array([-0.8, 0.2, 0.9])11
12for name, grad_b in (("aligned", grad_task_b_aligned), ("conflicting", grad_task_b_conflict)):13    shared_update = grad_task_a + grad_b14    print(f"{name}: cosine(A, B) = {cosine(grad_task_a, grad_b):.3f}, "15          f"shared update still helps A: {cosine(shared_update, grad_task_a):.3f}")16# -> aligned: cosine(A, B) = 0.989, shared update still helps A: 0.99717# -> conflicting: cosine(A, B) = -0.635, shared update still helps A: 0.357

When the two tasks' gradients are nearly aligned, the shared update stays almost perfectly aimed at helping task A. When they conflict, the shared update's alignment with task A collapses from a perfect 1.0 down to 0.357 — a visible, measurable version of negative transfer, not just a hand-wavy concern.

Watch Out For

The Quick Version

• Hard parameter sharing trains one shared trunk across several tasks, with a separate small head per task, for the cost of one shared network instead of several separate ones.
• Sharing acts as a form of regularization, since the trunk has to remain useful across every task rather than overfitting to one.
• Negative transfer happens when tasks pull the shared trunk in conflicting directions, visible directly as negative cosine similarity between their gradients.
• Task weighting matters because an unweighted loss sum lets whichever task's loss has the largest scale dominate the combined gradient.

What to Read Next

• Transfer Learning trains tasks sequentially rather than simultaneously, and is worth comparing directly against a multi-task setup.
• Meta-Learning also trains across many tasks, but optimizes for fast adaptation to a new task rather than joint performance across a fixed set.
• Domain Adaptation covers a related but distinct problem: one task under a shifted input distribution, rather than several genuinely different tasks.
• Overfitting and Underfitting is relevant to why multi-task sharing's regularizing effect works in the first place.