Multi-Task Learning
Train one shared network on several related tasks at once, so each task's gradient shapes the same trunk and the shared layers help every head, not just one.
Why Does This Exist?
A driving perception stack must detect vehicles, segment road, and estimate depth from the same frame on one computer. Three separate networks triple weights and triple inference cost. Worse, each network relearns edges and shapes the others already know, since all three tasks read the same pixels.
Multi-task learning trains one shared trunk on every task at once. Each batch updates its own small head plus the common trunk, so vehicle gradients improve the road features and road gradients improve the depth features. This page covers the shared trunk pattern, when sharing helps, and when tasks fight. The reuse-across-models story lives in transfer learning. The learn-without-labels story lives in self-supervised learning. The learn-by-comparison story lives in contrastive learning.
Two basics first. A trunk is the stack of layers every task shares. A head is the small output layer owned by one task. Overfitting means fitting quirks of one dataset that fail elsewhere, covered in overfitting and underfitting.
Think of It Like This
One kitchen training three cooks
Three trainee cooks share one kitchen and one set of knives, pans, and stoves. Each cook still owns one dish: soup, bread, and pastry. Every soup shift sharpens the shared knives, every bread shift seasons the shared pans, and every pastry shift calibrates the shared oven, so all three dishes improve from work none of them did alone.
The kitchen is the shared trunk, each dish is a task head, and knife sharpening is a gradient step. Where it stops: cooks can agree to clean up after themselves, while task gradients cannot negotiate. When soup needs a hotter oven than pastry survives, sharing the oven hurts one dish, which is negative transfer.
How It Actually Works
The shared update rule
Call the shared weights , meaning every layer before the heads. Call the gradient of task A loss on , meaning the direction task A wants to move. Call the same for task B. One joint step moves along . Cosine similarity between and tells whether that sum still serves A: near 1 means the tasks pull together, near minus 1 means B partly undoes A.
Worked example: aligned pair against conflicting pair
Fix . Take an aligned partner and a conflicting partner . Norms first:
Dots next. Aligned dot is . Conflicting dot is . Divide:
Now the shared updates and . Their alignment with is for the aligned pair and for the conflicting pair. Same trunk, same arithmetic: one partner preserves the task direction, the other collapses it. Those four numbers sit in the diagram chips.
Why sharing regularizes, and when it breaks
A trunk that must serve three tasks cannot memorize quirks of one, because the other two keep pulling it back toward shared structure. An auxiliary task uses this on purpose: nobody ships its output, it exists only to shape the trunk. Sharing breaks when tasks need different features, which shows up as persistently negative cosine or as every task scoring below its single-task baseline. Related-task transfer without joint training is the alternative in domain adaptation.
Weighting the losses
Losses have different scales, so a plain sum lets the loudest loss steer the trunk. Teams weight each task loss by hand or by uncertainty and gradient size during training, then check per-task scores rather than the summed number.
Code
The point is the check: rebuild both cosine pairs and both shared-update alignments from the worked example.
import numpy as np
def cosine(a: np.ndarray, b: np.ndarray) -> float: return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))
grad_a = np.array([1.0, 0.5, -0.2])grad_b_aligned = np.array([0.9, 0.6, -0.1])grad_b_conflict = np.array([-0.8, 0.2, 0.9])
for name, grad_b in (("aligned", grad_b_aligned), ("conflicting", grad_b_conflict)): shared = grad_a + grad_b print(f"{name}: cosine(A, B) = {cosine(grad_a, grad_b):.3f}, shared vs A = {cosine(shared, grad_a):.3f}")# -> aligned: cosine(A, B) = 0.989, shared vs A = 0.997# -> conflicting: cosine(A, B) = -0.635, shared vs A = 0.357The printout matches the hand computation: 0.989 with 0.997 against minus 0.635 with 0.357. Swap one sign in the conflicting gradient and watch the second alignment climb back toward 1. That sensitivity is the diagnostic teams run before adding a head.
Watch Out For
Adding heads without a single-task baseline
Every new head looks productive because total loss falls while shared features improve early. Then one task quietly degrades past its solo score and nobody notices under the summed number.
Symptom: the joint model beats each solo model on average but loses on the one task that matters, often with negative gradient cosine for weeks. Fix: train each task alone first, keep those scores, and drop or reweight any head that cannot beat its own baseline after tuning.
Letting loss scale pick the winner
A depth loss in the hundreds next to a classification loss near 1 steers almost every shared step, regardless of business priority. The trunk becomes a depth network with passengers.
Symptom: one task improves each week while the others stall, and its gradient norm dwarfs the rest by ten times or more. Fix: scale each loss deliberately, log per-task gradient norms on the trunk, and judge by per-task validation scores rather than the summed loss.
The Quick Version
- Multi-task learning trains one shared trunk plus one small head per task, so each task gradient shapes layers every task reuses.
- Aligned gradients at cosine 0.989 keep the shared update at 0.997 with task A.
- Conflicting gradients at cosine minus 0.635 drag that alignment down to 0.357 with task A.
- Sharing regularizes because the trunk cannot memorize one task without hurting the others.
- Auxiliary tasks exist only to shape the trunk, and loss weights stop the loudest task from steering every step.
- Any head that cannot beat its single-task baseline is negative transfer, not free help.