Skip to content
AI360XpertBeta

Multi-Task Learning

Train one shared network on several related tasks at once, so each task's gradient shapes the same trunk and the shared layers help every head, not just one.

One shared trunk feeds three task heads, so aligned gradients at cosine 0.989 keep the shared update at 0.997 while conflicting gradients at minus 0.635 drag it to 0.357
One shared trunk feeds three task heads, so aligned gradients at cosine 0.989 keep the shared update at 0.997 while conflicting gradients at minus 0.635 drag it to 0.357

Why Does This Exist?

A driving perception stack must detect vehicles, segment road, and estimate depth from the same frame on one computer. Three separate networks triple weights and triple inference cost. Worse, each network relearns edges and shapes the others already know, since all three tasks read the same pixels.

Multi-task learning trains one shared trunk on every task at once. Each batch updates its own small head plus the common trunk, so vehicle gradients improve the road features and road gradients improve the depth features. This page covers the shared trunk pattern, when sharing helps, and when tasks fight. The reuse-across-models story lives in transfer learning. The learn-without-labels story lives in self-supervised learning. The learn-by-comparison story lives in contrastive learning.

Two basics first. A trunk is the stack of layers every task shares. A head is the small output layer owned by one task. Overfitting means fitting quirks of one dataset that fail elsewhere, covered in overfitting and underfitting.

Think of It Like This

One kitchen training three cooks

Three trainee cooks share one kitchen and one set of knives, pans, and stoves. Each cook still owns one dish: soup, bread, and pastry. Every soup shift sharpens the shared knives, every bread shift seasons the shared pans, and every pastry shift calibrates the shared oven, so all three dishes improve from work none of them did alone.

The kitchen is the shared trunk, each dish is a task head, and knife sharpening is a gradient step. Where it stops: cooks can agree to clean up after themselves, while task gradients cannot negotiate. When soup needs a hotter oven than pastry survives, sharing the oven hurts one dish, which is negative transfer.

How It Actually Works

The shared update rule

Call the shared weights ww, meaning every layer before the heads. Call gAg_A the gradient of task A loss on ww, meaning the direction task A wants ww to move. Call gBg_B the same for task B. One joint step moves ww along gA+gBg_A + g_B. Cosine similarity between gAg_A and gBg_B tells whether that sum still serves A: near 1 means the tasks pull together, near minus 1 means B partly undoes A.

Worked example: aligned pair against conflicting pair

Fix gA=[1.0,0.5,−0.2]g_A = [1.0, 0.5, -0.2]. Take an aligned partner gB1=[0.9,0.6,−0.1]g_{B1} = [0.9, 0.6, -0.1] and a conflicting partner gB2=[−0.8,0.2,0.9]g_{B2} = [-0.8, 0.2, 0.9]. Norms first:

∥gA∥=1.29≈1.136,∥gB1∥=1.18≈1.086,∥gB2∥=1.49≈1.221\lVert g_A \rVert = \sqrt{1.29} \approx 1.136, \quad \lVert g_{B1} \rVert = \sqrt{1.18} \approx 1.086, \quad \lVert g_{B2} \rVert = \sqrt{1.49} \approx 1.221

Dots next. Aligned dot is 0.9+0.3+0.02=1.220.9 + 0.3 + 0.02 = 1.22. Conflicting dot is −0.8+0.1−0.18=−0.88-0.8 + 0.1 - 0.18 = -0.88. Divide:

cos⁡(A,B1)=1.22/(1.136×1.086)≈0.989\cos(A, B1) = 1.22 / (1.136 \times 1.086) \approx 0.989 cos⁡(A,B2)=−0.88/(1.136×1.221)≈−0.635\cos(A, B2) = -0.88 / (1.136 \times 1.221) \approx -0.635

Now the shared updates s1=gA+gB1=[1.9,1.1,−0.3]s_1 = g_A + g_{B1} = [1.9, 1.1, -0.3] and s2=gA+gB2=[0.2,0.7,0.7]s_2 = g_A + g_{B2} = [0.2, 0.7, 0.7]. Their alignment with AA is 0.9970.997 for the aligned pair and 0.3570.357 for the conflicting pair. Same trunk, same arithmetic: one partner preserves the task direction, the other collapses it. Those four numbers sit in the diagram chips.

Why sharing regularizes, and when it breaks

A trunk that must serve three tasks cannot memorize quirks of one, because the other two keep pulling it back toward shared structure. An auxiliary task uses this on purpose: nobody ships its output, it exists only to shape the trunk. Sharing breaks when tasks need different features, which shows up as persistently negative cosine or as every task scoring below its single-task baseline. Related-task transfer without joint training is the alternative in domain adaptation.

Weighting the losses

Losses have different scales, so a plain sum lets the loudest loss steer the trunk. Teams weight each task loss by hand or by uncertainty and gradient size during training, then check per-task scores rather than the summed number.

Code

The point is the check: rebuild both cosine pairs and both shared-update alignments from the worked example.

import numpy as np

def cosine(a: np.ndarray, b: np.ndarray) -> float:    return float(a @ b / (np.linalg.norm(a) * np.linalg.norm(b)))

grad_a = np.array([1.0, 0.5, -0.2])grad_b_aligned = np.array([0.9, 0.6, -0.1])grad_b_conflict = np.array([-0.8, 0.2, 0.9])
for name, grad_b in (("aligned", grad_b_aligned), ("conflicting", grad_b_conflict)):    shared = grad_a + grad_b    print(f"{name}: cosine(A, B) = {cosine(grad_a, grad_b):.3f}, shared vs A = {cosine(shared, grad_a):.3f}")# -> aligned: cosine(A, B) = 0.989, shared vs A = 0.997# -> conflicting: cosine(A, B) = -0.635, shared vs A = 0.357

The printout matches the hand computation: 0.989 with 0.997 against minus 0.635 with 0.357. Swap one sign in the conflicting gradient and watch the second alignment climb back toward 1. That sensitivity is the diagnostic teams run before adding a head.

Watch Out For

Adding heads without a single-task baseline

Every new head looks productive because total loss falls while shared features improve early. Then one task quietly degrades past its solo score and nobody notices under the summed number.

Symptom: the joint model beats each solo model on average but loses on the one task that matters, often with negative gradient cosine for weeks. Fix: train each task alone first, keep those scores, and drop or reweight any head that cannot beat its own baseline after tuning.

Letting loss scale pick the winner

A depth loss in the hundreds next to a classification loss near 1 steers almost every shared step, regardless of business priority. The trunk becomes a depth network with passengers.

Symptom: one task improves each week while the others stall, and its gradient norm dwarfs the rest by ten times or more. Fix: scale each loss deliberately, log per-task gradient norms on the trunk, and judge by per-task validation scores rather than the summed loss.

The Quick Version

  • Multi-task learning trains one shared trunk plus one small head per task, so each task gradient shapes layers every task reuses.
  • Aligned gradients at cosine 0.989 keep the shared update at 0.997 with task A.
  • Conflicting gradients at cosine minus 0.635 drag that alignment down to 0.357 with task A.
  • Sharing regularizes because the trunk cannot memorize one task without hurting the others.
  • Auxiliary tasks exist only to shape the trunk, and loss weights stop the loudest task from steering every step.
  • Any head that cannot beat its single-task baseline is negative transfer, not free help.