Piecewise Functions Like ReLU

A piecewise function switches recipes at boundaries: one rule here, another there. ReLU, tax brackets, shipping rates and robust losses all switch this way.

ReLU lies flat at zero left of the origin then climbs as y equals x to the right, with one kink at zero where the two recipes meet

Why Does This Exist?

Many rules change at a boundary. Income tax takes 10% up to a threshold, 20% above it. Shipping is flat to 5kg, per-kilo after. ReLU outputs zero for negatives, the input itself for positives. Each is one function with two recipes, and the boundary is part of the definition, not an afterthought.

The kink where recipes meet is the interesting point: the function stays continuous but its slope jumps, so derivatives need special handling exactly there. Robust losses switch recipes to limit outlier damage in common loss functions; slopes everywhere else are derivatives and partial derivatives.

The one idea you need first. A function gives each input exactly one output. Piecewise keeps that contract by assigning every input, including boundary points, to exactly one recipe. Overlaps and gaps at the joint are the only new way to break the contract.

Think of It Like This

Shipping rates with a weight threshold

A courier charges a flat $\$ 8 $up to 5kg, then$ $8 $plus$ $2 $per kilo above. A 3kg parcel costs$ $8 $; a 7kg parcel costs$ $12$. The price function has two recipes with a joint at 5kg, and the joint belongs to exactly one of them, or the 5kg parcel gets two prices.

ReLU is a courier with a free tier: nothing below zero, full price above. The kink at zero is the warehouse door where the pricing changes. Every piecewise function is a rate card; the boundaries are the doors.

Where it stops: couriers round and cap, while math joints are exact down to the decimal. The analogy shows the switching; the definition below shows the precision.

How It Actually Works

Write each recipe with its interval: ReLU is $0$ for $x < 0$ and $x$ for $x \ge 0$ . The boundary $x = 0$ belongs to the second recipe, so every input has exactly one rule. Absolute value is the symmetric twin: $-x$ for $x < 0$ , $x$ for $x \ge 0$ .

Evaluate by routing: test the input against the boundaries, apply the matching recipe. At the kink the slope jumps: ReLU's derivative is $0$ left of zero and $1$ right of it, undefined exactly at zero, where frameworks simply pick a side.

Worked example: ReLU values and a two-bracket tax

ReLU: $\text{ReLU}(-3) = 0$ (negative recipe), $\text{ReLU}(2.5) = 2.5$ (positive recipe), $\text{ReLU}(0) = 0$ (boundary belongs to the second rule). Absolute value: $|-4| = -(-4) = 4$ .

Tax: 10% on income up to $50,000, 20% above. On $60,000: $50000 \times 0.10 = 5000$ plus $10000 \times 0.20 = 2000$ , total $7,000. Each bracket taxes only its slice, which is why the marginal rate is not the average rate.

Code

1def relu(x: float) -> float:2    return x if x >= 0.0 else 0.03
4
5def abs_val(x: float) -> float:6    return -x if x < 0.0 else x7
8
9print(relu(-3.0), relu(2.5), relu(0.0))  # -> 0.0 2.5 0.010print(abs_val(-4.0))  # -> 4.011
12income: float = 60000.013tax: float = 0.10 * min(income, 50000.0) + 0.20 * max(income - 50000.0, 0.0)14print(tax)  # -> 7000.0

The boundary comparisons (>=, <) are the joints made executable. The tax line taxes each slice with min and max, the standard two-bracket pattern. Note relu(0.0) returns $0.0$ : the framework picked a side, as all of them must.

Watch Out For

Differentiating exactly at the kink

ReLU's slope is $0$ from the left and $1$ from the right at zero, so no single derivative exists there. Autograd picks one silently (usually 0), and a unit sitting exactly at zero receives a gradient that is convention, not math. Networks rarely notice; hand-derived proofs that assume differentiability break on it.

Treat kinks as special cases in any derivation. In code, know your framework's convention and never assert gradient values at exact boundaries.

Double-counting the joint with overlapping rules

Rules $x \le 0$ and $x \ge 0$ both claim $x = 0$ : two recipes, one input, contract broken. Bucketing code with <= on both sides counts boundary samples twice, inflating the edge bin. The symptom is a histogram spike or a tax total off by one bracket's worth.

Make joints half-open: one side <, the other >=. Audit every boundary comparison in pairs, never singly.

The Quick Version

One function, several recipes, each owning its interval; boundaries belong to exactly one recipe.
ReLU: $0$ below zero, $x$ at and above; $\text{ReLU}(-3) = 0$ , $\text{ReLU}(2.5) = 2.5$ .
Kinks keep continuity but break differentiability; frameworks pick a side silently.
Tax each bracket's slice only; marginal rate is not average rate.
Write joints half-open (< plus >=) so no input gets two recipes.