Why Bias Matters
A ReLU network with no bias terms stalls at about 50% on the three-class spiral no matter how wide you make it. Turn the biases on and the same network gets about 98%. Five steps, each one interactive, show why.
class 0class 1class 2
One hidden unit is a hinge over a line
A hidden unit computes relu(w·x + b). The points where w·x + b = 0 form a straight line. On one side the unit outputs zero; on the other it ramps up with distance from the line.
Drag the angle to rotate the line. Drag the bias to slide it. Then lock the bias at zero and try to move the line anywhere that doesn't pass through the origin.
The bias is the only knob that moves the line off the origin.
Without bias, every line goes through the origin
A hidden layer is many such units. Each one draws its own line, and the network's decision regions are stitched together from the pieces those lines cut out.
Here are the lines for a randomly initialized layer. Toggle the bias. Without it, every line is pinned to the origin and the plane is cut into wedges. Adding more units adds more wedges, never a line somewhere else.
Width adds lines. Only bias lets a line leave the origin.
No bias means the logits scale along a ray
Scale the input by a positive c. The first matrix product scales by c; ReLU keeps the same entries positive, so its output scales by c; the second matrix product scales by c again.
Move your pointer over the plane. The right plot shows the three logits as you walk out from the origin along that ray. Without bias they are straight lines through zero, so the largest one stays largest and the whole ray gets one class. With bias they bend and cross.
A bias-free network classifies directions, not points.
The spiral puts several classes on one ray
The data is the toy spiral from the Stanford CS231n neural network case study, a standard example of data a linear classifier can't separate. Each arm sweeps about 4 radians of angle while its radius grows from 0 to 1. Three arms wind almost twice around the origin.
Move the pointer. The points within a narrow angular band of that ray light up, and the strip below lays them out by radius. Nearly every direction crosses two or three classes at different radii.
The class depends on radius at a fixed angle. That is exactly the information a bias-free network throws away, and the best angle-only rule scores about 54%.
Train it and watch the regions
The same two-layer ReLU MLP, trained live with minibatch SGD on the spiral and scored on a fresh spiral it never saw. Toggle the bias and reset.
Without bias the regions stay wedges meeting at the origin and held-out accuracy stalls around 0.5. With bias the boundaries wrap around the arms. Turn on the hidden unit lines to see them pinned or free.
Same data, same width, same optimizer. The only difference is whether a boundary may leave the origin.