A multi-layer network trains without errors but its accuracy lands exactly where the logistic-regression baseline sits — and stays there. Adding more layers changes nothing. On XOR-style or curved-boundary data the score hovers near a coin flip while a hand-drawn curve would separate the classes easily. No warning, no crash: the model quietly refuses to be deep.
First hand-built networks in module 2.1, and early PyTorch models in 2.3 where an
activation was deleted during a refactor or never added between two Linear layers.
Especially common when layers are composed in a loop and the activation call sits
outside the loop by mistake. The code reads as "five layers", so the builder trusts
the depth and hunts for problems in the data instead. Linked module: 2.1.
Stacked linear layers with nothing between them collapse into a single matrix:
X @ W1 @ W2 equals X @ (W1 @ W2) by plain algebra, so any depth of
activation-free layers is one linear model in disguise. A linear model can only
draw a straight boundary, which is why the score pins to the linear baseline.
Confirm in two minutes: multiply all the weight matrices together into one, run the
single-matrix model, and compare predictions — if they match the "deep" network
prediction for prediction, the collapse is proven. Then audit the forward pass line
by line and list where a bend (ReLU, sigmoid, tanh) actually executes; the gap will
be visible.
Insert an activation between every pair of consecutive layers — after each hidden
layer's x @ W + b, before the next layer reads it. In NumPy that is one
np.maximum(0, z) per hidden layer; in PyTorch, one nn.ReLU() between the
Linear modules (a common miss is two Linear calls back to back inside
Sequential). Rerun the collapse check afterwards: the multiplied-out single
matrix must now disagree with the network, and on bent-boundary data the accuracy
should climb clear of the linear baseline.
Make the activation audit a habit: for any network, count layers and count activations — hidden layers and activations must pair one to one. Keep 2.1's collapse demo as a mental unit test: if you can write your network as one matrix multiply, it is not deep. When a deep model exactly ties a linear baseline, suspect architecture before data — that specific tie is this failure's signature.
Q: "A candidate's five-layer network scores identical to logistic regression on a
clearly non-linear problem. What do you check first?" Strong answer: missing
activations — state the collapse X @ W1 @ W2 = X @ (W1 @ W2) in one line, propose
the multiply-the-matrices test as proof, and note that a deep-equals-linear tie is
the tell. Bonus: mention that this is why the activation function, not the layer
count, is what makes a network deep.