Neural Network Layers as Matrices
Everything we have learned about matrices comes together here. Neural networks are built from layers, and each layer is fundamentally a matrix operation. When you hear that a model has "billions of parameters," those parameters live in matrices, and every prediction is a chain of matrix operations.
How a Single Neuron Works
A single neuron takes inputs, multiplies each by a weight, sums them up, and adds a bias term.
inputs: [x1, x2, x3]
weights: [w1, w2, w3]
bias: b
output = (w1 * x1) + (w2 * x2) + (w3 * x3) + b
You may recognize the first part as a dot product from Module 1. A neuron computes the dot product of the input vector and its weight vector, then adds a bias. One neuron does one dot product -- but a layer has many neurons, and that is where matrices come in.
A Layer as a Matrix Operation
A layer is a group of neurons that all receive the same inputs but have their own weights. Instead of computing each dot product separately, we pack all weights into a single matrix:
output = activation(W * input + b)
- W is the weight matrix
- input is the input vector
- b is the bias vector
- activation is a function like ReLU or sigmoid
Weight Matrix Shape
The weight matrix shape tells you the layer's architecture:
- Rows = number of output neurons
- Columns = number of input neurons
A layer taking 3 inputs and producing 2 outputs has a 2x3 weight matrix. Each row holds all the weights for one output neuron.
Concrete Example: 3 Inputs to 2 Outputs
Let's walk through a complete forward pass for a single layer.
Input vector (3 elements): x = [1.0, 0.5, 2.0]
Weight matrix (2x3): W = | 0.4 0.8 -0.2 |
| 0.1 -0.5 0.3 |
Bias vector (2 elements): b = [0.1, 0.2]
Step 1: Matrix-vector multiplication (W * x)
Row 1: (0.4 * 1.0) + (0.8 * 0.5) + (-0.2 * 2.0) = 0.4 + 0.4 - 0.4 = 0.4
Row 2: (0.1 * 1.0) + (-0.5 * 0.5) + (0.3 * 2.0) = 0.1 - 0.25 + 0.6 = 0.45
W * x = [0.4, 0.45]
Step 2: Add the bias vector
W * x + b = [0.4 + 0.1, 0.45 + 0.2] = [0.5, 0.65]
Step 3: Apply activation function
Using ReLU (keeps positive values, replaces negatives with zero):
ReLU([0.5, 0.65]) = [0.5, 0.65] (both positive, no change)
The final output is [0.5, 0.65]. These 2 values become the input to the next layer.
Bias Vectors
The bias vector b has one element per output neuron. It shifts each neuron's output independently of the inputs. Without bias, a neuron with all-zero inputs would always output zero. The bias gives the network more flexibility.
Forward Pass: Chaining Layers
A deep network applies multiple layers in sequence. The output of each layer feeds into the next:
Layer 1: h1 = activation(W1 * input + b1)
Layer 2: h2 = activation(W2 * h1 + b2)
Layer 3: output = activation(W3 * h2 + b3)
For a network with 3 inputs, a hidden layer of 4 neurons, and 2 outputs:
| Layer | Weight Shape | Bias Shape | Input | Output |
|---|---|---|---|---|
| 1 | 4x3 | 4 | 3 | 4 |
| 2 | 2x4 | 2 | 4 | 2 |
The entire forward pass is just a chain of matrix multiplications, vector additions, and activation functions.
Why This Representation Is Powerful
Parallelism -- matrix operations split across thousands of GPU cores. An entire layer computes in a single operation instead of one neuron at a time.
Batching -- stack multiple inputs into an input matrix and process them all at once. A batch of 32 samples with 3 features becomes a 32x3 matrix, and all 32 predictions happen simultaneously.
Optimization -- libraries like cuBLAS and cuDNN perform billions of matrix operations per second, built on decades of numerical linear algebra research.
Deep Networks as Chains of Matrix Operations
A 100-layer network is a chain of 100 matrix operations, each transforming data into a different representation:
Raw pixels -> edges -> shapes -> objects -> classification
(W1) (W2) (W3) (W4) (W5)
Each arrow is a matrix multiplication, bias addition, and activation function. The entire intelligence of the network is encoded in the numbers within these weight matrices, learned through the gradient updates we covered in the previous lesson.
Summary
- A single neuron computes a dot product of inputs and weights, then adds a bias
- A layer packs neuron weights into a weight matrix (rows = outputs, columns = inputs)
- The core equation is output = activation(W * input + b)
- The bias vector gives each neuron an adjustable baseline
- A forward pass chains layers, each feeding its output to the next
- Matrix operations enable parallelism, batching, and optimized GPU computation
- Deep networks are chains of matrix operations that progressively transform data
In Module 3, we will dive into matrix multiplication itself -- the most important single operation in all of AI. You will learn exactly how it works, why it is computationally expensive, and how it powers the attention mechanism in transformers like GPT and BERT.