What is a matrix? | Derek's Notes

1. Many rules, one table

Put mathematics aside for a moment. Imagine you have taken over a bubble tea shop. Your menu has three signature drinks, each with its own recipe using the same four ingredients: tea, milk, sugar and pearls.

Here are the recipes, with numbers representing portions:

Pearl milk tea: 2 portions of tea + 1 of milk + 0 of sugar + 3 of pearls.
Milk pudding: 0 portions of tea + 4 of milk + 1 of sugar + 2 of pearls.
Traditional sweet tea: 1 portion of tea + 0 of milk + 3 of sugar + 1 of pearls.

Today you are buying ingredients. Let the price per portion of each ingredient be $x_1$ , $x_2$ , $x_3$ and $x_4$ . You want to know immediately: how much does each drink cost to make?

The calculation is simple. Use the portions in each recipe as weights and take a weighted sum of the ingredient prices:

\begin{aligned} y_1 &= 2x_1+1x_2+0x_3+3x_4 &&\text{(pearl milk tea)} \\ y_2 &= 0x_1+4x_2+1x_3+2x_4 &&\text{(milk pudding)} \\ y_3 &= 1x_1+0x_2+3x_3+1x_4 &&\text{(traditional sweet tea)} \end{aligned}

Three drinks give us three weighted sums. The weights are simply the quantities of each ingredient in the recipes. In practice, there might be hundreds of products and hundreds of ingredients. A neural network can have millions of weights.

Rather than writing each formula separately, mathematicians arrange all the recipe quantities into a rectangular table, called a matrix.

A recipe table is a matrix.

$W$ — portions per recipe
Recipe	Tea	Milk	Sugar	Pearls
Pearl milk tea	2	1	0	3
Milk pudding	0	4	1	2
Traditional sweet tea	1	0	3	1

Figure 3-1.

W

is a

3\times4

matrix: each row is a recipe, and each column is an ingredient. Read “3 × 4” as “three rows and four columns”.

We describe a matrix's size as $m \times n$ : $m$ rows — the number of drinks here, or the number of outputs — and $n$ columns — the number of ingredients, or the number of inputs. In this example, $m = 3$ and $n = 4$ .

Keep this picture in mind: one recipe per row, one ingredient per column. We can return to this table to understand the matrix operations that follow.

2. Matrix times vector: one dot product per row

Continue with the bubble tea shop. Suppose today's prices per portion are: tea 1 yuan, milk 2 yuan, sugar 1 yuan, and pearls 0 yuan — the pearls are free as part of a promotion. Write these prices as the column vector $\mathbf{x}$ , with components $[1, 2, 1, 0]$ .

To calculate the costs of all three drinks at once, multiply $W\mathbf{x}$ . This means: take the dot product of each row of $W$ — each recipe — with $\mathbf{x}$ — the ingredient prices — and collect the results in a cost vector. We learned the dot product in the previous lesson: multiply corresponding components and add them.

$W$

\begin{bmatrix}2 & 1 & 0 & 3 \\ 0 & 4 & 1 & 2 \\ 1 & 0 & 3 & 1\end{bmatrix}

$\mathbf{x}$

\begin{bmatrix}1 \\ 2 \\ 1 \\ 0\end{bmatrix}

$\mathbf{y}$

\begin{bmatrix}4 \\ 9 \\ 4\end{bmatrix}

Row 1 · Pearl milk tea

$2 \times 1 + 1 \times 2 + 0 \times 1 + 3 \times 0 = 4$

Row 2 · Milk pudding

$0 \times 1 + 4 \times 2 + 1 \times 1 + 2 \times 0 = 9$

Row 3 · Traditional sweet tea

$1 \times 1 + 0 \times 2 + 3 \times 1 + 1 \times 0 = 4$

Figure 3-2. Take the dot product of each row with x. The three results form the output column vector

\mathbf{y}=[4,9,4]

The results are 4 yuan for pearl milk tea, 9 yuan for milk pudding, and 4 yuan for traditional sweet tea. One matrix multiplication calculates the costs of the entire menu together.

This is what makes matrices useful: they turn “calculate many weighted sums at once” into a single operation. Each neural network layer uses this operation, with a much larger table of weights. A complete layer can also add a bias and apply an activation function; here we are focusing on the matrix–vector multiplication.

Pause and think

$W$ is a $3\times4$ matrix. Why must $\mathbf{x}$ have four components, rather than three or five?

I've thought about it — show the answerHide answer

Each column of $W$ corresponds to one component of $\mathbf{x}$ . They must line up to calculate a dot product. If $W$ has $n$ columns, $\mathbf{x}$ must have $n$ components; otherwise, we cannot pair the positions one by one, and the multiplication is not defined. The general rule is: an $m\times n$ matrix multiplies an $n$ -dimensional vector, producing an $m$ -dimensional vector.

3. From n dimensions to m dimensions

There is one central rule for matrix–vector multiplication. Remember the shapes:

\underbrace{W}_{m\times n}\underbrace{\mathbf{x}}_{n\times1}=\underbrace{\mathbf{y}}_{m\times1}

Figure 3-3. The number of columns of W must match the input dimension. The number of rows determines the output dimension:

(m\times n)\times(n\times1)=(m\times1)

Two points matter:

The columns must match the input: if $W$ has $n$ columns, $\mathbf{x}$ must have $n$ components. A mismatch means the product cannot be calculated.
The rows determine the output: if $W$ has $m$ rows, $\mathbf{y}$ has $m$ components. A matrix can map an $n$ -dimensional input space to an $m$ -dimensional output space.

This ability to move “from one number of dimensions to another” is central to the next lesson, What is a linear transformation? A matrix is more than a calculation tool. Geometrically, it represents a transformation of space, such as rotation, scaling, compression or stretching.

4. What you have discovered

The takeaway. A matrix neatly arranges many sets of weights: each row is a set of coefficients, and each column corresponds to one input. Multiplying a matrix by a vector, $W\mathbf{x}$ , means taking the dot product of each row with $\mathbf{x}$ . The $m$ rows produce $m$ results, forming an $m$ -dimensional output vector. The dimension rule is $W_{m\times n}\mathbf{x}_{n\times1}=\mathbf{y}_{m\times1}$ : the inner dimensions must match, and the outer dimensions determine the result.

What you have discovered in this lesson:

Matrix: an $m \times n$ table of numbers with $m$ rows and $n$ columns, organising multiple sets of coefficients into one object.
Matrix–vector multiplication: $W\mathbf{x}$ takes the dot product of each row with $\mathbf{x}$ , calculating several weighted sums at once.
Dimension rule: $(m \times n) \times (n \times 1) = (m \times 1)$ . $W$ turns an $n$ -dimensional input into an $m$ -dimensional output.
Matching inner dimensions: the number of columns of $W$ must equal the number of components of $\mathbf{x}$ , so that the dot products can line up.

Check your understanding

Choose an answer and select “Submit answer” to see the correct option and an explanation.

1. A neural network layer turns one collection of numbers into another. Which mathematical object naturally represents this linear transformation?

AA single number.BA vector.CA matrix.DA written rule.

2. When a

3\times4

matrix multiplies a vector, how many components must the input have, and how many does the output have?

AThe input must have 4 components; the output has 3.BThe input must have 3 components; the output has 4.CBoth input and output must have 3 components.DAny input dimension works; the result is unchanged.