But what is a neural network? | Deep learning chapter 1

3Blue1Brown18:392017-10-05English

This introductory deep learning lesson uses handwritten digit recognition to show that a neural network is a function that computes weighted sums, adds biases, and applies an activation function layer by layer. Learning means finding suitable weights and biases from data.

Overview

But what is a neural network?

Key points

  • The example network's input layer has 784 neurons, each storing a pixel's grayscale value. There are two hidden layers with 16 neurons each, and the output layer has 10 neurons corresponding to the digits 0 through 9.
  • The activations produced by the input image pass from one layer to the next. The neuron with the highest activation in the output layer ultimately determines which digit the network predicts.
  • Ideally, the hidden layers would first identify short edges from the pixels, then combine those edges into parts such as loops or long lines, and finally recognize the digit. This is the intuition behind the design, not a conclusion about how this particular network actually behaves.
  • In the example, a neuron first computes a weighted sum of the previous layer's activations, then adds a bias and applies sigmoid. The weights determine which patterns it is sensitive to, while the bias determines how high the weighted sum must be for the neuron to become noticeably active.
  • The entire network has about 13,000 weights and biases. Learning means having the computer find values for these parameters that allow it to perform the task.
  • By writing the activations as a column vector and the weights as a matrix, the calculation for one layer can be expressed as σ(Wa+b), with sigmoid applied element by element.
  • The ending introduces ReLU, which is easier to train with. It takes the larger of 0 and the input value and works well in very deep neural networks.

Chapter notes

Understanding Network Structure Through Handwritten Digits

  • Even when a handwritten 3 is messy and the image is only 28×28 pixels, people can recognize it easily. Different ways of writing it may produce very different pixel values and stimulate very different photoreceptor cells, yet the visual cortex recognizes them as the same digit. Writing a program that can identify the digit from the same pixel grid is much harder.
  • Handwritten digit recognition serves as an example for understanding a neural network first as a mathematical structure. This lesson looks only at the most basic network structure; the next lesson explains how the network learns.
  • A neuron here can be thought of as a unit that holds a number between 0 and 1, called its activation. The 784 neurons in the input layer correspond one-to-one with the 28×28 pixels. Their activations represent pixel grayscale values: black is 0 and white is 1.
  • The output layer has 10 neurons corresponding to the digits 0 through 9. Each activation indicates how strongly the network favors classifying the image as the corresponding digit. Between the input and output layers are two hidden layers with 16 neurons each. Two layers make the later explanation easier, and 16 neurons fit on the screen; neither number is a fixed requirement.
  • In a trained network, pixel brightness first determines the pattern of activations in the input layer, and the activations in each layer then determine those in the next. At the output layer, the digit corresponding to the neuron with the highest activation, shown as the brightest, is the network's prediction.

From Local Edges to Parts of Digits

  • The intuition behind layered recognition comes from how people recognize digits: by putting local parts together. A 9 consists of a loop at the top and a line on the right, an 8 has one loop above another, and a 4 can essentially be broken down into three particular lines.
  • Ideally, each neuron in the second-to-last layer would correspond to one kind of digit part. As long as there is a roughly loop-shaped pattern at the top of the image, the corresponding neuron's activation would be close to 1, regardless of the exact pixel arrangement. The output layer would then only need to learn which combinations of parts correspond to which digits.
  • Recognizing parts can be broken down further: a loop can be identified by the multiple short edges that make it up. The long lines in 1, 4, and 7 can be viewed either as single long edges or as specific arrangements of multiple short edges.
  • For a 9, the input image might first activate neurons in the second layer corresponding to about 8 to 10 short edges, then activate the neurons for a loop at the top and a long vertical line, and finally activate the output neuron for 9. This is an idealized picture of what the layered structure could do. Whether the trained network actually works this way will be discussed after training is introduced.
  • Edge and pattern detection can also be used in other image recognition tasks, and layer-by-layer abstraction is not limited to images. Speech processing can distinguish different sounds in raw audio and then combine them, layer by layer, into syllables, words, phrases, and more abstract ideas.

Weights, Biases, and Activation Functions

  • The idea is for the network to combine pixels into edges, edges into patterns, and patterns into digits, one layer at a time. Take a neuron in the second layer that detects a local edge: it assigns a weight to each connection from the first layer and computes a weighted sum of the first layer's activations.
  • To make a neuron respond to an edge in a particular location, you can give the central region positive weights, the surrounding region negative weights, and everything else zero weights. The weighted sum is highest when the central pixels are bright and the surrounding pixels are dark. The weighted sum can be any real number, but the sigmoid function squeezes it between 0 and 1: the more negative the input, the closer the output is to 0; the more positive the input, the closer the output is to 1.
  • If you want the neuron to become noticeably active only when the weighted sum exceeds 10 rather than 0, add −10 before passing it through sigmoid. This extra number is called the bias. The weights determine which pixel patterns the neuron responds to, and the bias determines how high the weighted sum must be for the neuron to become noticeably active.
  • Each of the 16 neurons in the second layer connects to all 784 pixel neurons in the first layer. Each connection has its own weight, and each neuron has a bias of its own. So the connections from the first layer to the second alone account for 784×16 weights and 16 biases. Including the other layers, the network has about 13,000 parameters in all.
  • Training means having the computer find a set of values for these weights and biases that solves the problem at hand. You could also imagine tuning the parameters by hand so that the second layer recognizes edges and the third recognizes patterns. Understanding what these parameters mean helps you try to improve the structure when the network does not behave as expected, and lets you test your original assumptions when the network works for unexpected reasons.

Matrix Notation and a Note on ReLU

  • Arrange the activations of one layer's neurons in a column vector and the connection weights in a matrix. Each row of the matrix corresponds to one neuron in the next layer, so each component of the matrix-vector product is the weighted sum received by that neuron. Then arrange all the biases in a vector and add it directly to the product.
  • Apply sigmoid to each component of the vector after adding the biases to get the next layer's activations. The whole process can be written as sigmoid(Wa + b). This notation makes the code simpler and can speed up calculations by taking advantage of the matrix multiplication optimizations in many libraries.
  • It is better to think of each neuron as a function: it takes the outputs of all the neurons in the previous layer and returns a number between 0 and 1. The whole network is a function too: it takes 784 input numbers and produces 10 output numbers. Layer by layer, it multiplies a matrix by a vector, adds biases, and applies sigmoid, using about 13,000 weight and bias parameters to capture patterns.
  • Leisha Li notes that early networks used sigmoid to squeeze inputs between 0 and 1, inspired in part by the biological analogy of a neuron firing or not firing. Modern networks use sigmoid less often. ReLU, or the rectified linear unit, is easier to train with and computes max(0, A): if A is positive, it returns A unchanged; otherwise, it returns 0.
  • For a time, training networks with sigmoid was difficult. People tried switching to ReLU and found that it worked very well in very deep neural networks.

What it answers

How does the network turn an image into a digit prediction?
It takes the grayscale values of the 784 pixels as input, passes the activations from one layer to the next, and then chooses the digit represented by the neuron with the highest activation in the output layer.
Why put hidden layers between the input and output layers?
One idea is to have an earlier hidden layer detect short edges and the next layer combine those edges into loops or long lines, so the output layer can recognize digits from those parts. The video does not claim that the trained network actually works this way.
What do weights and biases each control?
The weights determine which input patterns produce a larger weighted sum. For example, positive weights can make the neuron respond to a bright line, while negative weights suppress the response to surrounding areas that are just as bright. The bias determines how large the weighted sum has to be for the neuron to become noticeably active.
What does it mean for a neural network to “learn” here?
It means having the computer use data to find weights and biases that allow it to perform the recognition task. This episode does not go into how it finds them.

Quotes

So the weights tell you what pixel pattern this neuron in the second layer is picking up on, and the bias tells you how high the weighted sum needs to be before the neuron starts getting meaningfully active.
All said and done, this network has almost exactly 13,000 total weights and biases.
Really, the entire network is just a function, one that takes in 784 numbers as an input and spits out 10 numbers as an output.

Mentioned

  • Leisha Li — Studies deep learning theory as part of her PhD and discusses the choice of activation function with the presenter at the end of the video.
  • 激活值 — The number a neuron represents. In this example's input layer, it corresponds to a pixel's grayscale value.
  • 隐藏层 — Located between the input and output layers. This example has two hidden layers with 16 neurons each.
  • 权重 — A number on each connection that determines how an activation from the previous layer contributes to the weighted sum.
  • 偏置 — A number added to the weighted sum before applying the activation function. It adjusts how large the weighted sum must be for the neuron to become noticeably active.
  • sigmoid — The activation function used in the example network, which maps real numbers to values between 0 and 1.
  • ReLU — The rectified linear unit returns the larger of the input value and 0. It is presented at the end as an option that makes training easier.