Transformer论文逐段精读【论文精读】
跟李沐学AI1:27:042021-10-28Chinese
Chapters
This close reading breaks down “Attention Is All You Need”: the Transformer uses attention to aggregate sequence information in parallel, replacing recurrent layers, but the full model still relies on position-wise feed-forward networks, residual connections, and positional encoding.
Overview
Transformer论文逐段精读【论文精读】
- Introduces the paper’s impact, its randomized author order, and its statement of contributions, then draws out the abstract’s central claim that attention alone can replace recurrence and convolution.
- Reviews the translation results, the open-source implementation, and the authors’ plans for other modalities, then explains why passing hidden states step by step limits parallelism and can cause RNNs to lose early information.
- Compares how convolution and self-attention connect distant positions, then uses the architecture diagram to show how the encoder represents the input and the decoder generates the output step by step.
- Breaks down the six-layer encoder and its fixed 512-dimensional representations, uses sequences of varying lengths to explain the difference between LayerNorm and BatchNorm, and covers the decoder’s third sublayer and causal mask.
- Starts with what queries, keys, and values mean, derives the batched matrix calculation, and explains how scaling reduces softmax saturation and how masking sets the weights of hidden positions to zero.
- Explains how multiple learnable projections recognize different patterns, then traces the sources of the inputs to encoder self-attention, decoder masked self-attention, and encoder–decoder attention.
- Explains how a shared two-layer feed-forward network processes the representation at each position, and how embedding weights are shared and sine and cosine positional encodings give the model information about word order.
- Compares global self-attention, recurrent layers, convolutional layers, and local self-attention in terms of computational complexity, the number of steps that must run sequentially, and the paths that information takes.
- Goes through the translation data, the shared BPE vocabulary, training hardware, learning-rate warmup, and regularization, and compares the settings for the base and big models.
- Considers why the paper is concise but hard to read, reviews how the Transformer gave different modalities a shared model architecture, and explains why attention alone cannot do everything the full model does.
Key points
- The Transformer’s innovation was not to introduce self-attention for the first time, but to replace recurrent layers with attention in an encoder–decoder architecture for sequence transduction, so it no longer relied on recurrence or convolution to aggregate sequence information.
- The paper reports 28.4 BLEU on English-to-German translation, about 2 BLEU above the previous best. A single model also achieved better results on English-to-French translation, taking 3.5 days to train on 8 GPUs.
- The base model stacks 6 layers each in the encoder and decoder. The decoder generates output autoregressively, and during training a mask prevents the current position from accessing future information.
- Scaled dot-product attention uses the dot products of queries and keys to compute weights, then takes a weighted sum of the values. Dividing by the square root of the key dimension helps prevent high-dimensional dot products from driving softmax into saturation and shrinking the gradients.
- Multi-head attention uses different learnable projections to identify different patterns. The base model splits the 512-dimensional representation across 8 heads, each computing attention in a 64-dimensional space, then combines and projects the results.
- Attention aggregates information across positions, and the same feed-forward network then processes each position independently. Because attention itself provides no information about word order, the model adds positional vectors made from sine and cosine functions of different periods to the word embeddings.
- Global self-attention lets any two positions pass information in a single step, but the amount of computation grows quadratically with sequence length. The speaker also notes that fewer structural assumptions may mean needing more data and a larger model.
Chapter notes
The paper’s significance, author credit, and abstract
- The Transformer is regarded as one of the most important deep learning papers of the past three years, establishing a fourth major class of models after MLPs, CNNs, and RNNs. Stanford worked with more than 100 authors on a survey of more than 200 pages discussing the Transformer and its variants and proposing the name “foundation models” for this class of models.
- The title “Attention Is All You Need” sums up the paper’s claim while playing on an everyday English expression used to remind people to pay attention. “X Is All You Need” later became a title format that many papers copied.
- All eight authors’ names have asterisks, but the footnote explicitly says the order was random, not ranked by contribution. The asterisks should not simply be taken to mean that all eight contributed exactly equally. Machine learning papers usually list authors in order of contribution, with the first author typically receiving the most credit.
- The paper describes what each author actually contributed: Jakob proposed replacing RNNs with self-attention; other authors’ work included implementing the first model, running experiments, and writing the paper; and two authors refactored the code for the open-source Tensor2Tensor implementation. The speaker thinks that, for a paper with many collaborators, this approach makes clear what each person did rather than just listing names.
- Sequence transduction means generating one sequence from another, such as translating an English sentence into Chinese. Earlier mainstream models used an encoder–decoder architecture and relied on relatively complex recurrent or convolutional neural networks. The best-performing models also typically added an attention mechanism between the encoder and decoder.
- The abstract proposes a simple architecture called the Transformer, whose main contribution is to rely only on attention mechanisms, without recurrence or convolution.
Translation results and the motivation for replacing recurrence
- The Transformer reached 28.4 BLEU on English-to-German translation, about 2 points above the best result at the time. On English-to-French translation, a single model outperformed all previous models and took just 3.5 days to train on 8 GPUs.
- The paper describes the Transformer as the first sequence transduction model to use only attention mechanisms. It replaces every recurrent layer with multi-head self-attention. The authors hoped to apply it to images, speech, and video, and identified reducing generation’s dependence on sequential processing as another direction for research. The paper’s code is in the tensor2tensor library.
- An RNN processes a sequence word by word from left to right: the hidden state h_t at step t is determined by the previous hidden state h_{t-1} and the current word. This is how information from earlier in the sequence gets passed along.
- Each step has to wait for the previous one to finish, so 100 words take 100 sequential steps, making it hard to use the many threads available on a GPU or TPU. The longer the sequence, the more likely early information is to get lost along the way. A larger hidden state helps retain information, but it has to be saved at every step, increasing memory use.
- Earlier attention mechanisms were used alongside RNNs, mainly to pass information from the encoder to the decoder. The Transformer removes recurrent layers entirely so different sequence positions can be computed in parallel, shortening training time and improving results.
Related work and the architecture diagram
- Earlier work replaced recurrent neural networks with convolutional neural networks to reduce the amount of computation that had to proceed in time-step order. But a convolution only sees a local window at a time, such as a 3×3 patch of pixels, so information from distant positions has to pass through several layers before it can be combined. The Transformer’s attention can see the whole sequence in a single layer.
- Different output channels in a convolution can recognize different patterns, and the Transformer uses multi-head attention to achieve a similar effect. This paper did not introduce self-attention. The authors claim that, to their knowledge, the Transformer is the first model to build an encoder–decoder architecture using only self-attention.
- The encoder converts an input sequence of length n into a sequence of vectors of the same length, where zₜ is the vector representation of input position xₜ. The decoder uses these vectors to generate an output of length m, which does not have to equal n. For example, a sentence and its translation can have different numbers of words.
- The encoder can see the entire input at once, while the decoder generates words autoregressively. It first generates y₁ given the encoded input, and when it generates yₜ, it takes the preceding outputs y₁ through yₜ₋₁ as input.
- In the architecture diagram, the encoder receives the source sentence and the decoder receives the output sequence shifted one position to the right, so previously generated words become inputs to later positions. An embedding layer first turns words into vectors, and positional encodings are then added. The encoder is a stack of blocks with the same structure, each containing multi-head attention and a feed-forward network.
- The encoder blocks also use residual connections and normalization, and the encoder output is passed to the decoder. The decoder blocks add masked multi-head attention to a similar structure. Each block contains three sublayers, the blocks are stacked, and an output layer and softmax produce the final output.
Layer stacking, residual connections, and normalization
- The encoder is a stack of 6 layers with the same structure. Each layer has two sublayers: multi-head self-attention and a position-wise fully connected feed-forward network, which is essentially an MLP.
- Each sublayer first adds its input to its output, then applies LayerNorm: LayerNorm(x + Sublayer(x)). Residual addition requires matching input and output dimensions; otherwise, a projection is needed. Each encoder layer therefore represents every word with a 512-dimensional vector. The main settings that can be adjusted are the number of layers and the representation dimension.
- For two-dimensional input, BatchNorm computes the mean and variance of each feature column across the minibatch and normalizes each column to a mean of 0 and a variance of 1. LayerNorm instead computes statistics for each example row and normalizes the features within that example.
- Sequence inputs have three dimensions: batch, sequence length, and features. For each feature, BatchNorm pools across sequence positions throughout the batch. When example lengths vary widely, minibatch statistics can fluctuate, and the global statistics saved during training may not apply to particularly long sequences encountered at prediction time. LayerNorm uses statistics from each example itself and does not need to save global means and variances, making it relatively more stable when lengths vary.
- The decoder also stacks 6 layers with the same structure. It keeps the encoder’s two types of sublayers and adds a third multi-head attention sublayer, again using residual connections and normalization.
- The decoder predicts autoregressively, but ordinary attention can see the entire input. During training, a causal mask therefore prevents position t from seeing inputs after t, so the information available during training matches what is available at prediction time.
Scaled dot-product attention and masking
- Attention maps a query and a set of key-value pairs to an output vector. The output is a weighted sum of the value vectors and has the same dimension as they do. Each value’s weight is determined by the similarity between the query and its corresponding key.
- In scaled dot-product attention, queries and keys have dimension d_k, while values and outputs have dimension d_v. For a given query, first take its dot product with each key, then divide the scores by √d_k and apply softmax. This produces nonnegative weights that sum to 1, which are used to take a weighted sum of the value vectors.
- In the batched calculation, Q and K contain n queries and m keys. First compute QKᵀ to obtain an n×m score matrix. Each row contains the dot products of one query with all the keys. Divide by √d_k, then apply softmax independently to each row. Multiply the resulting weight matrix by the m×d_v matrix V to get an n×d_v output matrix, with one row per query.
- Batched attention consists mainly of two matrix multiplications and can process the positions in a sequence in parallel. Unlike additive attention, which can handle queries and keys with different dimensions, the Transformer’s scaled dot-product attention adds a scaling factor to ordinary dot-product attention.
- When d_k is small, scaling makes little difference. When d_k is large, the dot-product scores can spread farther apart, pushing the largest softmax weight close to 1 and the others close to 0. This shrinks gradients and makes training hard to move forward. Dividing by √d_k helps reduce this saturation.
- Masking lets the model compute all the dot-product scores first, then, before softmax, replace the scores for query t at position t and all later positions with a very large negative number, such as −10¹⁰. Exponentiating these scores gives 0, so their weights also become zero. The output therefore uses only v₁ through vₜ₋₁, making the information visible during training match what is visible at prediction time.
Multi-head attention and the three types of attention
- For each head, multi-head attention uses separate learnable matrices WQ, WK, and WV to project Q, K, and V into lower dimensions, then computes scaled dot-product attention. The head outputs are concatenated and passed through a linear projection with WO to produce the final output.
- The similarity calculation in dot-product attention has no learnable parameters of its own, but different projections can learn different spaces for measuring similarity, allowing each head to match different patterns. The paper uses 8 heads, each projecting to 512 ÷ 8 = 64 dimensions. The results are concatenated and projected back to 512 dimensions, so the input and output dimensions match for residual connections.
- In encoder self-attention, Q, K, and V all come from the same set of inputs: n vectors of dimension d. Each query has one output, so the result is still n vectors of dimension d. Each output is a weighted sum of the input vectors, with weights determined by how similar the query is to each key.
- The decoder’s masked self-attention likewise uses the same inputs for Q, K, and V. The sequence length can be m, and the vector dimension is still d. The difference is that when computing the output at a given position, the attention weights for later positions are set to zero, so the current query cannot see what comes after it.
- Encoder–decoder attention uses the n vectors of dimension d from the encoder’s final layer as K and V, and the m vectors of dimension d from the decoder’s masked self-attention as Q. Each decoder query takes a weighted sum of the encoder outputs based on its similarity to each encoder key.
- For example, when translating “hello world” into “你好世界,” the model can give the encoder vector for “hello” a higher weight when computing “好,” and give “world” a higher weight when computing “世.” This lets the decoder pull more relevant information from the encoder outputs based on the current query.
Position-wise feed-forward networks and positional encoding
- The position-wise feed-forward network independently applies the same MLP to each token in the sequence, operating only along the last dimension. It uses ReLU between two linear layers: the first expands the representation from 512 to 2,048 dimensions, and the second projects it back to 512 for the residual connection.
- Attention first aggregates relevant information from across the sequence, so the MLP only needs to process each position’s representation independently, mapping it into the desired semantic space. An RNN, by contrast, passes sequence information along by feeding the output from one time step into the next.
- Word embeddings map each token to a 512-dimensional vector. The encoder input embeddings, decoder input embeddings, and linear layer before softmax all share the same weights. The input embeddings are multiplied by √512 so they are on a similar numerical scale to the positional encodings that are added next.
- The output of attention is a weighted sum of the values. The weights depend on the relationship between the queries and keys, not on the positions of the tokens. Without positional information, shuffling the word order would merely rearrange the outputs in the same way without changing their corresponding values, so the outputs could not reflect differences in meaning caused by word order.
- Positional encoding uses sine and cosine functions with different periods to represent each position as a 512-dimensional vector whose components lie between −1 and 1. Adding that vector element by element to the word embedding at the same position puts word-order information directly into the model input.
The efficiency and cost of self-attention
- Section 4 of the paper uses a table to compare global self-attention, recurrent layers, convolutional layers, and restricted self-attention and explain why self-attention was chosen. It compares three measures: computational complexity, the number of computation steps that must run sequentially (fewer makes parallelization easier), and the longest path for information to travel between any two positions (shorter is better).
- Let n be the sequence length and d the vector dimension. Global self-attention computes all query–key pairs, giving it a complexity of O(n²d). It uses a fixed number of matrix operations, so the number of sequential steps is O(1). Each query’s output is a weighted sum of all the values, so information can pass between any two positions in one step, no matter how far apart they are.
- A recurrent layer performs a d×d matrix operation at each of the n positions in turn, for a complexity of O(nd²). Whether it costs more than global self-attention therefore depends on whether n or d is larger. Each step must wait for the previous one, and information has to travel O(n) steps to cover the longest distance. The speaker uses this long path to explain why RNNs tend to lose early information when processing long sequences.
- A one-dimensional convolution has a complexity of O(knd²), where the kernel width k is typically only 3 or 5, so the amount of computation is close to that of a recurrent layer. But the positions can be computed in parallel, so it is usually faster. A single layer can only pass information within a window of width k. Information outside that window has to pass through multiple layers, and the speaker describes this path as logarithmic.
- Restricted self-attention has each query interact with only its r nearest neighbors, avoiding the n² pairings of global attention at the cost of requiring multiple steps to pass information over long distances. The speaker thinks attention is often used precisely to combine information across long sequences, which is why the unrestricted version is more common in practice.
- When the sequence length and model width are similar and the networks have the same depth, global self-attention, recurrent layers, and convolutional layers have roughly comparable computational complexity. Self-attention and convolution are easier to parallelize, and self-attention makes it easier to combine information from distant positions. Still, the table alone does not establish that self-attention will necessarily perform better on long sequences in practice without being slower.
Translation experiments and training setup
- The English-to-German experiment uses the standard WMT 2014 dataset, which contains 4.5 million sentence pairs. BPE splits words with different endings into reusable subwords, so each word form does not have to be a separate token, reducing the vocabulary size.
- English and German share a vocabulary of 37,000 tokens, allowing the encoder and decoder to share word embedding weights. English-to-French uses a larger dataset. The models were trained on 8 P100 GPUs. The base model took about 0.4 seconds per batch and trained for 100,000 steps in 12 hours. The big model took about 1 second per batch and trained for 300,000 steps in 3.5 days.
- The optimizer is Adam. Wider models use a lower learning rate, scaled by the model width raised to the power of −0.5. The learning rate gradually increases over the first 4,000 steps, then decays with the number of training steps raised to the power of −0.5.
- Residual dropout is applied to the outputs of sublayers such as multi-head attention and the MLP, before the residual connection and layer normalization. Dropout is also applied after the word embeddings and positional encodings are added together. The base setting uses a dropout rate of 0.1.
- Label smoothing removes the requirement that the predicted probability of the correct word approach 1, so softmax does not need extremely large outputs to achieve that. This increases perplexity but improves accuracy and BLEU scores, which matter more.
- Both the base and big models have 6 layers. The base model has a width of 512, an MLP hidden layer width of 2,048, and 8 attention heads, with key and value dimensions of 64 per head. The big model doubles the model width, the MLP hidden layer width, and the number of heads, while keeping the per-head dimensions unchanged and increasing the dropout rate from 0.1 to 0.3.
The paper’s writing and the architecture’s impact
- The additional experiment was meant to show that the model not only does well on machine translation but also works on another task. The paper is short, and nearly every sentence conveys a piece of information, yet the close reading took about an hour and a half. The main text essentially proposes the model, describes its structure, compares it with CNNs and RNNs, and reports the results, without a narrative to help readers understand the argument and be convinced by it.
- The writing could be improved by cutting material or moving secondary details to an appendix, leaving the main text for the research motivation, design principles, and reasoning behind them. The Transformer later expanded to nearly every NLP task, while BERT and GPT greatly improved performance on these tasks by training large pretrained models.
- Much as CNNs provided a general framework for computer vision, the Transformer freed NLP from having to do extensive text preprocessing and design a different architecture for each task. Pretrained models also made training simpler. The Transformer made progress on images, speech, and video as well, giving research fields that once used different architectures a shared model framework.
- A shared architecture can help technical breakthroughs in one field reach other fields faster. It can also put features from text, images, speech, and video into the same semantic space, making it possible to combine multimodal data and train larger, better-performing models.
- Although the Transformer performs very well in experiments, our understanding of it is still at an early stage. Attention mainly aggregates sequence information; if you remove the MLPs and residual connections, attention alone is basically unable to learn anything useful. One explanation is that attention does not explicitly model order or assume a spatial structure, but its broader inductive bias allows it to handle more general kinds of information and compete with RNNs and CNNs.
- More general assumptions also come at a cost: the model has a harder time extracting the information it needs from the data, so it needs more data and a larger model to achieve the desired results, and training costs more. The Transformer’s success showed that there are viable alternatives to CNNs and RNNs, and later work has achieved good results on image and text tasks using only MLPs or simpler architectures.
What it answers
- Why is the Transformer easier to parallelize than an RNN?
- An RNN must wait for the previous hidden state before computing the current position. Attention can use matrix multiplication to compute interactions across multiple positions at once. Global self-attention also lets distant positions exchange information without passing it through the sequence one step at a time.
- Why does the decoder need both masked self-attention and encoder–decoder attention?
- Masked self-attention retains the target-side context that has already been generated while blocking future positions. Encoder–decoder attention uses queries from the target side to pull the information it currently needs from the encoded source sequence.
- Why use multiple attention heads instead of computing attention just once?
- Different heads learn different projections, so queries and keys can match different patterns in different spaces. The results from the heads are concatenated and then projected back to the model dimension.
- Without recurrent layers, how does the Transformer know what order the words are in?
- Attention itself does not infer word order from the arrangement of keys and values. The model therefore represents positions with sine and cosine functions of different periods, then adds the positional vectors to the word embeddings.
- Does “Attention Is All You Need” mean the other components can be removed?
- No. The speaker emphasizes that attention mainly aggregates sequence information, and that position-wise feed-forward networks and residual connections are also essential.
Quotes
当你一篇文章有比较多作者的时候,能把每个作者对这个文章的贡献明明白白写出来,其实是一件很好的事情。
When you have quite a few authors on a paper, it’s actually a very good thing to be able to spell out clearly what each author contributed to it.
虽然你的key value并没有变,但是随着你query的改变,因为权重分配不一样,导致你的输出会有不一样。
Even though your keys and values haven’t changed, when your query changes, the weights are distributed differently, so your output can be different.
所以我觉得在未来肯定会有更多新的架构出现,让整个这个领域更加有意思一些。
So I think there will definitely be more new architectures in the future, which will make the whole field a bit more interesting.
Mentioned
- 李沐 — The speaker in this close reading, who explains the paper section by section and comments on its writing and impact.
- Attention Is All You Need — The paper covered in this close reading, which proposes the attention-based Transformer architecture for sequence transduction.
- Transformer — An encoder–decoder model that uses attention to exchange information across a sequence and position-wise feed-forward networks to process representations.
- Google — The institution that most of the paper’s authors were affiliated with, which the speaker also uses to explain the context behind the training hardware.
- Tensor2Tensor — The open-source library containing the paper’s code; the authors’ contribution statement mentions that the implementation code was refactored.
- BLEU — The metric the paper uses to report and compare machine translation quality.
- 多头注意力 — Projects queries, keys, and values separately into multiple lower-dimensional spaces, computes attention, and then combines the outputs of the heads.
- LayerNorm — The normalization operation placed after the residual addition in the original architecture, which the speaker contrasts with BatchNorm, which relies on minibatch statistics.
- 位置编码 — Represents sequence positions as vectors and adds them to word embeddings to give the model information about order.
- WMT 2014 — The machine translation benchmark mentioned in the talk, whose English-to-German training data contains about 4.5 million sentence pairs.
- BERT — The pretrained model the speaker uses to illustrate how the Transformer extended from machine translation to more natural language processing tasks.
- GPT — A later pretrained model the speaker mentions alongside BERT to illustrate the value of reusing the original architecture.