[1hr Talk] Intro to Large Language Models
Andrej Karpathy59:472023-11-23English
Chapters
In this introductory talk, Andrej Karpathy explains how large language models go from predicting the next word to calling tools to solve problems. The key to understanding these systems is distinguishing what the model has learned in its weights, what behaviors fine-tuning has shaped, and what external resources are available at runtime.
Overview
[1hr Talk] Intro to Large Language Models
- Using Llama 2 70B, this section distinguishes weights from inference code and compares the cost of running an existing model with the cost of training it to produce the weights.
- This section explains how a simple prediction objective lets models learn facts but also produce hallucinations, and why knowing how a Transformer computes its output isn't the same as understanding how its weights work together.
- This section explains how high-quality question-and-answer examples turn a base model into an assistant, how human corrections help refine it, and how comparing preferred answers can improve it further.
- This section uses Chatbot Arena to explain blind rankings and the difference between open-weight and closed-source models, then looks at how parameter count and training data volume affect prediction performance.
- A fundraising analysis demo brings together search, calculation, plotting, and image generation, followed by examples of image understanding and voice interaction.
- Fast and slow thinking and AlphaGo's self-play illustrate two lines of research and highlight the lack of easily automated reward criteria for open-ended language tasks.
- Starting with custom instructions and file retrieval, this section compares the model to a kernel that coordinates tools and the context window to limited working memory.
- This section outlines three types of risk—jailbreaks, prompt injection, and data poisoning—and explains why new capabilities create new security problems and why attacks and defenses will keep evolving.
Key points
- Running an open-weight model requires the weights and inference code. Llama 2 70B, for example, has 70 billion parameters stored at 2 bytes each, so its weights take up about 140 GB. The talk estimates that training it took about 6,000 GPUs running for 12 days and cost about $2 million.
- Next-word prediction can be seen as a form of lossy compression of the training text: the model learns some facts about the world, but it also imitates document formats and invents plausible-looking details such as ISBNs.
- Pretraining and supervised fine-tuning use the same prediction objective. The main difference is the data: pretraining uses a large amount of internet text, while fine-tuning uses a small set of high-quality question-and-answer examples. Human comparisons of candidate answers can further improve performance.
- The scaling laws presented in the talk show that the number of parameters and the amount of training text can predict next-word prediction performance fairly reliably. The speaker notes that these gains also correlate with improvements on several practical evaluations.
- Tool use means the model doesn't have to rely only on what it knows from its weights. It can retrieve information, add the results to its context, and then use a calculator or Python to run calculations and make plots. The valuation projections in the demo came from a fitted trend, not from a lookup.
- The speaker compares the context window to limited working memory and the large language model to an operating system kernel that coordinates external information and computing tools.
- The talk groups large language model security issues into categories such as jailbreaks, prompt injection, and data poisoning. It also notes that adding image input expands the attack surface. Some of the issues demonstrated may since have been patched.
Chapter notes
Model Files and Training Costs
- With 70 billion parameters, Llama 2 70B is the largest model in the Llama 2 family released by Meta AI. Meta made the weights, model architecture, and paper public so users could run the model themselves. By contrast, users can access ChatGPT through a website but don't have direct access to its underlying model.
- Running the model requires only two kinds of files: the weights and the inference code. The 70 billion weights are stored in float16 at 2 bytes each, for a total of about 140 GB. The inference code can be written in about 500 lines of C with no external dependencies. Once compiled, it can load the weights on a MacBook and run offline.
- In the demo, the model responded to a prompt by writing a poem for Scale AI, but it was actually the 7-billion-parameter version. The 70-billion-parameter version is about 10 times slower. The inference code implements the known network architecture and runs the forward pass; training the model to produce the weights is what really takes computing power.
- Training Llama 2 70B required roughly 10 TB of text scraped from the web and about 6,000 GPUs running for 12 days, at a cost of about $2 million.
- Training is like compressing about 10 TB of text into 140 GB of weights, a rough compression ratio of about 100 to 1. But unlike lossless ZIP compression, the weights preserve only broad features of the training text, not a copy that can be restored exactly.
- At the time of the talk, frontier models such as ChatGPT, Claude, or Bard were estimated to be trained at 10 times this scale or more. They require larger clusters and datasets, and a single training run could cost tens or even hundreds of millions of dollars. Once you have the weights, inference takes much less computing power by comparison.
Next-Word Prediction, Hallucinations, and Interpretability
- The neural network's basic job is to predict the next word from the sequence it has already seen. For example, after seeing "cat sat on a," it might assign a 97% probability to "Matt." Prediction and compression are closely related mathematically: the more accurately you predict the next word, the better you can use those predictions to compress the dataset. Training can therefore be viewed as a way of compressing internet text.
- Predicting the next word sounds simple, but it pushes the model to compress a great deal of knowledge about the world into its weights. To predict information-rich words in Ruth Handler's Wikipedia entry, for example, the model needs to learn who she was, when she was born and died, and what she did.
- During inference, the model samples a word from its predicted distribution, adds it back to the input, and keeps predicting and sampling. This lets it generate text that looks like a web page. The generated Amazon product page looks authentic, but the title, author, and ISBN are all made up.
- The model can mimic the numerical format of an ISBN. It can also produce a broadly accurate description of the blacknose dace that doesn't appear word for word in the training set. This kind of lossy compression can preserve some knowledge, but it also produces plausible hallucinations. The output alone doesn't tell you whether the content is correct or which parts the model remembered.
- The math at each stage of a Transformer is well defined. Training can repeatedly adjust the roughly 100 billion parameters spread throughout the network and measure whether next-word prediction improves. But knowing how to perform the calculations and optimize the model isn't the same as knowing how the parameters work together to make a prediction.
- The "reversal curse" shows that the same fact isn't necessarily accessible in both directions: GPT-4 can say that Tom Cruise's mother is Mary Lee Pfeiffer, but it can't answer when asked who her son is. Mechanistic interpretability tries to work out what each part of the network does, but it still can't explain the model fully. In practice, people still mainly look at inputs and outputs and use detailed evaluations to measure model behavior.
From Base Model to Assistant
- Turning a base model into an assistant doesn't change the training objective: it's still predicting the next word. What changes is the data. Internet documents are replaced with high-quality question-and-answer conversations written according to annotation guidelines, perhaps around 100,000 examples. The resulting model answers questions rather than simply continuing a document.
- Even if a particular question wasn't in the training set, a fine-tuned model can produce an answer word by word in the style of a helpful assistant while drawing on the knowledge it gained during pretraining. Roughly speaking, pretraining builds up knowledge, while fine-tuning shapes how the model responds to match expectations. That's what's meant by "alignment" here.
- Pretraining compresses large amounts of internet text into model parameters to produce a base model. It can cost millions of dollars, so a company might do it every few months or even just once a year. Fine-tuning is much cheaper: a company can work with a partner such as Scale AI to collect around 100,000 ideal question-and-answer examples according to guidelines. Training might then take a day rather than months.
- After evaluating and deploying the assistant, the team keeps monitoring it and collecting incorrect answers. People replace them with correct answers, and the corrected conversations go into the training set for the next round of fine-tuning. Because fine-tuning is relatively cheap, this cycle can run weekly or even daily.
- Meta released both base and assistant versions of Llama 2. Given a question, the base model might generate more questions instead of answering it, but users can skip expensive pretraining and fine-tune it themselves. If they just want to ask questions and get answers, they can use the assistant model.
- An optional third stage uses human preferences to fine-tune the model further. For example, it's easier for an annotator to pick the best haiku about a paper clip from several generated by the model than to write one from scratch. OpenAI calls this process of using comparison labels to further improve model performance "reinforcement learning from human feedback" (RLHF).
Model Rankings and Scaling Laws
- Chatbot Arena is run by a team at Berkeley and uses an Elo rating system like the one used in chess. Users enter a question, compare two answers without knowing which models produced them, and pick a winner. The platform calculates ratings from the results of head-to-head matchups, with higher scores placing models higher in the rankings.
- At the time, the top spots were mostly held by closed-source models, such as OpenAI's GPT family and Anthropic's Claude family. Users could generally access them only through a website and couldn't download their weights or fine-tune them themselves. Open-weight models ranked lower, but their weights were available and they were usually described in papers. Examples include Meta's Llama 2 family and Zephyr 7B-beta, which is based on the Mistral family. Their performance was already good enough for some applications.
- Scaling laws describe how next-word prediction accuracy changes with the number of network parameters N and the amount of training text D. Knowing just those two quantities lets you predict accuracy with considerable confidence. The observed trends are smooth and don't yet show signs of leveling off, so training larger models on more text is expected to improve prediction accuracy even without algorithmic advances.
- In practice, people care more about how models do on capability tests than about next-word prediction itself, but the two are empirically correlated. For example, accuracy rose on several tests from GPT-3.5 to GPT-4. Because scaling up training is a relatively predictable path to improvement, organizations are competing for larger GPU clusters and more data, while algorithmic advances can bring additional gains.
Tool Use and Multimodal Capabilities
- When asked to compile Scale AI's funding history, the fine-tuned ChatGPT doesn't answer straight from memory. It issues a tool call, hands the query to Bing Search, and uses the returned web text to put together a table of the Series A through E rounds.
- The funding table lists dates, amounts raised, valuations, and source links for verification, but only the amounts raised were found for Series A and B. ChatGPT calls a calculation tool and uses the ratios of amounts raised to valuations from Series C, D, and E to estimate Series A and B valuations of about $70 million and $283 million, respectively.
- ChatGPT uses Python's Matplotlib to make a two-dimensional plot with gridlines, dates on the horizontal axis, and valuation on a logarithmic vertical scale. As requested, it then adds a linear trend line and a vertical line marking the day of the demo. Extrapolating the trend gives a valuation of about $150 billion on the day of the demo and about $2 trillion at the end of 2025. These are fitted estimates, not valuations found in a search.
- Drawing on the Scale AI information in its context, ChatGPT also calls OpenAI's DALL-E to turn a natural-language description into an image representing the company. The demo shows that a language model can do more than generate text: it can string together external tools for search, calculation, programming, and image generation within a single task.
- Multimodal capabilities also include understanding images. Greg Brockman gives ChatGPT a pencil sketch of a website called My Joke, and the model writes working HTML and JavaScript so users can click to reveal the joke's punchline.
- ChatGPT can also listen and speak, allowing voice-to-voice conversations. In the iOS app, users can speak without typing and get a spoken reply, much like talking with the AI in the movie Her.
Slow Reasoning and Self-Improvement
- Thinking, Fast and Slow distinguishes between two modes of thought. System 1 is fast, intuitive, and automatic: you know that 2 + 2 = 4 without working it out. System 2 is slower and more deliberate, as when you calculate 17 × 24 or work through possible moves one by one in a chess game with enough time, rather than moving on instinct as you would in speed chess.
- The speaker compares large language models at the time to having only System 1. The neural network predicts the next word from the sequence so far, taking about the same amount of time at each step rather than first building a tree of possibilities to reason through. Researchers want to give models a System 2 so they can spend more time exploring different approaches, reflecting, and reframing the problem instead of answering right away.
- The goal of slow reasoning is to trade time for accuracy. If thinking time is on the horizontal axis and answer accuracy is on the vertical axis, the ideal curve should rise monotonically. The speaker notes that models at the time couldn't do this yet.
- DeepMind's AlphaGo was initially trained in two stages. First, the team selected records of games played by strong players so the neural network could imitate human experts. Then it used self-play to go beyond what imitation alone could achieve. Go is a closed environment, and winning or losing provides a simple reward that is cheap to evaluate automatically. AlphaGo could therefore improve its win rate by playing millions of games. According to the Elo ratings shown, it took 40 days to surpass some of the strongest human players.
- At the time, large language models were still mostly at the imitation stage: they were trained to reproduce answers written by human annotators. It's hard to surpass human answer accuracy using that approach alone. Open-ended language tasks vary widely and lack a general reward criterion as quick and automatic as deciding who wins a game of Go. It might be possible to establish such criteria and achieve self-improvement in narrow domains, but it remains an open problem in the general case.
Custom Assistants and the Operating System Analogy
- Different tasks call for different kinds of expertise. With the GPTs App Store announced by Sam Altman, OpenAI is trying to make large language models customizable by letting users create their own GPTs in ChatGPT. There are currently two ways to customize them: set custom instructions or upload files to add knowledge. The latter uses retrieval-augmented generation (RAG) to find relevant text passages in the files and use them as references when answering.
- In the future, users might be able to provide their own training data for fine-tuning, so multiple models can specialize in particular tasks instead of having one model do everything. Rather than thinking of a large language model as a chatbot or text generator, it's better to think of it as the kernel process of an emerging operating system, coordinating memory and computing tools to solve problems.
- Such a kernel might one day browse the web, search local files, call Calculator or Python, and understand and generate images, video, and audio. It might also use System 2 for extended reasoning, improve itself in narrow domains with a reward function, or coordinate several specialized models from an app store to solve a problem together.
- In the operating system analogy, the disk or the internet is a source of information that can be accessed on demand. The context window is like RAM: it's limited working memory that determines how much material the model can refer to when predicting the next word. Acting as the kernel, the model can move relevant information into or out of context as the task requires. The speaker also suggests further parallels with multithreading, multiprocessing, speculative execution, and user space and kernel space.
- The ecosystems are similar too. Desktop computing has proprietary systems such as Windows and macOS alongside an open-source ecosystem built around Linux. Large language models have proprietary families such as GPT, Claude, and Google's Bard, alongside a rapidly growing open-source ecosystem that at the time was built mainly around the Llama family. The analogy helps explain a new computing architecture in which a large language model coordinates tools to solve problems and users interact with it in natural language.
Model Security and New Attack Surfaces
- Large language models are a new computing stack with security issues of their own. The first category is jailbreaks. ChatGPT refuses direct requests for harmful instructions, but the role-play example in the talk shows that the model sometimes answers the same request when it treats it as a fictional scenario.
- A model's ability to understand a form of expression doesn't mean its safety training covers it. Claude can read Base64, but its refusal training, which used mostly English examples, didn't make it consistently refuse the same harmful request across different forms of expression.
- The adversarial text suffixes in the research look like gibberish. Even if defenses are patched to block one suffix, researchers believe another can be found by rerunning the optimization. Image understanding is useful for solving problems, but it also expands the attack surface, making visual input another possible route around safety restrictions.
- The key to prompt injection is that the model mistakes external content it should merely read for instructions it should follow. In one example, text in an image derailed ChatGPT's image-reading task and made it mention a 10% discount at Sephora. In another, instructions on a web page made Bing slip a scam link claiming to offer a $200 Amazon gift card into its movie recommendations.
- Prompt injection in shared documents can also lead to privacy leaks. A user might only ask Bard to process a Google Doc, but instructions in the document could trick it into gathering personal information it can access and disclosing it to someone who shouldn't see it.
- Data poisoning is also called a backdoor attack. Contaminated training data can cause a model to produce abnormal outputs when it encounters a specific trigger word, or even misclassify a threat as harmless. The paper cited in the talk demonstrated this only in fine-tuning. The speaker wasn't aware of a convincing demonstration in pretraining. Defenses exist for all three types of attack, and many older examples may no longer work, but attacks and defenses will continue to evolve.
What it answers
- Why can a model learn facts when it's only trained to predict the next word?
- Because some words can only be predicted accurately with background knowledge. The talk uses Ruth Handler's Wikipedia page as an example: to predict the information-rich words on the page, the model needs to learn some facts about her life in its parameters.
- Why can a base model continue writing a web page but not necessarily answer a question?
- A base model learns how internet text tends to continue, so when it sees a question, it may generate more questions. Assistant fine-tuning switches to high-quality question-and-answer examples so the model learns to follow a question with a helpful answer.
- How are Chatbot Arena rankings calculated?
- Users submit a question, see answers from two anonymous models, and choose the better one. The platform calculates Elo ratings from the wins and losses in head-to-head matchups.
- Why is it hard to apply AlphaGo's self-improvement method directly to general-purpose language models?
- In Go, you can automatically determine who won or lost at low cost. Open-ended language tasks vary widely and don't have an equally simple and fast general reward criterion. Some narrow domains might be exceptions.
- What's the difference between prompt injection and jailbreaking?
- A jailbreak tries to get the model to bypass a refusal it would normally give for safety reasons. Prompt injection gets the model to mistake untrusted content in an image, web page, or document for new instructions, leading it away from the user's task.
Quotes
But this is not exactly a zip file because a zip file is lossless compression.
So long story short, think of LLMs as kind of like mostly inscrutable artifacts.
Intuitively, what we want to do is we want to convert time into accuracy.
I think it's a lot more correct to think about it as the kernel process of an emerging operating system.
Mentioned
- Llama 2 70B — The 70-billion-parameter open-weight model released by Meta AI is the talk's main example for explaining weights, inference, and training costs.
- Meta AI — Released the Llama 2 weights and architecture and provided both base and assistant models.
- Transformer — The talk uses it to show that knowing each mathematical operation doesn't mean you know how the many parameters work together.
- ChatGPT — The talk uses it to demonstrate how an assistant can call tools, process images, and interact by voice.
- OpenAI — In the talk, OpenAI is the developer of ChatGPT and DALL-E and the source of the InstructGPT annotation examples.
- 基于人类反馈的强化学习(RLHF) — The talk introduces it as an optional follow-up fine-tuning method that improves model performance using human comparisons of candidate answers.
- Chatbot Arena — Run by a team at Berkeley, it ranks models using anonymous answer comparisons and Elo ratings.
- Thinking, Fast and Slow — The talk draws on the distinction between System 1 and System 2 popularized by this book to explain the difference between answering immediately and taking time to reason.
- AlphaGo — Developed by DeepMind, this Go program illustrates the shift from imitating humans to improving through self-play.
- 检索增强生成 — Using files uploaded by a user as an example, the talk explains how the model retrieves relevant text passages and uses them to generate an answer.
- 上下文窗口 — The limited amount of information a model can use when generating the next word. In the operating system analogy, it corresponds to working memory.
- 提示注入 — This is when a model mistakes text in external content for new instructions and behaves differently from what the user originally intended.