
E117|对话贾扬清:大模型的开源之战与应用时代|AIGC特辑
www.xiaoyuzhoufm.com57:37Chinese
Chapters
Silicon Valley 101 talks with Lepton AI founder Yangqing Jia: Open pretrained models make it easier to experiment, but whether large models create lasting value depends on whether they can solve specific business problems and bring the cost of everyday inference down to a level users are willing to pay for.
Overview
- Hongjun introduces Jia's background, and they compare the cost, fit, and control of paying per use for a closed-source API versus deploying an open model yourself.
- From Android and Linux to pretrained models, Jia explains how open source lowers development costs and attracts contributors, and why releasing model parameters does more to advance large-model innovation than releasing code alone.
- The two discuss how much model providers can do to curb misuse, and distinguish small teams' advantages in technical innovation from large companies' ability to promote AI through their existing businesses.
- From GitHub Copilot and Jasper to how long closed-source models might stay ahead, Jia explains why application value, accuracy in specific fields, and an understanding of users' needs matter.
- Jia discusses how reports about GPT-4 partly confirmed his suspicions about mixture-of-experts models. He then explains why training and inference costs work differently and what problems Lepton AI wants to solve.
- The two consider whether a more economical architecture can replace the Transformer and why ad design, Miaoya Camera, and business productivity are areas where value can be seen early.
- Jia explains Lepton AI's name, how the team works together, and what customers need. He uses a financial-report Q&A experiment to show what industry-specific fine-tuning can do and why model hallucinations are still a problem.
- Jia looks back on his move from technical work to management and then to founding a company. He discusses what he looks for in recruits, his views on xAI, and the potential of general methods when backed by large amounts of data and computing power.
Key points
- Llama 2's availability for commercial use lets developers fine-tune an existing pretrained model. For small companies and research institutions that cannot afford to train a model from scratch, code alone is nowhere near enough. Having a ready-to-use model is what really lowers the barrier to experimentation.
- Closed-source APIs are a good fit for early experiments and small-scale use because users can pay for what they use without building their own clusters. Once a business reaches sufficient scale and a general-purpose model is no longer a good fit, self-hosting and custom models may be more cost-effective.
- Small teams can concentrate talent and quickly learn by trial and error. Large companies have existing businesses and distribution channels. Even if a startup builds a useful feature first, Microsoft can put a similar capability directly into Office, which users already use.
- Jia isn't expecting a single blockbuster app. Instead, he's optimistic about AI improving productivity across industries. In his own experience, GitHub Copilot can write about 40% of his code, mostly routine parts rather than code that requires deep thought.
- Jia estimates that closed-source large models might have a six-month to one-year lead, but that may not give them a long-term moat. What matters more to him is whether a team can keep producing new research and spot the problems users actually need solved.
- In Jia's experience, the total cost of inference in a stable application can be five to ten times the cost of training. Training is a fixed investment that the business must recoup later, while inference costs grow with usage.
- Lepton AI mainly provides a machine learning platform and standardized services for fine-tuning and deploying models. It relies on cloud providers for the underlying infrastructure and doesn't make industry-specific business decisions for customers.
Chapter notes
Llama 2 opens for commercial use, and how to deploy it
- Lepton AI founder Yangqing Jia joined Google after graduating in 2013 and was one of the authors of Google Brain's TensorFlow. In February 2016, he joined Facebook and helped develop Caffe2Go, Caffe2, and PyTorch. In March 2019, he joined Alibaba as a group vice president and president of Alibaba Cloud's Intelligent Computing Platform business unit. He began building his startup in April of the year discussed in the episode.
- Jia cites Alex Krizhevsky's 2012 AlexNet as an example: A big jump in model performance opened up new applications for computer vision. Likewise, better language understanding models can solve problems common to many applications.
- Jia considers Llama easy to use and among the better models in terms of quality. Open models give developers fewer concerns about costs and data security when building applications on top of them. The earlier Llama was free for research but couldn't be used commercially. Llama 2 can be used to build commercial applications.
- Using OpenAI's GPT-4 or GPT-3.5 through an API has a low upfront cost. For lightweight tasks like polishing a document or drafting an email, users can pay as they go without building a large cluster to run the model themselves.
- Closed-source general-purpose models have to serve many uses, which can limit how well they fit industry-specific tasks. As a business scales, a custom or self-hosted model may work better for a particular use case and may also cost less. For small businesses, going straight to a closed-source API is therefore usually simpler and cheaper. Companies that can afford to support large-scale applications can consider deploying an open model themselves.
The economics of open source and the value of community
- You can use iOS on an iPhone, but it's hard for users to modify it themselves. Android makes the source code for the whole operating system public, so developers who know Java, how to compile code, and embedded systems can deploy it directly rather than rewrite it from scratch. Open source doesn't remove the technical barrier, but it makes reuse and rebuilding less costly. Linux, which runs on the vast majority of servers, is another example.
- When Yann LeCun joined Meta to build up FAIR, he argued that research had to be conducted openly because he believed it was hard to make academic progress in a closed environment. Caffe2, the deep learning framework Jia developed at Meta, also made a quick impact in the industry after it was open-sourced.
- Open-source software and the people who build applications are complementary: The easier it is to get the software, the less it costs to train engineers to use it and iterate on it, and the more efficient development becomes overall. Jia compares this to selling low-cost printers to drive ink cartridge sales, and notes that Microsoft went from resisting open source to supporting GitHub and open-sourcing VS Code.
- Once OpenAI made its API available, developers could try all kinds of applications on top of it. But the closed-source model still made it hard to study or modify the model itself or improve it independently. Llama 2 lets researchers open the black box, fine-tune an existing model, and explore new techniques.
- Some computer vision models used to take only a week to train on two or three cards, but large models require substantial data and computing power. Releasing only the code doesn't put training costs in the tens of millions within reach of individuals or small organizations. By providing a pretrained model, Llama lets developers try fine-tuning and making improvements without training from scratch.
- Even if open models don't immediately translate into reported revenue, they can improve Meta's image, attract talent, and give more developers a chance to solve problems together. Linux shows how this works: It's easier for users to adopt it, and they're more willing to give back. Closed-source systems like Solaris and HP-UX aren't just harder to use; they also make it hard for outside developers to contribute.
Open-source safety and competition between large and small companies
- OpenAI previously cited safety as a reason for not releasing its models. That raises the question of how much Meta can do if someone uses open-source Llama 2 for phishing, fraud, or fake news. Jia says that, as far as he knows, Llama's terms don't specify who can and can't use it. A provider can state that inhumane uses are prohibited, but it isn't a law enforcement agency. Actual enforcement still falls to courts and law enforcement agencies.
- The competitors discussed include OpenAI and Meta. Microsoft is allied with both, which is why some see it as the biggest player. Google has Google Brain, BERT, and DeepMind, while newcomer xAI has recruited researchers from OpenAI and DeepMind and hopes to use mathematical methods to solve problems with large models.
- Jia believes competition in technology ultimately comes down to talent. The small team of Geoffrey Hinton, Ilya Sutskever, and Alex Krizhevsky produced notable results in computer vision, showing that innovation doesn't necessarily require an industrial-scale effort. Small teams are more nimble. Large companies have more resources, but they have to spread them across multiple existing businesses, so they aren't necessarily the best places to innovate.
- On the business side, Jia believes AI has to be combined with specific activities like document editing or advertising to create value. Jasper does document editing but faces relatively high customer acquisition costs. If Microsoft added the rumored Copilot editing features to Office, by contrast, it could reach existing users directly. That's the advantage large companies have when rolling something out at scale.
- Jia attributes his emphasis on applications to his career path. He started out in research, but while many people were flocking to AI research, he moved from algorithms to systems and engineering, then into product work. He has also observed that emerging technologies often go through a period when everyone is especially excited about them.
Productivity applications and whether models have a moat
- After the AI frenzy dies down, people will learn the limits of what AI can do by trying it repeatedly, and development will shift toward integrating the technology into real products. Rather than hunting for a blockbuster app that can replace all human labor, it makes more sense to start with the actual needs of different industries and use AI to improve productivity in existing work.
- Pretty much every engineer at Jia's company has GitHub Copilot. He estimates that it writes about 40% of his code, mostly routine work like API calls and rewriting existing snippets. It roughly doubles his efficiency, but he still has to write complex code himself. Similar opportunities to save time exist in traditional businesses, such as finding the most useful onboarding materials for new employees among documents scattered in different places.
- Users don't care much about the technology behind Jasper. They care whether it works well and gives accurate results. Its founder had worked in advertising, SEO, and HR, which helped him apply models to real needs early while many others were busy training large ones. To hold on to its first-mover advantage, Jasper can't just call a general-purpose API. It can use higher-quality models and fine-tuning to get more accurate results in fields like law and finance.
- Jia tends to see keeping models closed as a business choice. He thinks closed-source models may have a six-month to one-year lead, but not necessarily a lasting moat. Outside users can use a closed-source model but can't study it. Even if open-source models start out weaker, they can catch up because it's easier to get involved and research can move faster.
- The more durable moat comes from two kinds of insight that talented people bring. One is doing what DeepMind does: producing research that others wouldn't think of or couldn't pull off. The other is understanding users and their businesses. Technical teams need to get beyond the trap of focusing only on their own needs, understand what customers are good at and where they struggle, and find where their technology can genuinely fill a gap.
- Users want an iPhone that's simple and easy to use, not a lecture from Apple on the most efficient way to use it. Likewise, technology providers shouldn't force customers to adapt to their way of doing things. Understanding users doesn't mean taking every request at face value, either. The provider's value is in turning those needs into scalable solutions that keep improving efficiency, then giving those solutions back to users.
Expert models, inference costs, and Lepton AI's role
- Jia thinks reports about GPT-4 partly confirmed his suspicions about mixture-of-experts models: Once a model reaches a certain size, simply piling more parameters and computing power into a single model may not be enough. It may need to branch out into distinct expert models. OpenAI has already started trying this approach to specialization.
- OpenAI has also been trying ways to bring large models' assistant capabilities into traditional systems, such as through OpenAI plugins. General-purpose large models still have room to develop, but you don't need a model on the level of someone with three PhDs every time you solve one specific problem. A specialized model may work better and cost less.
- The interview cites a training cost of 63 million for GPT-4. Based on past experience, Jia estimates that in a stable application, total inference costs are about 5 to 10 times the cost of training. With models currently being trained over and over in trial runs, training costs are more visible. Once a model proves useful and is widely used, repeated calls add up to substantial inference costs.
- Training is a fixed investment that has to be recouped through later sales of the application. Inference, as with Jasper calling OpenAI, grows roughly linearly with the number of users and business revenue. High inference costs can't simply be passed on through higher prices, because prices depend on demand and what users are willing to pay. Bringing inference costs down is what lets applications serve more use cases.
- Lepton AI wants to improve both system efficiency and people's productivity: It aims to handle more work with a limited number of machines and ease the shortage of AI specialists by letting application engineers and data scientists use models the way they use a microservice or database, so they can build applications quickly.
- Lepton AI mainly provides PaaS, helping customers set up machine learning platforms and manage open-source models or models they've fine-tuned themselves. It also offers a small set of standardized fine-tuning and deployment services that are closer to SaaS, so customers don't have to write a lot of code once their data is ready. It relies on existing cloud providers for the underlying IaaS, doesn't build its own clusters, and doesn't run customers' industry-specific operations or make business decisions for them.
Transformer limitations and early applications
- As models like GPT-4 and Llama move from proving that something is possible to being used in applications, inference costs become a practical concern: Every user request incurs an inference cost. Inference is relatively expensive with Transformers, so the two discuss whether architectures like RWKV and Retentive Network could replace them.
- Jia thinks extending the context so a Transformer can remember more is a bit like treating the same coworker as a new hire every day and retraining them with one long prompt. He hopes the Transformer itself or another architecture will offer a better approach, but doesn't claim to know which will win.
- If a better architecture proves itself, GPT and Llama may turn out to be stepping stones. Similar shifts have happened before. Before AlexNet, computer vision commonly used hand-extracted SIFT features and large-scale linear SVMs. In natural language understanding, RNNs and LSTMs were also gradually replaced by Transformers.
- A shift in architecture could affect the chip industry, but the open-source community can shorten the time from a paper to a working demo and get software into use faster. By contrast, sending a new chip to TSMC or another foundry for tape-out means waiting at least six months.
- Content creation and interaction with content are areas where results can be seen early. AIGC tools such as Midjourney can help advertisers design ads faster. The WeChat mini program Miaoya Camera uses a model similar to Stable Diffusion to turn 20 photos uploaded by a user into a digital likeness and photos in settings such as mountain climbing. When the app is popular, users may have to wait in line for a whole day after uploading their photos.
- Consumer apps draw users with novel, fun experiences or by meeting specific needs, such as sharing photos or taking ID photos. Some people joked that AI cameras would threaten HIMO. Productivity gains in business applications are often less visible, but companies are more willing to pay for them.
Startup life and putting models to work in companies
- Before meeting investors, the team noticed that the memo still said NewCo and rushed to find a name. They couldn't find a brain-related domain and eventually chose Lepton.AI. A lepton is a type of particle. The name reflects the team's preference for keeping things lightweight, and the association with electrons also brings to mind Andrew Ng's line: “AI is the new electricity.”
- Large companies put more emphasis on keeping large teams moving steadily, and the complexity of their businesses means more coordination across teams. Small companies can iterate faster and hold an all-hands meeting over lunch, giving them a better chance of putting ideas into practice quickly.
- Jia's day-to-day work at the startup has no fixed rhythm. He occasionally has to deal with rent and utilities, but spends more time talking to prospective customers about their problems and possible applications, keeping up with new technology, and coordinating customer needs with the pace of system development. Writing code comes second. He picks up work the team doesn't have time for, such as setting up CI and connecting the front-end and back-end APIs.
- Customers' two biggest problems are efficiency and not knowing how to put seemingly cool models to work in their actual businesses. The team co-wrote a blog post with LangChain and fine-tuned a model on information from financial reports so it would sound more like a CFO when answering questions. It wasn't necessarily a real-world application. The main goal was to test the full process of quickly fine-tuning base models like Llama on data from a specific field.
- Model hallucinations haven't been fully solved. Combining large models with traditional search may be one way forward. For example, Perplexity.ai cites sources sentence by sentence in its answers, and Bing uses a similar format. Information about financial reports found online may be fact or opinion, so it still needs to be checked. But finding the source and reading two paragraphs is more efficient than spending half an hour reading the whole report.
Leaving your comfort zone and exploring technology for the long term
- Jia believes the best way to grow is to leave your comfort zone and do something useful that you're not yet good at. When Xingdian invited him to join Alibaba, he suggested that Jia keep doing technical work while exploring how technology could fit with the business. Coordinating teams and dealing with customers could be more frustrating than writing code, but the experience taught Jia not to look at problems only in terms of his technical strengths.
- Starting a company was another step outside his comfort zone. Large tech companies are relatively stable and have room for long-term projects. Small companies often have a planning horizon of just six months. They have to think about survival first and find product-market fit as quickly as possible. Jia's startup team had fewer than 10 people at the time, and all the co-founders were already in place.
- When recruiting, Jia cares most about whether people are passionate about the work, not whether they can directly apply what they've done before, because the opportunities AI is bringing across industries are new to everyone. He would rather work with people who see a new technology and quickly build a demo to test an idea, whether the attempt succeeds or fails.
- Jia doesn't make a firm prediction about how competitive xAI will be because too little is publicly known. Musk's track record at SpaceX and Tesla makes him think this effort could go very well too. xAI has already brought together a group of talented people, so it has a chance to build something.
- Jia agrees with Richard Sutton's takeaway from 70 years of AI research: Using general methods to leverage large amounts of computing power also means using large amounts of data. Compared with hand-designed features, simple methods may find simpler statistical patterns when given enough data and compute. The results of combining data, compute, and statistical methods shouldn't be underestimated.
- The host points out that internal tensions at large companies can limit the research environment, while OpenAI is willing to commit a lot of money and computing power. Jia recalls that he didn't initially expect OpenAI to be so successful either. The team experimented early on with robotics and using a robotic hand for imitation. That history convinced him that a small group of smart people can build something different if they keep exploring, even after failures.
What it answers
- When does it make sense to use a closed-source API, and when is it worth self-hosting?
- During experiments or when usage is low, paying for API calls as needed is usually cheaper and easier. Self-hosting is more likely to come out ahead on cost and performance once the business grows and a general-purpose model can no longer meet its industry-specific needs.
- Why is releasing pretrained models more important than releasing training code alone?
- Releasing the code doesn't remove the barriers of data, computing power, and funding needed for training. Releasing an already trained model is what lets small teams fine-tune it directly and test their ideas.
- Why should mature applications pay more attention to inference than to training?
- Training is a fixed investment that the business has to recoup later. Inference incurs a cost every time the model is used. In Jia's experience, the total inference cost of a stable application can be five to ten times its training cost.
- How can a startup like Jasper respond to Microsoft adding AI to Office?
- Microsoft can rely on its existing users to make rollout easier. Jasper's advantage is that it knows real workflows in areas like advertising, but its next step still has to be improving accuracy in specific fields like law and finance, rather than just using a general-purpose API.
- How can companies use a model without fully trusting its answers?
- Jia says hallucinations haven't been fully solved. One option is to combine a large model with traditional search so answers come with sources that can be checked. Even if someone still has to verify the answer manually, jumping straight to the relevant paragraphs is faster than reading the entire financial report.
Quotes
所以我觉得 Llama 2 呢,相当于把这个神秘的黑盒子给打开,说你看它都可以用了。
So I think Llama 2 basically opens up this mysterious black box and says, look, you can actually use it now.
那意思 AI 本身不是一个业务,它需要和一个业务结合才能够产生价值。
What that means is that AI isn't a business in itself. It has to be combined with a business to create value.
因为到最后的话,其实任何一个商品的定价都不是根据成本来的,都是根据需求来的。
Because in the end, the price of any product isn't really based on cost. It's based on demand.
我宁可一个人先告诉我说去那个地方查,然后我去查了,然后我看,我看两段话是比较快的。
I'd rather have someone tell me, 'Go look there,' and then I go and check. Then I read, well, reading two paragraphs is pretty quick.
Mentioned
- 泓君 — The host of Silicon Valley 101, who asks Jia about barriers to development, business competition, costs, and more.
- 贾扬清 — The founder of Lepton AI, who helped develop several deep learning frameworks and previously led Alibaba's intelligent computing platform.
- Lepton AI — The company founded by Jia, focused on providing a machine learning platform and standardized services for fine-tuning and deploying models.
- Meta — The company that released Llama 2. The episode discusses how open models could help it build a developer ecosystem and attract talent.
- Llama 2 — The episode uses it to show how allowing commercial use and providing a pretrained model lower barriers to application development and research.
- OpenAI — The model provider whose API helped people experiment with building applications on top of its models. Jia sees it as a leader in both research and showing the way for applications.
- GPT-4 — Reports of a leak about it prompted a discussion of closed-source strategy, mixture-of-experts architecture, and model costs.
- Jasper — The episode uses it as an example of an AI writing application to discuss first-mover advantages in specific fields and competitive pressure from Office.
- GitHub Copilot — Jia draws on his own programming experience to explain how it can write routine code and improve productivity but can't replace engineers.
- Transformer — The episode discusses the inference cost of this model architecture and whether more efficient approaches will replace the practice of continually extending the context.
- Perplexity.ai — Jia uses it as an example of how Q&A-style search can cite sources so users can verify answers themselves.
- Richard Sutton — The episode cites this AI researcher's view that general methods combined with large amounts of computation and data may outperform hand-designed features.