2023-03-27 • 20 min read

Summary #

• William Eden forecasts an AI winter. He argues that AI systems (1) are too unreliable and too inscrutable, (2) won’t get that much better (mostly due to hardware limitations) and/or (3) won’t be that profitable. He says, “I’m seeing some things that make me think we are in a classic bubble scenario, and lots of trends that can’t clearly continue.”
• I put 5% on an AI winter happening by 2030, with all the robustness that having written a blog post inspires, and where AI winter is operationalised as a drawdown in annual global AI investment of ≥50%. (I reckon a winter must feature not only decreased interest or excitement, but always also decreased funding, to be considered a winter proper.)
• There have been two previous winters, one 1974-1980 and one 1987-1993. The main factor causing these seems to have been failures to produce formidable results, and as a consequence wildly unmet expectations. Today’s state-of-the-art AI systems show impressive results and are more widely adopted (though I’m not confident that the lofty expectations people have for AI today will be met).
• I think Moore’s Law could keep going for decades. But even if it doesn’t, there are many other areas where improvements are being made allowing AI labs to train ever larger models: there’s improved yields and other hardware cost reductions, improved interconnect speed and better utilisation, algorithmic progress and, perhaps most importantly, an increased willingness to spend. If 1e35 FLOP is enough to train a transformative AI (henceforth, TAI) system, which seems plausible, I think we could get TAI by 2040 (>50% confidence), even under fairly conservative assumptions. (And a prolonged absence of TAI wouldn’t necessarily bring about an AI winter; investors probably aren’t betting on TAI, but on more mundane products.)
• Reliability is definitely a problem for AI systems, but not as large a problem as it seems, because we pay far more attention to frontier capabilities of AI systems (which tend to be unreliable) than long-familiar capabilities (which are pretty reliable). If you fix your gaze on a specific task, you usually see a substantial and rapid improvement in reliability over the years.
• I reckon inference with GPT-3.5-like models will be about as cheap as search queries are today in about 3-6 years. I think ChatGPT and many other generative models will be profitable within 1-2 years if they aren’t already. There’s substantial demand for them (ChatGPT reached 100M monthly active users after two months, quite impressive next to Twitter’s ~450M) and people are only beginning to explore their uses.
• If an AI winter does happen, I’d guess some of the more likely reasons would be (1) scaling hitting a wall, (2) deep-learning-based models being chronically unable to generalise out-of-distribution and/or (3) AI companies running out of good-enough data. I don’t think this is very likely, but I would be relieved if it were the case, given that we as a species currently seem completely unprepared for TAI.

The Prospect of a New AI Winter #

What does a speculative bubble look like from the inside? Trick question – you don’t see it.

Or, I suppose some people do see it. One or two may even be right, and some of the others are still worth listening to. William Eden tweeting out a long thread explaining why he’s not worried about risks from advanced AI is one example, I don’t know of which. He argues in support of his thesis that another AI winter is looming, making the following points:

1. AI systems aren’t that good. In particular (argues Eden), they are too unreliable and too inscrutable. It’s far harder to achieve three or four nines reliability than merely one or two nines; as an example, autonomous vehicles have been arriving for over a decade. The kinds of things you can do with low reliability don’t capture most of the value.
2. AI systems won’t get that much better. Some people think we can scale up current architectures to AGI. But, Eden says, we may not have enough compute to get there. Moore’s law is “looking weaker and weaker”, and price-performance is no longer falling exponentially. We’ll most likely not get “more than another 2 orders of magnitude” of compute available globally, and 2 orders of magnitude probably won’t get us to TAI. “Without some major changes (new architecture/paradigm?) this looks played out.” Besides, the semiconductor supply chain is centralised and fragile and could get disrupted, for example by a US-China war over Taiwan.
3. AI products won’t be that profitable. AI systems (says Eden) seem good for “automating low cost/risk/importance work”, but that’s not enough to meet expectations. (See point (1) on reliability and inscrutability.) Some applications, like web search, have such low margins that the inference costs of large ML models are prohibitive.

I’ve left out some detail and recommend reading the entire thread before proceeding. Also before proceeding, a disclosure: my day job is doing research on the governance of AI, and so if we’re about to see another AI winter, I’d pretty much be out of a job, as there wouldn’t be much to govern anymore. That said, I think an AI winter, while not the best that can happen, is vastly better than some of the alternatives, axiologically speaking. I also think I’d be of the same opinion even if I had still worked as a programmer today (assuming I had known as much or little about AI as I actually do).

Past Winters #

There is something of a precedent.

The first AI winter – traditionally, from 1974 to 1980 – was precipitated by the unsympathetic Lighthill report. More fundamentally it was caused by AI researchers’ failure to achieve their grandiose objectives. In 1965, Herbert Simon famously predicted that AI systems would be capable of any work a human can do in 20 years, and Marvin Minsky wrote in 1967 that “within a generation […] the problem of creating ‘artificial intelligence’ will be substantially solved”. Of Frank Rosenblatt’s Perceptron Project the New York Times reported (claims of Rosenblatt which aroused ire among other AI researchers due to their extravagance), “[It] revealed an embryo of an electronic computer that it expects will be able to walk, talk, see, write, reproduce itself and be conscious of its existence. Later perceptrons will be able to recognize people and call out their names and instantly translate speech in one language to speech and writing in another language, it was predicted” (Olazaran 1996). Far from human intelligence, not even adequate machine translation materialised (it took until the mid-2010s when DeepL and Google Translate’s deep learning upgrade were released for that to happen).

The second AI winter – traditionally, from 1987 to 1993 – again followed unrealised expectations. This was the era of expert systems and connectionism (in AI, the application of artificial neural networks). But expert systems failed to scale, and neural networks learned slowly, had low accuracy and didn’t generalise. It was not the era of 1e9 FLOP/s per dollar; I reckon the LISP machines of the day were ~6-7 orders of magnitude less price-performant than that.

Wikipedia lists a number of factors behind these winters, but to me it is the failure to actually produce formidable results that seems most important. Even in an economic downturn, and even with academic funding dried up, you still would’ve seen substantial investments in AI had it shown good results. Expert systems did have some success, but nowhere near what we see AI systems do today, and with none of the momentum but all of the brittleness. This seems like an important crux to me: will AI systems fulfil the expectations investors have for them?

Moore’s Law and the Future of Compute #

Improving these days means scaling up. One reason why scaling might fail is if the hardware that is used to train AI models stops improving.

Moore’s Law is the dictum that the number of transistors on a chip will double every ~2 years, and as a consequence hardw