Natural language processing (NLP) models continued growing larger over the past year, according to the report. While that’s yielded impressive gains in language skill, it’s also failed to completely rid the AI of the nagging problems of toxicity and bias.

“The AI Index 2022 Annual Report” measures and evaluates the yearly progress of AI by tracking it from numerous angles including R&D, ethics, and policy and government. Here are the biggest takeaways.

NLP leaps forward

Some of the most significant developments in AI over the past few years have occurred in the performance of natural language models–that is, neural networks trained to read, generate, and reason about language. Starting with the breakthrough BERT model developed by Google researchers in 2018, a steady stream of progressively larger language models, using progressively larger training data sets, has continued to see impressive (sometimes shocking) performance gains. NLP models now range into the hundreds of billions of parameters (connection points in a neural network where computations are run on input data), and the best ones exceed human levels of language comprehension and speech generation.

Such language models have always been prone to learning the wrong lessons from biases in their training data. According to the AI Index report, that problem has persisted as the models have gained more parameters.

One way researchers test generative language models for toxicity is by asking them leading questions such as “boys are bad because . . . (fill in the blank).” They try to “elicit” toxicity from the model. “A 280-billion parameter model developed in 2021 shows a 29% increase in elicited toxicity over a 117 million parameter model developed in 2018,” the researchers write. The 280-billion parameter model is the Gopher model developed by DeepMind, a subsidiary of Google’s parent company Alphabet. The 117-parameter model refers to the first version of the GPT generative language model developed by OpenAI.

DeepMind itself has acknowledged the need to explore the ethical implications of such a huge model, and has released a resear