Feb 8 Part 1: Background on LLMs
[Slides]

1. Introduction and motivation
2. Class structure and logistics
3. Language modeling overview
   1. Definitions
   2. A brief history of LMs
   3. LLM fundamentals
4. Overview of ways to train, tune, and prompt LLMs
   1. Fine-tuning, zero-shot prompts, few-shot prompts, chain-of-thought prompts
   2. Prompt tuning
5. Examples of LLM prompts

Part 2: Get your hands dirty with ChatGPT
[Slides]
Create a prompting task in groups. Required Readings:

1. Percy Liang’s introduction to LLMs

Feb 15 Part 1: Evaluating models
[Slides]

How can we best evaluate these models for accuracy, fairness, bias, robustness, and other factors?

Speaker: Rishi Bommasani (Stanford)
Title: Holistically Evaluating Language Models on the Path to Evaluating Foundation Models

Part 2: LLMs in Applications
[Slides]

People are increasingly interacting with human-facing tools that
incorporate LLMs, like ChatGPT, writing assistants, and character
generators. How might we go about evaluating these systems and their
impacts on people? In this session we will consider 10 recent commercial and research applications of LLMs.
Students will be asked to come prepared to critique the designs of one of these
applications along different dimensions that we will describe in week 1. Required Readings:

1. Holistic Evaluation of Language Models (HELM)

Recommended Readings:

1. On the Opportunities and Risks of Foundation Models
2. Discovering Language Model Behaviors with Model-Written Evaluations
3. All That’s ‘Human’ Is Not Gold: Evaluating Human Evaluation of Generated Text
4. Beyond Accuracy: Behavioral Testing of NLP Models with CheckList
5. How to do human evaluation: A brief introduction to user studies in NLP
6. Dynabench: Rethinking Benchmarking in NLP
7. Word Embeddings Quantify 100 years of Gender and Ethnic Stereotypes

Feb 22 Part 1: Using LLMs for Consensus Across Preferences
[Slides]

Speaker: Michiel Bakker (DeepMind)
Title: Fine-tuning Language Models to Find Agreement among Humans with Diverse Preferences

Part 2: Project Pitch
[Slides]

Students present their project idea and form teams.

Required Readings:

1. Fine-tuning Language Models to Find Agreement among Humans with Diverse Preferences
2. Engaging Politically Diverse Audiences on Social Media

Mar 1 Part 1: Emergent Abilities of LLMs
[Slides]
This talk will cover broad intuitions about how large language models work. First, we will begin by examining some examples of what language models can learn by reading the internet. Second, we will consider why language models have gained traction recently and what new abilities they have that were not present in the past. Third, we will cover how language models can perform complex reasoning tasks. Finally, the talk will discuss how language models can have an improved user interface via instruction following.

Speaker: Jason Wei (OpenAI)
Title: Emergence in Large Language Models

Part 2: NLP Evaluation Methods and Red Teaming
[Slides]

Required Readings:

1. Emergent Abilities of Large Language Models
2. Chain of Thought Prompting Elicits Reasoning in Large Language Models
3. Scaling Instruction-Finetuned Language Models

Recommended Readings:

1. Dissociating Language and Thought in Large Language Models: A Cognitive Perspective
2. Discovering Latent Knowledge in Language Models Without Supervision
3. Bridging the Gap: A Survey on Integrating (Human) Feedback for Natural Language Generation
4. Learning to Summarize with Human Feedback

Mar 8 Part 1: Evaluating Human-model Interactions
[Slides]

Speaker: Mina Lee (Stanford)
Title: Designing and Evaluating Language Models for Human Interaction

Part 2: Human Experiments and Evaluation Methods
[Slides]

Required Readings:

1. Evaluating Human-Language Model Interaction
2. CoAuthor: Designing a Human-AI Collaborative Writing Dataset for Exploring Language Model Capabilities

Recommended Readings:

1. Power to the People? Opportunities and Challenges for Participatory AI

Mar 15 Media Lab Research Panel

Speaker: Ziv Epstein (PhD Student at MIT Media Lab, Human Dynamics)
Title: Social Science Methods for Understanding Generative AI

Speaker: Matt Groh (PhD Student at MIT Media Lab