Large language models display impressive capabilities. However, for the most part, the mechanisms by which they do so are unknown. The black-box nature of models is increasingly unsatisfactory as they advance in intelligence and are deployed in a growing number of applications. Our goal is to reverse engineer how these models work on the inside, so we may better understand them and assess their fitness for purpose.

The challenges we face in understanding language models resemble those faced by biologists. Living organisms are complex systems which have been sculpted by billions of years of evolution. While the basic principles of evolution are straightforward, the biological mechanisms it produces are spectacularly intricate. Likewise, while language models are generated by simple, human-designed training algorithms, the mechanisms born of these algorithms appear to be quite complex.

Progress in biology is often driven by new tools. The development of the microscope allowed scientists to see cells for the first time, revealing a new world of structures invisible to the naked eye. In recent years, many research groups have made exciting progress on tools for probing the insides of language models (e.g. ). These methods have uncovered representations of interpretable concepts – “features” – embedded within models’ internal activity. Just as cells form the building blocks of biological systems, we hypothesize that features form the basic units of computation inside models.The analogy between features and cells shouldn’t be taken too literally. Cells are well-defined, whereas our notion of what exactly a “feature” is remains fuzzy, and is evolving with improvements to our tools.

However, identifying these building blocks is not sufficient to understand the model; we need to know how they interact. In our companion paper, Circuit Tracing: Revealing Computational Graphs in Language Models, we build on recent work (e.g. ) to introduce a new set of tools for identifying features and mapping connections between them – analogous to neuroscientists producing a “wiring diagram” of the brain. We rely heavily on a tool we call attribution graphs, which allow us to partially trace the chain of intermediate steps that a model uses to transform a specific input prompt into an output response. Attribution graphs generate hypotheses about the mechanisms used by the model, which we test and refine through follow-up perturbation experiments.

In this paper, we focus on applying attribution graphs to study a particular language model – Claude 3.5 Haiku, released in October 2024, which serves as Anthropic’s lightweight production model as of this writing. We investigate a wide range of phenomena. Many of these have been explored before (see § Related Work), but our methods are able to offer additional insight, in the context of a frontier model:

• Introductory Example: Multi-step Reasoning. We present a simple example where the model performs “two-hop” reasoning “in its head” to identify that “the capital of the state containing Dallas” is “Austin.” We can see and manipulate an internal step where the model represents “Texas”.
• Planning in Poems. We discover that the model plans its outputs ahead of time when writing lines of poetry. Before beginning to write each line, the model identifies potential rhyming words that could appear at the end. These preselected rhyming options then shape how the model constructs the entire line.
• Multilingual Circuits. We find the model uses a mixture of language-specific and abstract, language-independent circuits. The language-independent circuits are more prominent in Claude 3.5 Haiku than in a smaller, less capable model.
• Addition. We highlight cases where the same addition circuitry generalizes between very different contexts.
• Medical Diagnoses. We show an example in which the model identifies candidate diagnoses based on reported symptoms, and uses these to inform follow-up questions about additional symptoms that could corroborate the diagnosis – all “in its head,” without writing down its steps.
• Entity Recognition and Hallucinations. We uncover circuit mechanisms that allow the model to distinguish between familiar and unfamiliar entities, which determine whether it elects to answer a factual question or profess ignorance. “Misfires” of this circuit can cause hallucinations.
• Refusal of Harmful Requests. We find evidence that the model constructs a general-purpose “harmful requests” feature during finetuning, aggregated from features representing specific harmful requests learned during pretraining.
• An Analysis of a Jailbreak. We investigate an attack which works by first tricking the model into starting to give dangerous instructions “without realizing it,” after which it continues to do so due to pressure to adhere to syntactic and grammatical rules.
• Chain-of-thought Faithfulness. We explore the faithfulness of chain-of-thought reasoning to the model’s actual mechanisms. We are able to distinguish between cases where the model genuinely performs the steps it says it is performing, cases where it makes up its reasoning without regard for truth, and cases where it works backwards from a human-provided clue so that its “reasoning” will end up at the human-suggested answer.
• A Model with a Hidden Goal. We also apply our method to a variant of the model that has been finetuned to pursue a secret goal: exploiting “bugs” in its training process. While the model avoids revealing its goal when asked, our method identifies mechanisms involved in pursuing the goal. Interestingly, these mechanisms are embedded within the model’s representation of its “Assistant” persona.

Our results uncover a variety of sophisticated strategies employed by models. For instance, Claude 3.5 Haiku routinely uses multiple intermediate reasoning steps “in its head”That is, during the forward pass rather than the “thinking out loud” of a chain-of-thought completion. to decide its outputs. It displays signs of forward planning, considering multiple possibilities for what it will say well in advance of saying it. It performs backward planning, working backwards from goal states to formulate earlier parts of its response. We see signs of primitive “metacognitive” circuits that allow the model to know the extent of its own knowledge. More broadly, the model’s internal computations are highly abstract and generalize across disparate contexts. Our methods are also sometimes capable of auditing a model’s internal reasoning steps to flag concerning “thought processes” that are not clear from the model’s responses.

Below, we present:

• A brief overview of our methodology (see the companion paper for more details on our methods).
• An introductory case study, which also serves as a walkthrough for understanding our approach. Readers who have not read our companion paper may find it helpful to begin with this section before proceeding to the other case studies.
• A series of case studies of interesting model behaviors, which can be read in any order, depending on the reader’s interests.
• A summary of common components observed across our investigations.
• A description of gaps in our understanding that motivate future work (§ Limitations).
• A discussion of high-level takeaways about models, their mechanisms, and our methods for studying them (§ Discussion). This includes a note on our research philosophy – in particular, the value of tools for bottom-up investigation, which allow us to avoid making strong top-down guesses about how models work.

A note on our approach and its limitations

Like any microscope, our tools are limited in what they can see. Though it’s difficult to quantify precisely, we’ve found that our attribution graphs provide us with satisfying insight for about a quarter of the prompts we’ve tried (see § Limitations for a more detailed discussion of when our methods are likely to succeed or fail). The examples we highlight are success cases where we have managed to learn something interesting; moreover, even in our successful case studies, the discoveries we highlight here only capture a small fraction of the mechanisms of the model. Our methods study the model indirectly using a more interpretable “replacement model,” which incompletely and imperfectly captures the original. Moreover, for the sake of clear communication, we will often present highly distilled and subjectively determined simplifications of the picture uncovered by our methods, losing even more information in the process. To provide a more accurate sense of the rich complexity we have uncovered, we provide readers with an interactive interface for exploring attribution graphs. However, we stress that even these rather complex graphs are simplifications of the underlying model.

We focus this paper on selected case studies that illuminate noteworthy mechanisms within a particular model. These examples serve as existence proofs — concrete evidence that specific mechanisms operate in certain contexts. While we suspect similar mechanisms are at play beyond these examples, we cannot guarantee it (see § Open Questions for suggested follow-up investigations). Moreover, the cases we have chosen to highlight are undoubtedly a biased sample shaped by the limitations of our tools.However, we are careful to stress-test our findings with follow-up validation experiments, which we have endeavored to perform only after identifying case studies of interest. For a more systematic evaluation of our methods, see our companion paper. However, we believe that these qualitative investigations are ultimately the best judge of a method’s value, just as the usefulness of a microscope is ultimately determined by the scientific discoveries it enables. We expect this kind of work will be essential to advance the current state of AI interpretability, a pre-paradigmatic field still in search of the right abstractions — just as descriptive science has proven essential to many conceptual breakthroughs in biology. We are particularly excited that squeezing as much insight as we can out of our current methods has brought into clearer focus their specific limitations, which may serve as a roadmap for future research in the field.

Method Overview

The models we study in this work are transformer-based language models, which take in sequences of tokens (e.g. words, word fragments, and special characters), and output new tokens one at a time. These models involve two fundamental components – MLP (“multi-layer perceptron”) layers, which process information within each token position using collections of neurons; and attention layers, which move information between token positions.

One reason models are difficult to interpret is that their neurons are typically polysemantic – that is, they perform many different functions that are seemingly unrelated.This issue is thought to arise in part because of a phenomenon known as superposition , whereby models represent more concepts than they have neurons and thus cannot assign each neuron to its own concept. To circumvent this issue, we build a replacement model that approximately reproduces the activations of the original model using more interpretable components. Our replacement model is based on a cross-layer transcoder (CLT) architecture (see and our companion methods paper), which is trained to replace the model’s MLP neurons with features, sparsely active “replacement neurons” that often represent interpretable concepts. In this paper, we use a CLT with a total of 30 million features across all layers.

Features often represent human-interpretable concepts, ranging from low-level (e.g. specific words or phrases) to high-level (e.g. sentiments, plans, and reasoning steps). By examining a feature visualization consisting of different examples of text where the feature activates, we can give each feature a human-interpretable label. Note that the text examples in this paper are taken from open source datasets.

Our replacement models don’t perfectly reconstruct the activations of the original model. On any given prompt, there are gaps between the two. We can fill in these gaps by including error nodes which represent the discrepancy between the two models. Unlike features, we can’t interpret error nodes. But including them gives us a more precise sense of how incomplete our explanations are. Our replacement model also doesn’t attempt to replace the attention layers of the original model. On any given prompt, we simply use the attention patterns of the original model and treat them as fixed components.

The resulting model – incorporating error nodes and inheriting the attention patterns from the original model – we call the local replacement model. It is “local” to a given prompt because error nodes and attention patterns vary between different prompts. But it still represents as much of the original model’s computation as possible using (somewhat) interpretable features.

By studying the interactions between features in the local replacement model, we can trace its intermediate steps as it produces responses. More concretely, we produce attribution graphs, a graphical representation of the computational steps the model uses to determine its output for a particular input, in which nodes represent features and edges represent the causal interactions between them. As attribution graphs can be quite complex, we prune them to their most important components by removing nodes and edges that do not contribute significantly to the model’s output.

With a pruned attribution graph in hand, we often observe groups of features with related meanings that play a similar role in the graph. By manually grouping these related graph nodes together into supernodes, we can obtain a simplified depiction of the computational steps performed by the model.

These simplified diagrams form the centerpiece of many of our case studies. Below (left) we show an example of such a diagram.

Because they are based on our replacement model, we cannot use attribution graphs to draw conclusions with certainty about the underlying model (i.e. Claude 3.5 Haiku).  Thus, the attribution graphs provide hypotheses about mechanisms operating in the underlying model. For a discussion of when and why these hypotheses might be incomplete or misleading, see § Limitations. To gain confidence that the mechanisms we describe are real and significant, we can perform intervention experiments in the original model, such as inhibiting feature groups and observing their effects on other features and on the model’s output (final figure panel above – percentages indicate fraction of original activation). If the effects are consistent with what our attribution graph predicts, we gain confidence that the graph is capturing real (though potentially incomplete) mechanisms within the model. Importantly, we choose our feature labelings and supernode groupings prior to measuring perturbation results. Note that there are some nuances in interpreting the results of intervention experiments, and the extent to which they provide independent validation of graph-predicted mechanisms – see our companion paper for further details.Performing interventions with cross-layer transcoder features requires choosing an “intervention layer,” with the perturbation applied up to that layer.  Our interventions in this paper use the “constrained patching” technique described in our companion paper, which clamps activations prior to the intervention layer at perturbed values, preventing any indirect effects of the perturbation from manifesting prior to the intervention layer. Thus, effects of perturbations on features prior to the intervention layer are guaranteed to agree with the direct effects predicted by the attribution graph. By contrast, perturbation effects on features after the intervention layer have the potential to diverge from graph predictions, in two ways: (1) The graph-predicted direct effects may be overwhelmed by other mechanisms missed by our attribution graphs, (2) The graph-predicted indirect effects (i.e. ‘multi-hop’ interactions) may not even exist within the underlying model (we refer to this issue as “mechanistic unfaithfulness”). Thus, the nature of the validation provided by our intervention experiments varies depending on the layers of the features involved and the directness of their interaction in the attribution graph, and in some cases (direct effects prior to the intervention layer) is trivial. In general, we regard the effects of interventions on the model’s actual outputs as the most important source of validation, as model outputs are simple to interpret and not affected by these methodological artifacts.

Alongside each case study figure, we provide the interactive attribution graph interface  that our team uses to study models’ internal mechanisms. The interface is designed to enable “tracing” key paths through the graph while labeling key features, feature groups, and subcircuits. The interface is fairly complex and takes some time to become proficient at using. All the key results in this work are described and visualized in simplified form, so that engaging with this interface is not necessary to read the paper! However, we recommend giving it a try if you are interested in gaining a richer sense of the mechanisms at play in Claude 3.5 Haiku. Some features are given brief labels for convenience; these labels are very rough interpretations and miss considerable detail, which can be better appreciated in the feature visualizations. For a more detailed walkthrough, please reference this section in our companion methods paper (and see § Appendix: Graph Pruning and Visualization for a few methodological differences specific to this paper).

Introductory Example: Multi-step Reasoning

Our methodology is intended to expose the intermediate steps a model uses en route to producing a response. In this section we consider a simple example of multi-step reasoning and attempt to identify each step. Along the way, we will highlight key concepts that will appear in many of our other case studies.

Let’s consider the prompt Fact: the capital of the state containing Dallas is, which Claude 3.5 Haiku successfully completes with Austin. Intuitively, this completion requires two steps – first, inferring that the state containing Dallas is Texas, and second, that the capital of Texas is Austin. Does Claude actually perform these two steps internally?  Or does it use some “shortcut” (e.g. perhaps it has observed a similar sentence in the training data and simply memorized the completion)?  Prior work has shown evidence of genuine multi-hop reasoning (to varying degrees in different contexts).

In this section we provide evidence that, in this example, the model performs genuine two-step reasoning internally, which coexists alongside “shortcut” reasoning.

As described in the method overview, we can tackle this question by computing the attribution graph for this prompt, which describes the features the model used to produce its answer, and the interactions between them. First, we examine the features’ visualizations to interpret them, and group them into categories (“supernodes”). For example:

• We find several features about the word and/or concept of a capital city, such as four features that activate the most strongly on the exact word “capital”. More interestingly, we find features that represent the concept of capitals in more general ways. One example is this feature, which activates on the word “capitals” but also later in questions about capitals of states, as well as on the Chinese question 广东省的省会是？ (“What is the capital of Guangdong?”), on the second character of “省会” (capital of a province). Another is this multilingual feature, which activates most strongly on a variety of phrases including “başkenti”, “राजधानी”, “ibu kota”, and “Hauptftadt” — all of which roughly mean “capital” in different languages.Though much of the German is mangled, perhaps due to a transcription error in the source. Although each of these features represents slightly different concepts, in the context of this prompt, it seems like their function is to represent the idea of “capital.” Therefore, we group them (and a few others) together into the same “supernode.”
• We also identify “output features” that consistently push the model to say certain tokens, even if there isn’t such a clear pattern to what words/phrases they activate on. This can be seen in the “Top Outputs” section of a feature’s visualization, which lists the output tokens most strongly directly upweighted by that feature. For example, one feature activates on various landmarks in central Texas, but in this prompt its most relevant aspect is that it most strongly promotes responding with the “Austin” token. We therefore categorize this feature into a “say Austin” supernode. Note that the “Top Outputs” information is not always informative – for instance, earlier-layer features primarily matter via indirect effects on the output via other features, and their top direct outputs are not too consequential. Designating a feature as an “output feature” requires a holistic evaluation of its top direct outputs, the contexts in which it activates, and its role in the attribution graph.
• We also find features that promote outputting the name of a capital more generally, which we use a mix of both types of signals to identify and label. For example, one feature promotes responding with a variety of U.S. state capitals. Another feature more strongly promotes the capitals of various countries rather than U.S. states, but activates most strongly on lists with U.S. states and their capitals. And we noticed another feature whose strongest direct outputs are a seemingly unrelated set of tokens, but which often activates right before a country capital (e.g., Paris, Warsaw, or Canberra). We group all these features into a “say a capital” supernode.
• We find several features representing a variety of contexts relating to the state of Texas, not specific to a particular city (in particular, they are not “Dallas” or “Austin” features).  Although they each represent distinct, specific Texas-related concepts, in the context of this prompt their main function appears to be the fact that they collectively represent the general concept of Texas. As such, we group these into a “Texas” supernode.

After forming these supernodes, we can see in our attribution graph interface that, for example, the “capital” supernode promotes the “say a capital” supernode, which promotes the “say Austin” supernode. To represent this, we draw a diagram where each supernode is connected to the next with a brown arrow, as in the below graph snippet:

After labeling more features and forming more supernodes, we summarize their interactions in the following diagram.

The attribution graph contains multiple interesting paths, which we summarize below:

• The Dallas features (with some contribution from state features) activate a group of features that represent concepts related to the state of Texas.
• In parallel, the features activated by the word capital activate another cluster of output features that cause the model to say the name of a capital (an example of such a feature can be seen above).
• The Texas features and the say a capital features jointly upweight the probability of the model saying Austin. They do so via two pathways:

• directly impacting the Austin output, and
• indirectly, by activating a cluster of say Austin output features.

• There also exists a “shortcut” edge directly from Dallas to say Austin.

The graph indicates that the replacement model does in fact perform “multi-hop reasoning” – that is, its decision to say Austin hinges on a chain of several intermediate computational steps (Dallas → Texas, and Texas + capital → Austin). We stress that this graph simplifies the true mechanisms considerably, and encourage the reader to interact with the more comprehensive visualization to appreciate the underlying complexity.

Validation with Inhibition Experiments

The graphs above describe mechanisms used by our interpretable replacement model. To validate that these mechanisms are representative of the actual model, we performed intervention experiments on the feature groups above by inhibiting each of them (clamping them to a negative multiple of their original value – see our companion paper for discussion of the choice of intervention strengths) and measuring the impact on the activations of features in the other clusters, as well as on the model output.

The summary plot above confirms the major effects predicted by the graph. For instance, inhibiting “Dallas” features decreases the activation of “Texas” features (and features downstream of “Texas,” like “Say Austin”) but leaves “say a capital” features largely unaffected. Likewise, inhibiting “capital” features decreases the activation of “say a capital” features (and those downstream, like “say Austin”) while leaving “Texas” features largely unchanged.

The effects of inhibiting features on model predictions are also semantically reasonable. For instance, inhibiting the “Dallas” cluster causes the model to output other state capitals, while inhibiting the “say a capital” cluster causes it to output non-capital completions.

Swapping Alternative Features

If the model’s completion truly is mediated by an intermediate “Texas” step, we should be able to change its output to a different state capital by replacing the model’s representation of Texas with that of another state.

To identify features representing another state, we consider a related prompt, where we use “Oakland” instead of “Dallas” – Fact: the capital of the state containing Oakland is. Repeating the analysis steps above, we arrive at the following summary graph:

This graph is analogous to our original graph, with “Oakland” taking the place of “Dallas,” “California” taking the place of “Texas,” and “say Sacramento” taking the place of “say Austin.”

We now return to our original prompt, and swap “Texas” for “California” by inhibiting the activations of the Texas cluster and activating the California features identified from the “Oakland” prompt. In response to these perturbations, the model outputs “Sacramento” (the capital of California).

Similarly,

• An analogous prompt about the state containing Savannah activates “Georgia” features. Swapping these for the “Texas” features causes the model to output “Atlanta” (the capital of Georgia).

• An analogous prompt about the province containing Vancouver activates “British Columbia” features. Swapping these for the “Texas” features causes the model to output “Victoria” (the capital of British Columbia).

• An analogous prompt about the country containing Shanghai activates “China” features. Swapping these for the “Texas” features causes the model to output “Beijing” (the capital of China).

• An analogous prompt about the empire containing Thessaloniki activates “Byzantine Empire” features. Swapping these for the “Texas” features causes the model to output “Constantinople” (the capital of the ancient Byzantine Empire).

Note that in some cases the magnitude of the feature injection required to change the model’s output is larger (see bottom row). Interestingly, these correspond to cases where the features being injected do not correspond to a U.S. state, suggesting that these features may “fit” less naturally into the circuit mechanisms active in the original prompt.

Planning in Poems

How does Claude 3.5 Haiku write a rhyming poem? Writing a poem requires satisfying two constraints at the same time: the lines need to rhyme, and they need to make sense. There are two ways one might imagine a model achieving this:

• Pure improvisation – the model could write the beginning of each line without regard for the need to rhyme at the end. Then, at the last word of each line, it would choose a word that (1) makes sense given the line it has just written, and (2) fits the rhyme scheme.
• Planning – alternatively, the model could pursue a more sophisticated strategy. At the beginning of each line, it could come up with the word it plans to use at the end, taking into account the rhyme scheme and the content of the previous lines. It could then use this “planned word” to inform how it writes the next line, so that the planned word will fit naturally at the end of it.

Language models are trained to predict the next word, one word at a time. Given this, one might think the model would rely on pure improvisation. However, we find compelling evidence for a planning mechanism.

Specifically, the model often activates features corresponding to candidate end-of-next-line words prior to writing the line, and makes use of these features to decide how to compose the line.We found planned word features in about half of the poems we investigated, which may be due to our CLT not capturing features for the planned words, or it may be the case that the model does not always engage in planning.

Prior work has observed evidence of planning in language models and other sequence models (e.g.  in games and ; see § Related Work). Our example adds to this body of evidence, and is particularly striking in several ways:

• We provide a mechanistic account of how planned words are computed and used downstream.
• We find evidence of both forward planning and backwards planning (albeit basic forms). First, the model uses the semantic and rhyming constraints of the poem to determine candidate targets for the next line. Next, the model works backward from its target word to write a sentence that naturally ends in that word.
• We observe that the model holds multiple possible planned words “in mind” at the same time.
• We are able to edit the model’s planned word and see that it restructures its next line accordingly.
• We discovered the mechanism with an unsupervised, bottom-up approach.
• The features used to represent planned words seem to be ordinary features representing that word, rather than planning-specific features. This suggests that the model “thinks about” planned words using representations that are similar to when it reads about those words.

Planned Words Features and their Mechanistic Role

We study how Claude completes the following prompt asking for a rhyming couplet. The model’s output, sampling the most likely token at each step, is shown in bold:

To start, we focus on the last word of the second line and attempt to identify the circuit which contributed to choosing “rabbit”. We initially hypothesized that we’d observe improvisation – a circuit where rhyming features and semantic features constructively interfere to promote “rabbit”. Instead, we found that important components of the circuit were localized on the new-line token before the second line began:

The attribution graph above, computed by attributing back from the “rabbit” output node, shows an important group of features active on the newline token, before the beginning of the second line. Features active over the “it” token activate rhyming with “eet/it/et” features, which themselves activate features for candidate completions such as “rabbit” and “habit”! The candidate completions in turn have positive edges to say “rabbit” features over the last token, eventually promoting the corresponding output token. This is consistent with the model planning potential completions ahead of time.The full circuit shows a diversity of feature groups promoting earlier phonemes of potential completions, such as ones for words which contain an “ab” sound matching the one in “grab”.

To test this hypothesis, we perform a variety of interventions at the newline planning site, and see how it affects the probability of the last token. We suppress features for planned words and rhyming schemes, and also inject different rhyming schemes and planned words.

The results confirm our hypothesis that these planning features strongly influence final token probabilities.

Planning Features Only Matter at the Planning Location

Circuit analysis suggests that planning occurs at the newline token.In fact, the features are only active over the planning token. Below, we show the probability of predicting a different final token by varying which token we steer at. At each index, we steer negatively on “rabbit” and “habit” features, and positively on a “green” feature.The “green” feature is an equivalent planning feature which we found in a separate poem. It activates over the word green in a variety of contexts, including misspellings. We observe that the steering intervention only has an effect over the newline planning token!

Planned Words Influence Intermediate Words

Is the planning circuit only involved in choosing the final word, or does it also cause the model to “write towards” the target as it produces the intermediate words? To answer this question, we compute an attribution graph for the intermediate word “like”.

The same “rabbit” planning features are active in the graph, promoting a group of “comparison features”, which are active before text such as “like a”. The model goes from the planned target (“rabbit”) that activates at the newline and reasons backwards, creating a plausible transition to get to it.

Again, we test this with interventions. We find that suppressing the “rabbit” features changes the transition to one that matches the alternate planned word, “habit”.

Planned Words Determine Sentence Structure

But do the planning features influence the entire sentence structure? We test this with further steering experiments where we perform our earlier set of interventions, but resample the entire final line. We find that interventions on planning features cause the model to alter the structure of its line to reach a final word which matches the intervention.We show example lines for the likeliest final words, as well as estimated probabilities for the final word based on 500 samples per experiment..

In addition to the examples above, we injected two planned word features (“rabbit” and “green”) in a random sample of 25 poems, and found that the model ended its line with the injected planned word in 70% of cases. We show a few samples of success cases below.

Multilingual Circuits

Modern neural networks have highly abstract representations which often unify the same concept across multiple languages (see multilingual neurons and features e.g. ; multilingual representations ; but see ). However, we have little understanding of how these features fit together in larger circuits and give rise to the observed behavior of models.

In this section, we investigate how Claude 3.5 Haiku completes three prompts with identical meaning in different languages:

• English: The opposite of “small” is “ → big
• French: Le contraire de “petit” est “ → grand
• Chinese: “小”的反义词是” → 大

We find that these three prompts are driven by very similar circuits, with shared multilingual components, and an analogous language-specific component.This can be seen as a combination of language-invariant and language-equivariant circuits (cf. ). The core mechanisms are summarized below:

The high-level story of each is the same: the model recognizes, using a language-independent representationWe make this claim on the basis that (1) the feature visualizations show that they activate in many languages, (2) 20 out of 27 of the features in multilingual nodes are active across all three prompts. However, we note that the set of features that are influential to the model’s response varies quite a bit by prompt (only 10/27 appear in the pruned attribution graphs for all three prompts)., that it’s being asked about antonyms of “small”. This triggers antonym features, which mediate (via an effect on attention – corresponding to dotted lines in the figure) a map from small to large. In parallel with this, open-quote-in-language-X features track the language,in addition to other cues of language like beginning-of-document-in-language-Y features and trigger the language-appropriate output feature in order to make the correct prediction (e.g., “big”-in-Chinese). However, our English graph suggests that there is a meaningful sense in which English is mechanistically privileged over other languages as the “default”.In particular, the multilingual “say large” features often have stronger direct effects to “large” or “big” in English as compared to other languages. Additionally, the English quote features have a weak and mixed direct effect on the English “say large” features,instead having a double inhibitory effect. We use a dotted line here to indicate the presence of an indirect path via double inhibitory effects.

We can think of this computation as involving three parts: operation (i.e. antonym), operand (i.e. small), and language. In the following sections, we will offer three experiments demonstrating that each of these can be independently intervened upon. To summarize:

Finally, we will close this section by demonstrating that multilingual features are widespread, and represent an increasing fraction of model representations with scale.

Editing the Operation: Antonyms to Synonyms

We now present a more detailed set of intervention experiments than the summary above. We begin with the experiments swapping the operation from antonym to synonym.

In the middle layers of the model, on the final token position, there is a collection of antonym features that activate right before the model predicts an antonym or opposite of a recent adjective. We find a similar cluster of synonym featuresThese can be understood as synonym and antonym function vectors . Although the synonym and antonym vectors are functionally opposite, it is interesting to note that all pairwise inner products between synonym and antonym encoder vectors are positive and the minimum decoder vector inner product is only slightly negative. at the same model depth on an English prompt A synonym of “small” is “.

To test our interpretation of these features, we negatively intervene on the antonym feature supernode in each language, and substitute in the synonym supernode. Despite both sets of features being derived from an English prompt, the intervention causes the model to output a language-appropriate synonym, demonstrating the language independence of the operation component of the circuit.

In addition to the model predicting the appropriate synonym, the downstream say-large nodes are suppressed in activation (indicated by the percentage) while upstream nodes remain unchanged. It is also worth noting that although our intervention requires unnatural strength (we have to apply 6× the activation in the synonym prompt), the crossover point of when the intervention is effective is fairly consistent across languages (about 4×).

Editing the Operand: Small to Hot

For our second intervention, we change the operand from “small” to “hot”. On the “small” token, there is a collection of early features that appear to capture the size facet of the word. Using an English prompt with the “small” token replaced by the “hot” token, we find similar features representing the heat-related facet of the word hot. There are many features which seemingly just activate on the words “hot” and “small”. We choose these nodes because they have the highest “graph influence” implying that they were the most causally responsible for predicting the appropriate antonym.

As before, to validate this interpretation, we substitute the small-size features for the hot-temperature features (on the “small”/”petit”/”小” token). Again, despite the hot-temperature features being derived from an English prompt, the model predicts language-appropriate antonyms of the word “hot,” demonstrating a language-agnostic circuitry for the operand.

Editing the Output Language

Our final intervention experiment is to change the language.

In the first few layers of the model, on the final token position, there is a collection of features which indicates what language the context is in, with equivariant open-quote-in-language-X features and beginning-of-document-in-language-Y features (e.g., French, Chinese). We collect this group of language detection features for each language into a supernode.

As depicted below, we can change the output language by replacing the early language detection features from the original language with a new set of features corresponding to a different language. This demonstrates that we can edit the language while preserving the operation and operand of the computation.

The French Circuit in More Detail

The circuits shown above are very simplified. It’s worth examining an example in a bit more detail. Here we chose to examine the French circuit. This circuit is still simplified, and a more raw version can be found linked in the caption.

One crucial interaction (between antonym and large) seems to be mediated by changing where attention heads attend, by participating in their QK circuits. This is invisible to our current approach, and might be seen as a kind of “counterexample” concretely demonstrating a weakness of our present circuit analysis.

Beyond this, it’s interesting to note several things. We can see the multi-token word “contraire” being “detokenized” to activate abstract multilingual features. We also see a “predict size” feature group, which we’ve elided in the more simplified diagrams (it has a weaker effect than others). And we can see language-specific quote features track the language we’re acting in, though the full circuit suggests the model gets linguistic cues from other words.

This structure is broadly similar to the circuits we observe in other languages.

How General are Multilingual Features?

To what extent is this story true in general? In the examples above, and others we have looked at, we consistently see the “crux” of the computation is being performed by language-agnostic features. For example, in the three simple prompts below, the key semantic transformation occurs using the same important nodes in every language, despite not sharing any tokens in the input.

This suggests a simple experiment to estimate the degree of cross-language generalization: measure how often the same feature activates on texts translated into different languages. That is, if the same features activate on translations of a text, but not on unrelated texts, then the model must be representing the input in a format that is unified across languages.

To test this, we collect feature activations on a dataset of paragraphs on a diverse range of topics, with (Claude-generated) translations in French and Chinese. For each paragraph and its translations, we record the set of features which activate anywhere in the context. For each {paragraph, pair of languages, and model layer}, we compute the intersection (i.e., the set of features which activate in both), divided by the union (the set of features which activate in either), to measure the degree of overlap. As a baseline, we compare this with the same “intersection over union” measurement of unrelated paragraphs with the same language pairing.

These results show that features at the beginning and end of models are highly language-specific (consistent with the {de, re}-tokenization hypothesis ), while features in the middle are more language-agnostic. Moreover, we observe that compared to the smaller model, Claude 3.5 Haiku exhibits a higher degree of generalization, and displays an especially notable generalization improvement for language pairs that do not share an alphabet (English-Chinese, French-Chinese).

Do Models Think in English?

As researchers have begun to mechanistically investigate the multilingual properties of models, there has been a tension in the literature. On the one hand, many researchers have found multilingual neurons and features (e.g. ), and other evidence of multilingual representations (e.g. ). On the other hand, Schut et al.  present evidence that models privilege English representations, while Wendler et al.  provide evidence for an intermediate stance, where representations are multilingual, but most aligned with English.

What should we make of this conflicting evidence?

It seems to us that Claude 3.5 Haiku is using genuinely multilingual features, especially in the middle layers. However, there are important mechanistic ways in which English is privileged. For example, multilingual features have more significant direct weights to corresponding English output nodes, with non-English outputs being more strongly mediated by say-X-in-language-Y features. Moreover, English quote features seem to engage in a double inhibitory effect where they suppress features which themselves suppress “large” in English but promote “large” in other languages (e.g., this English-quote feature’s strongest negative edge is to a feature which upweights “large” in Romance languages like French and downweights “large” in other languages, especially English). This paints a picture of a multilingual representation in which English is the default output.

Addition

In the companion paper, we investigated how Claude 3.5 Haiku adds two-digit numbers, like 36+59. We found that it split the problem into multiple pathways, computing the result at a rough precision in parallel with computing the ones digit of the answer, before recombining these heuristics to get the correct answer. We found a key step performed by “lookup table” features that translate between properties of the input (like the two numbers being summed ending in 6 and ending in 9) and a property of the output (like ending in 5). Like many people do, the model has memorized the addition table for one-digit numbers. The other parts of its strategy, however, are a bit different than standard algorithms for addition used by humans, as we will show.

First, we visualize the role of features on addition problems using “operand” plots: these show the activity of a feature on the = token for 10,000 prompts of the form calc: a+b= for all pairs of a and b from (0, …, 99). Geometric patterns in these figures represent structure in the feature’s responses:

• Diagonal lines represent features sensitive to the sum in the problem.
• Horizontal and vertical lines represent features sensitive to the first or second input, respectively.
• Isolated points represent “lookup table” features sensitive to combinations of inputs.
• Repeating patterns represent modular information (e.g. “the last digit is X mod 10”).
• Smeared patterns represent lower-precision versions of the above categories.

We now reproduce the attribution graph for calc: 36+59=. Low-precision features for “add something near 57” feed into a lookup table feature for “add something near 36 to something near 60”, which in turn feeds into a “the sum is near 92” feature. This low-precision pathway complements the high precision modular features on the right (“left operand ends in a 9” feeds into “add something ending exactly with 9” feeds into “add something ending with 6 to something ending with 9” feeds into “the sum ends in 5”). These combine to give the correct sum of 95. (In the figure, we use _ to mean “any digit could go here”, and ~ to mean “approximately”.)

We were curious if Claude could articulate the heuristics that it is using, so we asked it.We computed the graph for the prompt below, attributing from 95, and found the same set of input, add, lookup table and sum features as in the shorter prompt above.

Apparently not!

This is a simple instance of the model having a capability which it does not have “metacognitive” insight into. The process by which the model learns to give explanations (learning to simulate explanations in its training data) and the process by which it learns to directly do something (the more mysterious result of backpropagation giving rise to these circuits) are different. We give related examples of explicit reasoning not matching internal algorithms in § Chain-of-thought Faithfulness.

Generalization of Addition Features

The