About a month ago, the CPython project merged a new implementation strategy for their bytecode interpreter. The initial headline results were very impressive, showing a 10-15% performance improvement on average across a wide range of benchmarks across a variety of platforms.

Unfortunately, as I will document in this post, these impressive performance gains turned out to be primarily due to inadvertently working around a regression in LLVM 19. When benchmarked against a better baseline (such GCC, clang-18, or LLVM 19 with certain tuning flags), the performance gain drops to 1-5% or so depending on the exact setup.

When the tail-call interpreter was announced, I was surprised and impressed by the performance improvements, but also confused: I’m not an expert, but I’m passingly-familiar with modern CPU hardware, compilers, and interpreter design, and I couldn’t explain why this change would be so effective. I became curious – and perhaps slightly obsessed – and the reports in this post are the result of a few weeks of off-and-on compiling and benchmarking and disassembly of dozens of different Python binaries, in an attempt to understand what I was seeing.

At the end, I will reflect on this situation as a case study in some of the challenges of benchmarking, performance engineering, and software engineering in general.

I also want to be clear that I still think the tail-calling interpreter is a great piece of work, as well as a genuine speedup (albeit more modest than initially hoped). I am also optimistic it’s a more robust approach than the older interpreter, in ways I’ll explain in this post. I also really don’t want to blame anyone on the Python team for this error. This sort of confusion turns out to be very common – I’ve certainly misunderstood many a benchmark myself – and I’ll have some reflections on that topic at the end.

In addition, the impact of the LLVM regression doesn’t seem to have been known prior to this work (and the bug still isn’t fixed, as of this writing); thus, in that sense, the alternative (without this work) probably really was 10-15% slower, for builds using clang-19 or newer. For instance, Simon Willison reproduced the 10% speedup “in the wild,” as compared to Python 3.13, using builds from python-build-standalone.

Here are my headline results. I benchmarked several builds of the CPython interpreter, using multiple different compilers and different configuration options, on two machines: an Intel server (a Raptor Lake i5-13500 I maintain in Hetzner), and my Apple M1 Macbook Air. You can reproduce these builds using my nix configuration, which I found essential for managing so many different moving pieces at once.

All builds use LTO and PGO. These configurations are:

• clang18: Built using Clang 18.1.8, using computed gotos.
• gcc (Intel only): Built with GCC 14.2.1, using computed gotos.
• clang19: Built using Clang 19.1.7, using compute gotos.
• clang19.tc: Built using Clang 19.1.7, using the new tail-call interpreter.
• clang19.taildup: Built using Clang 19.1.7, computed gotos and some -mllvm tuning flags which work around the regression.

I’ve used clang18 as the baseline, and reported the bottom-line “average” reported by pypeformance/pyperf compare_to. You can find the complete output files and reports on github.

Observe that the tail-call interpreter still exhibits a speedup as compared to clang-18, but that it’s far less dramatic than the slowdown from moving to clang-19. The Python team has also observed larger speedups than I have (after accounting for the bug) on some other platforms.

A brief background  🔗︎

A classic bytecode interpreter consists of a switch statement inside of a while loop, looking something like so:

while (true) {
  opcode_t this_op = bytecode[pc++];
  switch (this_op) {
    case OP_IMM: {
      // push an immediate onto the stack
      break;
    }
    case OP_ADD: {
      // handle the add
      break;
    }
    // etc
  }
}

Most compilers will compile the switch into a jump table – they will emit a table containing the address of each case OP_xxx block, index into it with the opcode, and perform an indirect jump.

It’s long been known that you can speed up a bytecode interpreter of this style by replicating the jump table dispatch into the body of each opcode. That is, instead of ending each opcode with a jmp loop_top, each opcode contains a separate instance of the “decode next instruction and index through the jump table” logic.

Modern C compilers support taking the address of labels, and then using those labels in a “computed goto,” in order to implement this pattern. Thus, many modern bytecode interpreters, including CPython (before the tail-call work), employ an interpreter loop that looks something like:

static void *opcode_table[256] = {
    [OP_IMM] = &&TARGET_IMM,
    [OP_ADD] = &&TARGET_ADD,
    // etc
};

#define DISPATCH() goto *opcode_table[bytecode[pc++]]

DISPATCH();

TARGET_IMM: {
    // push an immediate onto the stack
    DISPATCH();
}
TARGET_ADD: {
    // handle the add
    DISPATCH();
}

Computed goto in LLVM  🔗︎

For performance reasons (performance of the compiler, not the generated code), it turns out that Clang and LLVM, internally, actually merges all of the gotos in the latter code into a single indirectbr LLVM instruction, which each opcode will jump to. That is, the compiler takes our hard work, and deliberately rewrites into a control-flow-graph that looks essentially the same as the switch-based interpreter!

Then, during code generation, LLVM performs “tail duplication,” and copies the branch back into each location, restoring the original intent. This dance is documented, at a high level, in an old LLVM blog post introducing the new implementation.

The LLVM 19 regression  🔗︎

The whole reason for the deduplicate-then-copy dance is that, for technical reasons, creating and manipulating the control-flow-graph containing many indirectbr instructions can be quite expensive.

In order to avoid catastrophic slowdowns (or memory usage) in certain cases, LLVM 19 implemented some limits on tail-duplication pass, causing it to bail out if duplication would blow up the size of the IR past certain limits.

Unfortunately, on CPython those limits resulted in Clang leaving all of the dispatch jumps merged, and entirely undoing the whole purpose of the computed goto-based implementation! This bug was first identified by another language implementation with a similar interpreter loop, but had not been known (as far as I can find) to affect CPython.

In addition to the performance impact, we can observe the bug directly by disassembling the resulting object code and counting the number of distinct indirect jumps:

$ objdump -S --disassemble=_PyEval_EvalFrameDefault ${clang18}/bin/python3.14 | 
  egrep -c 'jmps+*'
332

$ objdump -S --disassemble=_PyEval_EvalFrameDefault ${clang19}/bin/python3.14 | 
  egrep -c 'jmps+*'
3

Further weirdness  🔗︎

I am confident that the change to the tail-call duplication logic caused the regression: if you fix it, performance matches clang-18. However, I can’t fully explain the magnitud