In 2022, I had an idea that could decrease the size of all newly-published npm packages by about 5%, and it was completely backwards compatible. This would have improved performance and reduced storage costs.

I eagerly pitched this idea to the npm maintainers, convinced it was a clear win. But after a few months, my proposal was rejected. To be clear: I think this was the right call!

Here’s what happened. I hope this story will be useful to others.

Technical background

Two things to know before diving in: how npm packages are distributed, and about the Zopfli compressor.

npm packages are just gzipped tarballs

First thing to know: npm packages are distributed as tar archives compressed with gzip. In other words, they’re just .tar.gz or .tgz files.

You can download these archives using npm pack $PACKAGE_NAME:

npm pack express && ls
# => express-4.21.2.tgz

If you extract this tarball, you’ll see all of the package’s files, such as package.json.

Zopfli, a gzip-compatible compressor

The second thing to know about: Zopfli.

Zopfli can create gzip-compatible data that’s smaller than other tools can. For example, compare the zopfli command to the gzip command:

gzip -9 romeo_and_juliet.txt
du -h romeo_and_juliet.txt.gz
# => 64K     romeo_and_juliet.txt.gz

zopfli romeo_and_juliet.txt
du -h romeo_and_juliet.txt.gz
# => 60K     romeo_and_juliet.txt.gz

As you can see, zopfli produces smaller files than gzip.

If Zopfli produces smaller sizes than gzip, why not use it everywhere? Unfortunately, Zopfli is great but it’s much slower than regular gzip.

time gzip -9 romeo_and_juliet.txt
# => real   0m0.016s

time zopfli romeo_and_juliet.txt
# => real   0m0.462s

In this simple test, Zopfli is about 28 times slower! That means it’s bad for content that changes a lot, but good for content that doesn’t.

In my opinion, Zopfli’s killer feature is that it creates files that are backwards compatible with existing decompressors. Other compression algorithms, like LZMA, can be better than gzip, but you can’t decompress their results with gunzip or equivalent. With Zopfli, you can!

cat my_file.txt
# => Hello world

lzma -c my_file.txt | gunzip -c
# => gunzip: unknown compression format

zopfli -c my_file.txt | gunzip -c
# => Hello world

So I wondered: if npm packages are compressed with gzip, could npm packages be compressed better with Zopfli? The answer is “yes”.

Proof of concept

To see if this would work, I tried it on one of my own npm packages.

I did something like this:

1. Get the file that is published by default. I used HumanizeDuration.js, one of my more popular packages, as a test.

   cd HumanizeDuration.js
   npm pack
   # ...
   # => humanize-duration-3.32.1.tgz
   

   This created humanize-duration-3.32.1.tgz, which was ~17 kibibytes large. I could npm publish this right now, but let’s try shrinking it.

2. Decompress (but don’t unarchive) the file.

   gunzip humanize-duration-3.32.1.tgz
   

   This leaves us with an uncompressed tarball, humanize-duration-3.32.1.tar.

3. Re-compress it with Zopfli.

   zopfli humanize-duration-3.32.1.tar
   

   As expected, this produced a slightly smaller file by over a kilobyte. That’s promising!

4. Make sure I could still install it. I installed the tarball, and tried to use it.

   cd "$(mktemp -d)"
   
   npm install /path/to/recompressed/humanize-duration-3.32.1.tar.gz
   # => added 1 package in 123ms
   
   node -p 'require("humanize-duration")(1234)'
   # => 1.234 seconds
   

   It worked!

This saved 1114 bytes for a ~6.2% reduction. And this is completely backwards-compatible. That made sense; this is what Zopfli is supposed