I’m Ben Johnson. I wrote BoltDB, an embedded database that is the backend for systems like etcd. Now I work at Fly.io, on Litestream. Litestream is an open-source project that makes SQLite tenable for full-stack applications through the power of ✨replication✨. If you can set up a SQLite database, you can get Litestream working in less than 10 minutes.

The conventional wisdom of full-stack applications is the n-tier architecture, which is now so common that it’s easy to forget it even has a name. It’s what you’re doing when you run an “application server” like Rails, Django, or Remix alongside a “database server” like Postgres. According to the conventional wisdom, SQLite has a place in this architecture: as a place to run unit tests.

The conventional wisdom could use some updating. I think that for many applications – production applications, with large numbers of users and high availability requirements – SQLite has a better place, in the center of the stack, as the core of your data and persistence layer.

It’s a big claim. It may not hold for your application. But you should consider it, and I’m here to tell you why.

A Brief History of Application Databases

50 years is not a long time. In that time, we’ve seen a staggering amount of change in how our software manages data.

In the beginning of our story, back in the ’70s, there were Codd’s rules, defining what we now call “relational databases“, also known today as “databases”. You know them, even if you don’t: all data lives in tables; tables have columns, and rows are addressable with keys; C.R.U.D.; schemas; a textual language to convey these concepts. The language, of course, is SQL, which prompted a Cambrian explosion of SQL databases, from Oracle to DB2 to Postgres to MySQL, throughout the ’80s and ’90s.

It hasn’t all been good. The 2000s got us XML databases. But our industry atoned by building some great columnar databases during the same time. By the 2010s, we saw dozens of large-scale, open-source distributed database projects come to market. Now anyone can spin up a cluster and query terabytes of data.

As databases evolved, so too did the strategies we use to plug them in to our applications. Almost since Codd, we’ve divided those apps into tiers. First came the database tier. Later, with memcached and Redis, we got the caching tier. We’ve got background job tiers and we’ve got routing tiers and distribution tiers. The tutorials pretend that there are 3 tiers, but we all know it’s called “n-tier” because nobody can predict how many tiers we’re going to end up with.

You know where we’re going with this. Our scientists were so preoccupied with whether or not they could, and so on.

See, over these same five decades, we’ve also seen CPUs, memory, & disks become hundreds of times faster and cheaper. A term that practically defines database innovation in the 2010s is “big data”. But hardware improvements have made that concept slippery in the 2020s. Managing a 1 GB database in 1996? A big deal. In 2022? Run it on your laptop, or a t3.micro.

When we think about new database architectures, we’re hypnotized by scaling limits. If it can’t handle petabytes, or at least terabytes, it’s not in the conversation. But most applications will never see a terabyte of data, even if they’re successful. We’re using jackhammers to drive finish nails.

The Sweet Release of SQLite

There’s a database that bucks a lot of these trends. It’s one of the most popular SQL databases in the world, so standardized it’s an official archival format of the Library of Congress, it’s renowned for its reliability and its unfathomably encompassing test suite, and its performance is so good that citing its metrics on a message board invariably starts an argument about whether it should be disqualified. I probably don’t have to name it for you, but, for the one person in the back with their hand raised, I’m talking about SQLite.

SQLite is an embedded database. It doesn’t live in a conventional architectural tier; it’s just a library, linked into your application server’s process. It’s the standard bearer of the “single process application“: the server that runs on its own, without relying on nine other sidecar servers to function.

I got interested in these kinds of applications because I build databases. I wrote BoltDB, which is a popular embedded K/V store in the Go ecosystem. BoltDB is reliable and, as you’d expect from an in-process database, it performs like a nitro-burning funny car. But BoltDB has limitations: