Recently, a pair of critical vulnerabilities were reported in the OpenSSL project. Surprising absolutely nobody, the root cause of both vulnerabilities turned out to be a buffer overrun, which could be triggered by an attacker with a malicious payload to cause a crash and denial of service. Predictably, many Rust advocates (of which I am one) pointed out that this is exactly the kind of vulnerability that can be statically prevented by Rust, and is a clear example of where “Rewrite it in Rust” can have real benefits.

Setting aside the feasibility or advisability of a ground-up rewrite of a project as large, complex, and long-lived as OpenSSL, it’s worth talking about exactly how Rust is able to prevent these kinds of buffer overflow vulnerabilities. Specifically, since Rust isn’t a fully dependently typed language where we can prove the lengths of our buffers at compile time, it resorts to runtime bounds checks to ensure that indexing always remains safe. In practice, this means that every time you index into a slice, the Rust compiler will emit a sequence of instructions that checks if your index is within the bounds of that slice, and panics if it isn’t.

There are unsafe ways of opting out of bounds checking when indexing (slice::get_unchecked and slice::get_unchecked_mut). These are definitely used at least some of the time, but not all the time – and not even when it could be statically determined that the index is in-bounds. This is a good thing for safety, because these are exactly the kinds of edge case properties that are easy to accidentally break in a way that automated tests don’t cover.

To C programmers, this can sound like a downside of Rust – there are many places in real-world code where you do in fact statically know that your index is in-bounds, or places where you do multiple indexing operations into the same slice such that a single bounds check on the maximum index could cover all indexing operations. Overall, it’s probably not a Zero Cost Abstraction, at least under Bjarne Stroustrup’s original definition:

What you do use, you couldn’t hand code any better.

What is the actual cost of all this extra bounds checking, though? There’s a little bit of prior art here – for example this repository, which contains benchmarks which index into slices in tight loops using both the safe bounds-checked version and the unsafe, non-bounds-checked version. The conclusion there was that at least in this scenario bounds checking is usually only a cost when its presence prevents optimizations such as autovectorization (which can be a huge optimization in practice), but otherwise it’s basically a wash. On the other hand, I wasn’t able to find an extensive analysis of the cost of pervasive bounds checking on a real, large, production Rust codebase with high performance sensitivity. I happen to work on one of those, so I figured it might be interesting to take a look at the cost of bounds checks in the hot path.

For this experiment, I’m considering “the hot path” to be a hot (cached in memory) read of a single query, including both the cost of the cache read itself and the SQL protocol translation layer.

How often do bounds checks happen?

Before taking a look at the runtime cost of the bounds checks, I wanted to get an idea of just how frequently we’re doing these runtime bounds checks in the hot path. After stumbling around trying to figure out how to instrument this using either perf or dtrace, I found this StackOverflow question, explaining how to record the number of times a particular line of code is executed using gdb: just set a breakpoint on the line, ignore the next million or so times the breakpoint is hit, and then run the program. Then, info breakpoints will show the number of times the breakpoint was hit!

Using that technique, it’s quite easy to see how many times we’re doing bounds checks – I can just load up the release readyset-mysql binary in rust-gdb:

rust-gdb --args target/release/readyset-mysql 
   --standalone 
   -u root -p password 
   -a 0.0.0.0:3307 
   --deployment test 
   --upstream-db-url mysql://root:password@127.1

then run the binary so that I can get everything set up before starting my instrumentation:

(gdb) run
MySQL [test]> create table t (id int, name text);
 
MySQL [test]> insert into t values (1, 'a');
 
MySQL [test]> select * from t where id = 1;
+------+------+
| id   | name |
+------+------+
|    1 | a    |
+------+------+

Thanks to ReadySet’s query caching, once we’ve materialized the id = 1 key, all subsequent reads of that key will be served from an in-memory hash map, so we should only be measuring the “hot path” code that we’re concerned about here. We can drop back into the GDB shell by sending the readyset-mysql binary a Ctrl+C to allow us to set up our breakpoints. We want to measure every time a slice is indexed (either i