I’ve been maintaining
FreeBSD on the
Amazon EC2 platform ever since I
first got it booting in 2010, but in
November
2023 I added to my responsibilities the role of FreeBSD release
engineering lead — just in time to announce the availability of
FreeBSD 14.0, although Glen Barber did all the release engineering work for
that release. While I receive a small amount of funding from
Antithesis and from my
FreeBSD/EC2 Patreon, it
rapidly became clear that my release engineering duties were competing
with — in fact, out-competing — FreeBSD/EC2 for my available
FreeBSD volunteer hours: In addition to my long list of “features to
implement” stagnating, I had increasingly been saying “huh that’s weird…
oh well, no time to investigate that now”. In short, by early 2024 I was
becoming increasingly concerned that I was not in a position to be a good
“owner” of the FreeBSD/EC2 platform.

For several years leading up to this point I had been talking to Amazonians
on and off about the possibility of Amazon sponsoring my FreeBSD/EC2 work;
rather predictably, most of those conversation ended up with my
contacts at Amazon rhyming with “Amazon should definitely sponsor the work
you’re doing… but I don’t have any money available in my budget
for this”. Finally in April 2024 I found someone with a budget, and after
some discussions around timeline, scope, and process, it was determined
that Amazon would support me for a year via
GitHub Sponsors. I’m
not entirely sure if the year in question was June through May or July
through June — money had to move within Amazon, from Amazon to GitHub,
from GitHub to Stripe, and finally from Stripe into my bank account, so when
I received money doesn’t necessarily reflect when Amazon intended to
give me money — but either way the sponsorship either has come to an end
or is coming to an end soon, so I figured now was a good time to write about
what I’ve done.

Amazon was nominally sponsoring me for 40 hours/month of work on FreeBSD
release engineering and FreeBSD/EC2 development — I made it clear to
them that sponsoring one and not the other wasn’t really feasible, especially
given dependencies between the two — and asked me to track how much
time I was spending on things. In the end, I spent roughly 50 hours/month
on this work, averaging 20 hours/month spent on EC2-specific issues, 20
hours/month making FreeBSD releases happen, and 10 hours/month on other
release engineering related work — although the exact breakdown
varied dramatically from month to month.

Following FreeBSD’s
quarterly
release schedule (which I announced in July 2024, but put together and
presented at the FreeBSD developer summit at
BSDCan in May 2024), I managed four
FreeBSD releases during the past year: FreeBSD 13.4, in September 2024;
FreeBSD 14.2, in December 2024; FreeBSD 13.5, in March 2025; and FreeBSD
14.3, currently scheduled for release on June 10th. The work involved in
manging each of these releases — nagging developers to get their
code into the tree in time, approving (or disapproving!) merge requests,
coordinating with other teams, building and testing images (usually three
Betas, one Release Candidate, and the final Release), writing announcement
text, and fixing any release-building breakage which arose along the way
— mostly happened in the month prior to the release (I refer to the
second month of each calendar quarter as “Beta Month”) and ranged from a
low of 33.5 hours (for FreeBSD 13.5) to a high of 79 hours (for FreeBSD
14.2). As one might imagine, the later in a stable branch you get, the
fewer the number of things there are breaking and the lower the amount of
work required for doing a release; while I wasn’t tracking hours when I
managed FreeBSD 14.1, I suspect it took close to 100 hours of release
engineering time, and FreeBSD 15.0 is very likely to be well over that.

On the FreeBSD/EC2 side of things, there were two major features which
Amazon encouraged me to prioritize: The “power driver” for AWS Graviton
instances (aka “how the EC2 API tells the OS to shut down” — without
this, FreeBSD ignores the shutdown signal and a few minutes later EC2
times out and yanks the virtual power cable), and device hotplug on AWS
Graviton instances. The first of these was straightforward: On Graviton
systems, the “power button” is a GPIO pin, the details of which are
specified via an ACPI _AEI object. I added code to find those in ACPI and
pass the appropriate configuration through to the driver for the PL061 GPIO
controller; when the GPIO pin is asserted, the controller generates an
interrupt which causes the ACPI “power button” event to be triggered, which
in turn now shuts down the system. There was one minor hiccup: The ACPI
tables provided by EC2 specify that the GPIO pin in question should be
configured as a “Pull Up” pin, but the PL061 controller in fact doesn’t
have any pullup/pulldown resistors; this didn’t cause problems on Linux
because Linux silently ignores GPIO configuration failures, but on FreeBSD
we disabled the device after failing to configure it. I believe this EC2
bug will be fixed in future Graviton systems; but in the mean time I ship
FreeBSD/EC2 AMIs with a new “quirk”: ACPI_Q_AEI_NOPULL, aka
“Ignore the PullUp flag on GPIO pin specifications in _AEI objects”.

Getting hotp