A year of funded FreeBSD development by cperciva
I’ve been maintaining
FreeBSD on the
Amazon EC2 platform ever since I
first got it booting in 2010, but in
November
2023 I added to my responsibilities the role of FreeBSD release
engineering lead — just in time to announce the availability of
FreeBSD 14.0, although Glen Barber did all the release engineering work for
that release. While I receive a small amount of funding from
Antithesis and from my
FreeBSD/EC2 Patreon, it
rapidly became clear that my release engineering duties were competing
with — in fact, out-competing — FreeBSD/EC2 for my available
FreeBSD volunteer hours: In addition to my long list of “features to
implement” stagnating, I had increasingly been saying “huh that’s weird…
oh well, no time to investigate that now”. In short, by early 2024 I was
becoming increasingly concerned that I was not in a position to be a good
“owner” of the FreeBSD/EC2 platform.
For several years leading up to this point I had been talking to Amazonians
on and off about the possibility of Amazon sponsoring my FreeBSD/EC2 work;
rather predictably, most of those conversation ended up with my
contacts at Amazon rhyming with “Amazon should definitely sponsor the work
you’re doing… but I don’t have any money available in my budget
for this”. Finally in April 2024 I found someone with a budget, and after
some discussions around timeline, scope, and process, it was determined
that Amazon would support me for a year via
GitHub Sponsors. I’m
not entirely sure if the year in question was June through May or July
through June — money had to move within Amazon, from Amazon to GitHub,
from GitHub to Stripe, and finally from Stripe into my bank account, so when
I received money doesn’t necessarily reflect when Amazon intended to
give me money — but either way the sponsorship either has come to an end
or is coming to an end soon, so I figured now was a good time to write about
what I’ve done.
Amazon was nominally sponsoring me for 40 hours/month of work on FreeBSD
release engineering and FreeBSD/EC2 development — I made it clear to
them that sponsoring one and not the other wasn’t really feasible, especially
given dependencies between the two — and asked me to track how much
time I was spending on things. In the end, I spent roughly 50 hours/month
on this work, averaging 20 hours/month spent on EC2-specific issues, 20
hours/month making FreeBSD releases happen, and 10 hours/month on other
release engineering related work — although the exact breakdown
varied dramatically from month to month.
Following FreeBSD’s
quarterly
release schedule (which I announced in July 2024, but put together and
presented at the FreeBSD developer summit at
BSDCan in May 2024), I managed four
FreeBSD releases during the past year: FreeBSD 13.4, in September 2024;
FreeBSD 14.2, in December 2024; FreeBSD 13.5, in March 2025; and FreeBSD
14.3, currently scheduled for release on June 10th. The work involved in
manging each of these releases — nagging developers to get their
code into the tree in time, approving (or disapproving!) merge requests,
coordinating with other teams, building and testing images (usually three
Betas, one Release Candidate, and the final Release), writing announcement
text, and fixing any release-building breakage which arose along the way
— mostly happened in the month prior to the release (I refer to the
second month of each calendar quarter as “Beta Month”) and ranged from a
low of 33.5 hours (for FreeBSD 13.5) to a high of 79 hours (for FreeBSD
14.2). As one might imagine, the later in a stable branch you get, the
fewer the number of things there are breaking and the lower the amount of
work required for doing a release; while I wasn’t tracking hours when I
managed FreeBSD 14.1, I suspect it took close to 100 hours of release
engineering time, and FreeBSD 15.0 is very likely to be well over that.
On the FreeBSD/EC2 side of things, there were two major features which
Amazon encouraged me to prioritize: The “power driver” for AWS Graviton
instances (aka “how the EC2 API tells the OS to shut down” — without
this, FreeBSD ignores the shutdown signal and a few minutes later EC2
times out and yanks the virtual power cable), and device hotplug on AWS
Graviton instances. The first of these was straightforward: On Graviton
systems, the “power button” is a GPIO pin, the details of which are
specified via an ACPI _AEI object. I added code to find those in ACPI and
pass the appropriate configuration through to the driver for the PL061 GPIO
controller; when the GPIO pin is asserted, the controller generates an
interrupt which causes the ACPI “power button” event to be triggered, which
in turn now shuts down the system. There was one minor hiccup: The ACPI
tables provided by EC2 specify that the GPIO pin in question should be
configured as a “Pull Up” pin, but the PL061 controller in fact doesn’t
have any pullup/pulldown resistors; this didn’t cause problems on Linux
because Linux silently ignores GPIO configuration failures, but on FreeBSD
we disabled the device after failing to configure it. I believe this EC2
bug will be fixed in future Graviton systems; but in the mean time I ship
FreeBSD/EC2 AMIs with a new “quirk”: ACPI_Q_AEI_NOPULL, aka
“Ignore the PullUp flag on GPIO pin specifications in _AEI objects”.
Getting hotp
5 Comments
tiffanyh
Lots of respect for cperciva.
Don’t know how he manages all of this + Tarsnap.
ksec
I was rather hoping Amazon would spend and contribute more. But it seems they basically only want to pay for the minimum FreeBSD support.
Amazon isn't even on FreeBSD sponsors [1]. And Google only sponsored $9K last year. Apple isn't there. Edit: And Credit to Microsoft being at least on the list! And forgot to mention Meta / Facebook missing from it as well.
I would have expect them to sponsor FreeBSD and OpenBSD annually by default given they use and continue to benefits the work out of both.
[1] https://freebsdfoundation.org/our-donors/donors/?donationYea…
AndyKelley
Sweet! By the way we just added FreeBSD to the download page on ziglang.org (as of today), so FreeBSD users can grab master branch builds automatically built by the CI.
It's also now a first-class supported cross-compilation target, including when linking libc, so you can do stuff like `zig cc -o hello hello.c -target riscv64-freebsd`.
And then of course if you have any C/C++ dependencies, you can fetch and build them with the zig build system, so it should be possible to easily cross-compile even quite complex projects for FreeBSD now.
Hopefully that helps more projects decide to add FreeBSD support and respective testing to their CI!
msdrigg
There are some hilarious tidbits in here
> Starting in the first week of 2024, the FreeBSD boot process suddenly got about 3x slower. I started bisecting commits, and tracked it down to… a commit which increased the root disk size from 5 GB to 6 GB. Why? Well, I reached out to some of my friends at Amazon, and it turned out that the answer was somewhere between "magic" and "you really don't want to know"; but the important part for me was that increasing the root disk size to 8 GB restored performance to earlier levels.
net01
There is also a lot of work on the laptop front, I read that the BSD foundation invested $750k for this
implementing: (S0ix Sleep State, etc )
you can find the project laptop here https://github.com/FreeBSDFoundation/proj-laptop