For the past two weeks, I’ve been spending most of my time rewriting our CI scripts in GitHub Actions. This is the third time we’ve had to redo our CI setup—first GitHub Actions, then Earthly (which we moved away from because it was discontinued), and now, reluctantly, back to GitHub Actions.

Our CI is complex: merge queues, multiple runners (self-hosted, blacksmith.sh, GitHub-hosted), Rust builds, Docker images, and heavy integration tests. Every PR we merge burns through an hour of CI time, running across multiple parallel runners.

There are a few things we’d like to have (which we deem as “good software practice”) but it’s nothing unheard of:

1. Everything that goes into `main` must pass all tests.
2. Trivial mistakes (formatting, unused deps, lint issues) should be fixed automatically, not cause failures.
3. The artifacts we test with in CI should be the exact ones we release.
4. CI should complete quickly (to keep developers happy).

GitHub Actions technically allows all of this—but setting it up is a frustrating mess, full of hidden gotchas, inconsistent behavior, and a debugging experience that makes me question my choices.

Strange Way to Enforce Status Checks with Merge Queue

The key to enforcing a clean main branch is GitHub’s merge queue, which rebases a PR onto main before running CI. Sounds great. But here’s the fun part:

• We need CI to run before entering the queue to auto-fix trivial issues.
• We need CI to run again inside the queue to verify the final merge.
• GitHub Actions makes it weirdly hard to require both runs to pass.

The solution? Name the jobs identically in both phases. That’s it. GitHub treats them as the same check, so they both need to succeed. Solved by reading this answer in a Stack Overflow post after a few hours of debugging. Any other way you try to do this leads to either status checks being awaited before you put something in the queue (so it never starts the job) or worse, things just get merged even if the job you’d like to pass in the merge queue fails.

A security nightmare?

A few days ago, someone compromised a popular GitHub Action. The response? “Just pin your dependencies to a hash.” Except as comments also pointed out, almost no one does.

Even setting aside supply chain attacks, GitHub’s security model is a confusing maze to me: My point of view is that if I can’t understand a security model easily it’s probably doomed to fail or break at some point. Disclaimer: I’m writing this as a github actions user with only a vague understanding of it so I’d be delighted to hear that it is not just “things piled on top of things until it’s safe”, which is my current impression. I do understand very well that the problem of having secure CI for distributed source control is complicated.

In github, there is a “default” token called GITHUB_TOKEN. The way it works is that it gets initialized with some default permissions. You can set that default in the settings of your repository (under Actions -> General -> Workflow Permissions). Here is what the github documentation says about it:

If the default permissions for the GITHUB_TOKEN are restrictive, you may have to elevate the permissions to allow some actions and commands to run successfully. If the default permissions are permissive, you can edit the workflow file to remove some permissions from the GITHUB_TOKEN.

– Github Documenation

Removing permission that aren’t necessary sounds nice (though I do think a better “default” would be to start with no privileges and require the user to add whatever is needed). Unfortunately, there are many of them and it’s hardly clear for all of them what they are protecting if you’re not a github expert.

Your workflow permissions also don’t really depend on the action itself. Here is an example of such an instance, I’m using softprops/action-gh-release to