Reviewing pull requests for snaps has been pretty terrible ever since snaps were introduced. There are a few very simple reasons:
- Building snaps locally in order to review is too much work. This is a fine place to start, but it doesn’t scale, especially if it takes any amount of time to build the snap in question (mine takes over an hour).
- Most CI engines use Docker. You can build snaps in docker with hacks and tweaks that sort of work sometimes, but it’s not a supported approach, and it’ll just randomly break next week.
- Using private tokens to access the snap store from a fork will leak the token. You don’t want random people releasing snaps on your behalf. This concern is not limited to snaps, of course.
I dedicated an article to this back in 2019, introducing a solution that worked reasonably well for a number of years, but it had some downsides as well:
- It used Docker. See (2), above.
- It required your own infrastructure and a GitHub app, which was difficult to configure. No one wants to do that.
Ultimately, the GitHub Snap Builder fell apart for me personally because it used Docker, and Snapcraft broke my ability to build the snap I cared about (Nextcloud) in Docker. I needed a new solution, but I wasn’t in the mood to roll my own again. I bided my time, investigated, and experimented. Two years ago, I finally started down the path that ultimately led to success, and I’d like to share that with you.
GitHub Actions
For a long time, the Nextcloud snap used Circle CI as our CI system (Travis CI before that). It was the only CI system for years that could actually build snaps, since it supported a special runner type that was a virtual machine instead of being a Docker container. However, in 2021, they suddenly capped CI runtimes to one hour, with no way to override. We were forced to revisit the CI landscape, and were pleased to discover that GitHub Actions, introduced in the meantime, had all the capabilities we needed. We moved to that, and have been happy users ever since.
We were still in the lurch in terms of testing pull requests, though. We could build snaps in GitHub Actions, and run our tests against them, but without something like the now-defunct GitHub Snap Builder, we couldn’t upload them to the store. The Nextcloud snap is an open source project, and we receive pull requests from unknown contributors all the time. Using GitHub Actions to upload snaps to the store would leak our private tokens to those unknown contributors (see (3), above). Until 2020.
On December 15th, 2020, Jaroslav Lobacevski wrote a series of posts about GitHub Actions security. To be honest, I don’t actually know when the critical feature was added, but this series is what introduced me to it: the workflow_run
trigger.
Documentation has always been the Achilles heel of GitHub Actions. All they have are reference docs that are so light on detail I find myself wince-laughing half the time. I still can’t read the docs on workflow_run
and come away with any meaningful understanding, but Lobacevski’s blog posts convinced me it was what I needed. Let me explain by first explaining how the GitHub Snap Builder solved this problem.
GitHub Snap Builder
The GitHub Snap Builder obviously needs a token for it to be able to upload snaps to the snap store. It did this on pull requests from unknown contributors. How did it avoid leaking that token? By making a very clear delineation between trusted and untrusted code. It used two ephemeral containers:
- A container to build the snap. This was an unprivileged container with no tokens to leak. It was responsible for building the snap from the untrusted code in the pull request.
- A container to upload the snap. Once the snap was built by container 1, another container was fired up with the snap store token in its environment. It then used snapcraft to upload the snap to the store. Everything running in this container is trusted, coming from a known-good source. It’s uploading an untrusted snap, yes, but it’s not running anything out of that snap to do it. This process was hard-coded, and didn’t rely on (or run) anything in the pull request, so there’s nothing for a malicous pull request to compromise.
How does that relate to GitHub Actions?
The workflow_run
trigger allows for the same untrusted/trusted code separation that was possible in the GitHub Snap Builder. It supports a pull request workflow like this:
- Pull request is opened from fork
foo/nextcloud-snap
, branchbar
- Snap is built of the untrusted code in the pull request, and automatically tested
- That snap is saved as an artifact
Once that workflow finishes on that fork’s branch, it uses the workflow_run
trigger to fire up a new workflow on the master branch: trusted code. That branch has access to the snap store token, downloads the snap from the pull request workflow’s artifacts, and uploads it. Again, all without running untrusted code.
Show me how
Ultimately you will need two workflow files, one for each side of the trust boundary. In other words, you need one to run on pull requests and build snaps, and one to run on master and upload snaps. I’ll reduce this to the minimum you need to make this happen, but if you want to see a r