ℹ️ This is the second blog post discussing dynamic-derivations in Nix. Checkout the first part An early look at Nix Dynamic Derivations if you want a primer on the experimental feature.
I’m still in love with the experimental feature dynamic-derivations in Nix 🥰, but following my earlier post I had read comments from readers that the potential was still unclear.
This makes total sense. Nix is already quite a complex tool, ecosystem and language. The addition of something like dynamic-derivations muddles the capability to understand the potential it offers.
At the end of the last post, I echoed John Ericson’s (@ericson2314) call to action for others in the community to begin to tinker with the feature.
In the spirit of that request, I have put together a practical demonstration of what can be accomplished with dynamic-derivations in the tool MakeNix 💥🏃♂️🔥
Please checkout https://github.com/fzakaria/MakeNix and contribute any improvements, bug fixes or clarifications. The repository is meant to be an example for others to imitate. Contributions are always welcome.
Once again before we begin, if you want to play with it it’s important you use nix@d904921. Additionally, you need to enable experimental-features = ["nix-command" "dynamic-derivations" "ca-derivations" "recursive-nix"]. Here, there be dragons 🐲.
Here we have a rather simple C project that produces a binary that emits "Hello World"
> tree
├── Makefile
└── src
├── hello.c
├── hello.h
├── main.c
├── world.c
└── world.h
> make all
> ./main
Hello, World!
We could write a typical Nix derivation via mkDerivation that calls make and for this relatively small example it would be fine. However for larger projects, everytime we change a tiny bit of our code we must rebuild the whole thing from scratch. We don’t get to leverage all the prior object files that had been built.
That’s a bummer 🙁. Wouldn’t it be great if each object file (i.e. hello.o) was created in their own derivation?
We could do that ahead of time by writing a tool to create a bunch of tiny mkDerivation but everytime we change a dependency in our graph (i.e. add or remove a source file), we have to re-run the tool. That’s a bit of a bummer on the development loop.
If those generated Nix files were not committed to the repository and we wanted to add this package to nixpkgs, we’d need to also do a full nix build within the derivation itself via recursive-nix. 😨
Dynamic-derivations seeks to solve this callenge by having derivations create other derivations without having to execute a nix build recursively. Nix will realize the output of one derivation is another derivation and build it as well. 🤯
Let’s return to our C
/C++ project. GCC & Clang support an argument -MM which runs only the preprocessor and emits depfiles .d that contain Makefile targets with the dependency targets between files.
main.o: src/main.c src/hello.h src/world.h
The idea behind MakeNix is to generate these depfiles, parse them and create the necessary mkDerivation all at build time.
MakeNix includes a very simple Golang parser, parser.go (~70 lines of code), that parses the depfiles and generates the complete Nix expression.
Here is a sample of the Nix expression generated.
{ pkgs }:
let fs = pkgs.lib.fileset;
hello.o = pkgs.stdenvNoCC.mkDerivation {
name = "hello.o";
src = fs.toSource {
root = ./src;
fileset = fs.unions [
./src/hello.c
./src/hello.h
];
};
nativeBuildInputs = [ pkgs.gcc ];
buildPhase = ''
gcc -c hello.c -o hello.o
'';
installPhase = ''
cp hello.o $out
'';
};
main.o = ...;
