From d192288b750358231fc539b1334c0d2d029a21d9 Mon Sep 17 00:00:00 2001 From: Jade Lovelace Date: Sat, 23 Dec 2023 13:31:57 -0800 Subject: [PATCH] flakey flakey --- content/posts/flakes-arent-real.md | 664 +++++++++++++++++++++++++++++ 1 file changed, 664 insertions(+) create mode 100644 content/posts/flakes-arent-real.md diff --git a/content/posts/flakes-arent-real.md b/content/posts/flakes-arent-real.md new file mode 100644 index 0000000..c137854 --- /dev/null +++ b/content/posts/flakes-arent-real.md @@ -0,0 +1,664 @@ ++++ +date = "2023-11-25" +draft = true +path = "/blog/flakes-arent-real" +tags = ["nix"] +title = "Flakes aren't real and cannot hurt you: a guide to using Nix flakes the non-flake way" ++++ + +Inflammatory title out of the way, let's go. + +I think that Nix flakes have some considerable benefits, such as: + +* Convenient pinning of evaluation-time dependencies +* Eliminating pointless rebuilds of code by only including tracked files in + builds +* Making Nix code, on average, much more reproducible by pervasive pinning +* Allegedly caching evaluation +* Possibly making Nix easier to learn by reducing the amount of poking at + strange attribute sets and general `NIX_PATH` brokenness + +However, at the same time, there are a few things that one might be led to +think about flakes that are not the most effective way of doing things. I +personally use flakes relatively extensively in my own work, but there are +several ways I use them that are not standard, with reason. + +Flakes are *optional*, and as much as some people whose salary depends on it +might say otherwise, they are not the (only) future of Nix: they are simply a +special entry point for Nix code with a built in pinning system, nothing more, +nothing less. + +Nix continues to gather a reputation for bad documentation, in part because the +official documentation for nixpkgs and NixOS is *de facto* not allowed to talk +about flakes, as a policy. This situation is certainly partially due to a +divide between Nix developers and nixpkgs developers, which are groups with +surprisingly little overlap. + +Flakes also are a symptom or cause of much intra-community strife between +"pro-flakes" and "anti-flakes" factions, but this situation is at some level a +sign of broken consensus processes and various actors trying to sidestep them, +an assumption by many people that the docs are "outdated" for not using flakes, +and the bizarre proliferation of flakes everywhere in blog posts or tutorials +leading to a belief that they are required for everything. + +This post is about how to architect Nix projects in general, with a special eye +on how to do so with flakes while avoiding their limitations. It tries to +dispel misconceptions that can develop in such a monoculture. + +# "Flakes are the composition primitive in Nix" + +The Nix language, functions, and nixpkgs utilities are an effective composition +primitive, and are much better suited for putting parts of a project together, +especially if it is in a monorepo. + +The most flexible way of building large systems with Nix is to merely use +flakes as an entry point, and develop the rest using "old" tools. This is for +multitudinous reasons: + +- Flakes couple version control integration, dependency management and lockfile + management. In medium sized projects, even at the scale of my dotfiles, + locking dependencies of subprojects is often highly undesirable. + + They're not ideal for either working in the large or in the small: in the + small, there is too much overhead in writing a separate `flake.nix` for some + tiny utility, and in the large, for example, in nixpkgs, if flakes were + actually used for dependency management, `flake.nix` would be 100,000 lines + of `inputs` long. +- In terms of making flexible builds, flakes don't support configuration + [except through hilarious abuses of + `--override-input`][boolean-option]. This means that all build configuration + variants have to be anticipated ahead of time, or that traditional + nixpkgs/Nix language primitives need to be used instead. +- Flakes as a composition primitive is completely incompatible with cross + compilation. Due to the lack of configuration support, `packages.${system}` + cannot be used for cross compilation: there is nowhere to specify the + architecture to build with. + +[boolean-option]: https://github.com/boolean-option + +Because of all of this, even in a flakes world, to compose software in the +large *and* in the small reusably and efficiently, the other composition +primitives provided by Nix and nixpkgs remain the best choices to assemble +software. A flake can then be relegated to merely an entry point and a way of +acquiring dependencies that are required for evaluation (build-time +dependencies should use `pkgs.fetchurl`, `fetchFromGitHub`, etc. + +For example, to expose multiple configurations of a program, one might write it +the traditional way, using a lambda accepting some configuration parameters, +then call that lambda multiple times to expose multiple output attributes +inside the flake itself. This separates the capability to configure the +software from the actual defined configurations of the software, and avoids +letting the configuration non-system of flakes define how the internals of the +build definition work. + +One of the largest simultaneous advantages and disadvantages of the Nix +language is that it is a Turing complete language, which causes pain to static +analysis, but is also one of its largest assets: you can program it. This can +be seen as a problem, but it also is awesome: you can programmatically patch +packages, define configuration dynamically, read files of arbitrary formats and +more. + +Nix is a functional programming language, which means that its fundamental +composition primitive is the function. Even "fancy" objects like NixOS modules +or overlays are just functions that can be moved into separate files, imported, +or created through partial application of other functions (although, since +`imports` in modules are deduplicated by file name, NixOS modules generally +should be imported by path instead of generated by functions). + +See the next section for concrete ways of composing software together. + +# "Flakes are where you put your Nix code" + +Flakes are merely a fancy schema for making a standardized entry point into Nix +code. Most of the Nix code in a project of any significant size should not be +in `flake.nix`, for several reasons. + +The most trivial reason to put as little code as possible in `flake.nix` is +maintainability: there is as much rightward drift in `flake.nix` as in recent +German and Dutch elections (concerningly much!), so from just that perspective, +it's useful to move things out of it. + +Let's talk about some standard patterns that have existed before flakes did, +which still are relevant in a flakes world. + +## [`package.nix`][package-nix] + +[package-nix]: https://github.com/nixos/nixpkgs/blob/41acc25766fbc611cd10cb043bc7cab91d2fd088/pkgs/by-name/README.md + +I am using `package.nix` to refer to the standard way for writing packages in +nixpkgs style, which are invoked with `callPackage`. This is as opposed to +writing something directly in `flake.nix` using `pkgs`. + +A `package.nix` file looks something like so: + +```nix +{ # receives(*) pkgs.hello and pkgs.stdenv + hello, stdenv, + # can be overridden with `yourPackage.override { enableSomething = true; }` + enableSomething ? false +}: +finalAttrs: # optional finalAttrs to refer to the set below; preferred over using `rec` attr sets +stdenv.mkDerivation { + # ... +} +``` + +Package definitions should be written with `callPackage` if possible, rather +than inline in `flake.nix`, since using `package.nix` makes them into small, +composable, configurable, and portable units of software. Also, by using +`callPackage` and writing in nixpkgs style, it becomes a lot easier to move +packages between projects, and indeed to upstream them to nixpkgs, since they +look and work a familiar way. + +### Cross compilation + +A lesser-known fact is that `callPackage` is load-bearing for cross +compilation. If you write `pkgs.foo` in `nativeBuildInputs`, such a Nix +expression will break under cross compilation, but `foo` as an argument from +`callPackage` will not. This is because `callPackage` will magically resolve +`foo` appearing inside `nativeBuildInputs` to mean `pkgs.buildPackages.foo`; +that is, a package built for the build computer. + +`callPackage` evaluates a Nix file multiple times with different +arguments and splices the results together such that `buildInputs` magically +receives target packages, and `nativeBuildInputs` receives build packages, +even if the same package name appears in both. Magic ✨ + +That is, in the following intentionally-flawed-for-other-reasons `flake.nix`: + +```nix +{...}: { + outputs = { nixpkgs, ... }: + let pkgs = nixpkgs.legacyPackages.x86_64-linux; + in { + packages.x86_64-linux.x = pkgs.callPackage ./package.nix { }; + }; +} +``` + +then `package.nix`: + +```nix +{ stdenv, hello, openssl }: +stdenv.mkDerivation { + # ... + # things used in the build go in here + nativeBuildInputs = [ hello ]; + # libraries used by the resulting program go in here + buildInputs = [ openssl ]; +} +``` + +Incidentally, notice anything there? Yeah, it's flakes completely not +supporting cross compilation. See the next point. :D + +It's possible to use the `pkgs.buildPackages` attribute to pull things into +`nativeBuildInputs`, and `pkgs` for `buildInputs` but it is not conventional +to do so, and is quite verbose. + +[See the manual about these callPackage shenanigans][callPackage-intricacy] +for more details. See also: [the manual about dependency categories][cats]. + +## [Overlays] + +[Overlays]: https://nixos.org/manual/nixpkgs/stable/#sec-overlays-definition + +[An overlay][Overlays] is a function overriding nixpkgs which is evaluated +until it reaches a [fixed point]. An overlay takes two arguments, `final` and +`prev` (sometimes also called `self` and `super`), and returns an attribute set +that is shallowly replaced on top of nixpkgs with `//`. + +It is useful as a means for distributing sets of software outside of nixpkgs, +and still is useful in that role in a flakes world, since overlays are simple +functions that can be evaluated against any version of nixpkgs, allowing for +cross compilation to work properly. One may notice that the `overlays` flake +output is not architecture specific, which follows from their definition as +functions that take package sets and return modifications to make. + +Evaluation to a fixed point means that it is evaluated as many times as necessary +until it stops referring to the `final` argument (or overflows the stack). This +idea appears in many places, including tables of contents in LaTeX, [Typst] or +other typesetting programs: by generating the table of contents, you may affect +the layout of subsequent pages and change their page numbers, but after the +first run of that, the layout will probably not change, since the only change +is the numbers, so *that* iteration likely converges to the final result. + +`final` gives the *final* version of the attribute set, after overlays have +been evaluated as far as they will go; your overlay may be run multiple times +in evaluating an attribute in `final`, or even cause infinite recursion. `prev` +gives the version of nixpkgs prior to the present overlay or any further +overlays. + +For example, we could write an overlay to override GNU Hello to be a wrapper +that makes a [reference to an excellent retrocomputing series][hellorld]. +Content of `overlay.nix`: + +[hellorld]: https://www.youtube.com/watch?v=gQ6mwbTGXGQ + +```nix +final: prev: { + hello = final.writeShellScriptBin "hello" '' + ${prev.hello}/bin/hello -g "hellorld" "$@" + ''; +} +``` + +Then: + +``` +» nix run --impure --expr '(import { overlays = [ (import ./overlay.nix) ]; }).hello' +hellorld +``` + +Here, the attribute `hello` of our modified `nixpkgs` now is our script that +calls the original `hello` to say "hellorld". + +It's pretty easy to accidentally cause infinite recursion with overlays if +their laziness isn't correct. For example, attribute sets' attribute names are +evaluated strictly, with all names in an attribute set evaluated immediately, +but the values of attributes are lazily evaluated. [There have been attempts to +change this][lazy-attrs] but they were canned for performance reasons. Strict +attribute names can be a foot-gun, causing confusing infinite recursion in some +cases using `mapAttrs` or similar mechanisms on `prev` to generate the set of +things to override. + +Infinite recursion is not typically a problem if an overlay doesn't actually +replace anything or contain self-references, as may be the case for overlays +distributing very simple software, and we can take advantage of that as shown +in the next section. + +[Typst]: https://typst.app + +[fixed point]: https://en.wikipedia.org/wiki/Fixed-point_combinator +[lazy-attrs]: https://github.com/NixOS/nix/issues/4090 + +### The place of overlays in a flakes world + +*Flakes don't support cross compilation.* + +I am being a little bit tricky with the wording here. Flakes don't *stop* you +from doing cross compilation, but you have to do an end-run around flakes and +do it the "old" way. + +Because of this design fault in flakes, namely, the lack of support for +parameters, the most compatible way of writing packaging in a flake project +is to write the package definitions into an overlay first, then expose the +packages from the overlay. Consumers that need cross compilation can use the +overlay, and consumers that don't care can use it through `packages`. + +Keeping in mind ["1000 instances of nixpkgs"][1000-nixpkgs], a reasonable way +of writing a flake that *doesn't modify anything in nixpkgs*, just adds stuff +is: + +```nix +{ + inputs.nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable"; + inputs.flake-utils.url = "github:numtide/flake-utils"; + outputs = { self, nixpkgs, flake-utils, ... }: + let + out = system: + let + pkgs = nixpkgs.legacyPackages.${system}; + appliedOverlay = self.overlays.default pkgs pkgs; + in + { + packages.myPackage = appliedOverlay.myPackage; + }; + in + flake-utils.lib.eachDefaultSystem out // { + overlays.default = final: prev: { + myPackage = final.callPackage ./package.nix { }; + }; + }; +} +``` + +Downstream consumers that need to cross compile in spite of flakes can use +the overlay, and other consumers can use `packages` as normal. + +This uses a cute trick of calling the overlay, which is just a function, with +both `final` and `prev` as the final nixpkgs attribute set. This definitely +does not work on all overlays, since overlays can make self-references using +`final`, and indeed often need to do so, if they contain multiple derivations +that depend on each other. + +However, with a little bit more work, this can be overcome very cleanly, while +also avoiding any possibility of name shadowing problems! + + + +If you do have to use a real overlay that needs to replace things, import +nixpkgs again from your flake with the overlay as an argument. It's fine. It's +just a second of gratuitous evaluation time: + +```nix +let pkgs = import nixpkgs { inherit system; overlays = [ self.overlays.default ]; }; +in # .... +``` + +[1000-nixpkgs]: https://discourse.nixos.org/t/1000-instances-of-nixpkgs/17347/ + +[callPackage-intricacy]: https://nixos.org/manual/nixpkgs/stable/#ssec-cross-dependency-implementation +[cats]: https://nixos.org/manual/nixpkgs/stable/#ssec-stdenv-dependencies + +### NixOS modules + +NixOS modules are, like overlays and `package.nix`, fundamentally just +functions which are invoked in a fancy way, and are not a flakes construct. + +As used in flakes with the `nixosModules.*` output, they are +*architecture independent* since they are just functions, and if defining a +module for software that is built by the same flake, one would generally want to +use an overlay in [`nixpkgs.overlays`][nixpkgs.overlays] or the trick above, +invoking the overlay with `pkgs` twice, to actually bring it in (again, to +remain cross compilation compatible). + +To keep with the theme of putting things outside of `flake.nix` to enable +reusability, the code for the module can be placed in a separate file that is +imported. Then, `flake.nix` is used to import that module and inject +dependencies from its environment. + +[nixpkgs.overlays]: https://nixos.org/manual/nixos/stable/options#opt-nixpkgs.overlays + +#### Injecting dependencies + +There are a couple of ways to inject dependencies into NixOS modules from a +flake, one of which is mildly uglier. Injecting values from `flake.nix` into +NixOS is required for a couple of reasons, most notably, to use flakes-managed +dependencies inside NixOS configurations. It is also necessary to [properly +configure `NIX_PATH` so `` resolves in a flake +configuration][nixpath], since you need the actual inputs from `flake.nix` to +get a proper reference to nixpkgs suitable to create a dependency on the actual +flake input. + +[nixpath]: https://github.com/NixOS/nixpkgs/pull/254405 + +The simplest (and most reasonable, in my view) way to inject dependencies from +a flake is to write an inline module that has them in its lexical closure inside of +`flake.nix`. If you want to be fancy, you could even make an option to store +the injected dependencies: + +```nix +let depInject = { pkgs, lib, ... }: { + options.dep-inject = lib.mkOption { + type = with lib.types; attrsOf unspecified; + default = { }; + }; + config.dep-inject = { + # inputs comes from the outer environment of flake.nix + flake-inputs = inputs; + }; +}; +in { + nixosModules.default = { pkgs, lib, ... }: { + imports = [ depInject ]; + }; +} +``` + +The uglier and perhaps more well-known way to inject dependencies into NixOS +modules from flakes is [`specialArgs`][specialArgs]. This is uglier, since it gets dumped into +the arguments for every module, which is unlike how every other bit of data +flow works in NixOS, and it also doesn't work outside of the flake that's +actually invoking `nixpkgs.lib.nixosSystem`. The latter is the much more +sinister part, and the reason I would strongly recommend inline modules with +closures instead of `specialArgs`: they break flake composition. + +To use `specialArgs`, an attribute set is passed into `nixpkgs.lib.nixosSystem`, +which then land in the arguments of NixOS modules: + +```nix +# ... +nixosConfigurations.something = nixpkgs.lib.nixosSystem { + system = "x86_64-linux"; + specialArgs = { + myPkgs = nixpkgs; + }; + modules = { + { pkgs, lib, myPkgs }: { + # do something with myPkgs + } + }; +} +``` + +[specialArgs]: https://nixos.org/manual/nixos/unstable/options#opt-_module.args + +#### Example + +For example, this defines a very practical NixOS module that meows at the user +on the console on boot: + +```nix +{ + inputs.nixpkgs.url = "github:nixos/nixpkgs/nixos-unstable"; + outputs = { self, nixpkgs, ... }: { + overlays.default = final: prev: { + meow = final.writeShellScriptBin "meow" '' + echo meow + ''; + }; + + nixosModules.default = { pkgs, config, lib, ... }: { + imports = [ ./nixos-module.nix ]; + # inject dependencies from flake.nix, and don't do anything else + config = lib.mkIf config.services.meow.enable { + nixpkgs.overlays = [ self.overlays.default ]; + services.meow.package = lib.mkDefault pkgs.meow; + }; + }; + + }; +} +``` + +and `nixos-module.nix` containing the actual code: + +```nix +{ pkgs, config, lib, ... }: +let cfg = config.services.meow; in { + options = { + services.meow = { + enable = lib.mkEnableOption "meow"; + package = lib.mkOption { + description = "meow package to use"; + type = lib.types.package; + }; + }; + }; + config = lib.mkIf cfg.enable { + systemd.services.meow = { + description = "meow at the user on the console"; + serviceConfig = { + Type = "oneshot"; + ExecStart = "${cfg.package}/bin/meow"; + StandardOutput = "journal+console"; + }; + wantedBy = [ "multi-user.target" ]; + }; + }; +} +``` + +{% codesample(desc="How I tested the above") %} + +I put this into `flake.nix`: + +```nix +nixosConfigurations.test = nixpkgs.lib.nixosSystem { + system = "x86_64-linux"; + modules = [ + self.nixosModules.default + ({ pkgs, lib, ... }: { + fileSystems."/" = { + device = "/dev/sda1"; + }; + boot.loader.grub.enable = false; + boot.initrd.enable = false; + boot.kernel.enable = false; + documentation.enable = false; + environment.noXlibs = true; + + services.meow.enable = true; + + system.stateVersion = "23.05"; + }) + ]; +}; +``` + +Then I built the system configuration: + +``` +» nix build .#nixosConfigurations.test.config.system.build.toplevel +``` + +{% end %} + +# "Flakes are the future of Nix, and the only CLI" + +Many words have been spilled on the new CLI and its design, mostly focusing on +flakes. However, this is not the only mode of the new CLI: wherever it makes +sense, it actually fully supports non-flake usage. + +To get more exact equivalence with the old CLI, `-L` (`--print-build-logs`) and +`--print-out-path` are useful. Equally, the *old* CLI can have its output +improved to that of the new CLI by passing `--log-format bar-with-logs`. I +would be remiss not to mention [nix-output-monitor] as a much nicer way of +watching Nix builds, as well. + +[nix-output-monitor]: https://github.com/maralorn/nix-output-monitor + +Here is a table of the equivalences: + +
+ + + + + + + + + + + + + + + + + + + + + + + + + +
Old CLIEquivalent
nix-build -A hello
nix build -f . hello
nix-shell -A blah
nix develop -f . blah
-
nix run -f . hello
nix-build -E
'(import <nixpkgs> { config.allowUnfree = true; }).blah'
nix build --impure --expr
'(import <nixpkgs> { config.allowUnfree = true; }).blah'
nix-instantiate --eval --strict -E 'blah'
nix eval --impure --expr 'blah'
+
+ +# "Flakes are how to manage external dependencies" + +Flakes are one way of managing external dependencies but they have many flaws +in that role. + +One flaw is that all the dependencies need to be listed in one file, and there +is no way of scoping them into groups. + +A poorly documented limitation of both flake inputs and the built in fetchers +in general, which is the reason they are banned in nixpkgs (in addition to +[`restrict-eval`][restrict-eval] making them not work), is that they block +further evaluation while fetching. The alternative to this is to use a +fixed-output derivation that performs the fetching at build time, such as is +done with `pkgs.fetchFromGitHub`, `pkgs.fetchurl` and so on. + +The blocking is not necessarily the biggest problem if the dependencies are Nix +code required to evaluate the build, but it can be troublesome when the +dependencies are not required to evaluate, since it [slows down and serializes +evaluation][ifd], downloading just one thing at a time. If the dependencies are +required for evaluation, there is little way to make this better, but for +instance, for builds requiring many inputs such as a pile of tree-sitter +grammars, Haskell package sources, or such, it adds up badly. + +[ifd]: https://jade.fyi/blog/nix-evaluation-blocking/ + +## If not flakes then what? + +There's a perfectly reasonable argument to be made for just treating +dependencies the same way as nixpkgs and directly calling `pkgs.fetchurl` and +such inside Nix source. This works fine, is conventional, and avoids the +evaluation-time-build-dependency ("import from derivation" (IFD)) problems. + +It's nice to have tools to automatically update these and grab the appropriate +hash, though. + +There are several tools that can maintain a lock file with Nix hashes, such as +[Niv], [npins], and [gridlock]. The first two sadly ship Nix files that use +built-in fetchers and thus have the evaluation performance issues, and the +latter doesn't ship any Nix code. + +Thus, the solution is to ignore any provided Nix code for whichever one you choose +to use and write some code to read the tool's JSON file and pull the package +URL and hashes out, and call `pkgs.fetchurl` with them. This is quite easy to +do and we would recommend it. + +[Niv]: https://github.com/nmattia/niv +[npins]: https://github.com/andir/npins +[gridlock]: https://github.com/lf-/gridlock + +[restrict-eval]: https://nixos.org/manual/nix/stable/command-ref/conf-file.html#conf-restrict-eval + +# "Flakes are good for installing software locally" + +# What is all this crap for anyway? +