diff --git a/content/posts/pinning-packages-in-nix.md b/content/posts/pinning-packages-in-nix.md index eca3419..05bef38 100644 --- a/content/posts/pinning-packages-in-nix.md +++ b/content/posts/pinning-packages-in-nix.md @@ -1,24 +1,21 @@ +++ -date = "2024-04-02" -draft = true +date = "2024-05-19" +draft = false path = "/blog/pinning-packages-in-nix" tags = ["nix"] title = "Pinning packages in Nix" +++ -Although Nix supposedly makes pinning things easy, it really does not seem so: -it is not possible to simply write `package = "^5.0.1"` in some file somewhere -and get *one* package pinned at a specific version. Though this is frustrating, -there is a reason for this, and it primarily speaks to how nixpkgs is a Linux -distribution and is unlike a standard language package manager. +Although Nix supposedly makes pinning things easy, it really does not seem so +from a perspective of looking at other software using pinning: it is not +possible to simply write `package = "^5.0.1"` in some file somewhere and get +*one* package pinned at a specific version. Though this is frustrating, there +is a reason for this, and it primarily speaks to how nixpkgs is a Linux +distribution and how Nix is unlike a standard language package manager. This post will go through the ways to pin a package to some older version and why one would use each method. -## FIXME -mention that these methods can generally be overlayed. mention that overlaying -*across different nixpkgs* is probably a bad idea - # Simply add an older version of nixpkgs > Software regressed? No patches in master to fix it? Try 30-40 different @@ -26,9 +23,11 @@ mention that these methods can generally be overlayed. mention that overlaying pinning 30-40 versions of nixpkgs. Unlike most systems, it is fine to mix versions of nixpkgs, although it will -likely go wrong if, e.g. libraries are intermingled between versions. But, if -one package is all that is necessary, one can in fact simply import another -version of nixpkgs. +likely go wrong if, e.g. libraries are intermingled between versions (*in +particular*, it is inadvisable to replace some program with a version +from a different nixpkgs from within an overlay for this reason). But, if one +package is all that is necessary, one can in fact simply import another version +of nixpkgs. This works because binaries from multiple versions of nixpkgs can coexist on a computer and simply work. However, it can go wrong if they are loading @@ -123,30 +122,189 @@ with the desired package version and then copy the `package.nix` or `default.nix` or such into your own project, and then call it with `callPackage`. +You can find it with something like: + +``` + » nix eval --raw -f '' hello.meta.position +/nix/store/0qd773b63yg8435w8hpm13zqz7iipcbs-source/pkgs/by-name/he/hello/package.nix:41 +``` + +Or, equivalently, with `nix repl -f ''`, `:e hello` or to do the same +as above, `hello.meta.position`. + +Then, vendor that file into your configurations repository. + +Once it is vendored, it can be used either from an overlay: + +```nix +final: prev: { + hello = final.callPackage ./hello-vendored.nix { }; +} +``` + +or directly in your use site: + +```nix +{ pkgs, ... }: { + environment.systemPackages = [ + (pkgs.callPackage ./vendored-hello.nix { }) + ]; +} +``` + +
Difficulty
Slight effort
Rebuilds
-None, but will bring in another copy of nixpkgs and any dependencies (and -transitive dependencies). +For the overlay use case, this will build the overridden package and anything +depending on it. For the direct at use site case, this will just rebuild the +package, and anything depending on it will get the version in upstream nixpkgs.
# Patch the package with overrides -maybe explain what .override does +nixpkgs offers several separate methods to "override" things that mean +different things. In short: + +- [`somePackage.override`][override] replaces the dependencies of a package; + more specifically the dependencies injected by `callPackage`. It accepts an + attribute set but can also accept a lambda of one argument, providing the + previous dependencies of the package. +- [`somePackage.overrideAttrs`][overrideAttrs] replaces the `stdenv.mkDerivation` + arguments of a package. This lets you replace the `src` of a package, in + principle. +- [`overrideCabal`][overrideCabal] replaces the `haskellPackages.mkDerivation` + arguments for a Haskell package in a similar way that `overrideAttrs` does for + `stdenv.mkDerivation`. This is internally implemented by methods equivalent + to the evil crimes below. + +[override]: https://nixos.org/manual/nixpkgs/stable/#sec-pkg-override +[overrideAttrs]: https://nixos.org/manual/nixpkgs/stable/#sec-pkg-overrideAttrs +[overrideCabal]: https://nixos.org/manual/nixpkgs/stable/#haskell-overriding-haskell-packages + +Here are some examples: + +Build an openttd with a different upstream source by putting this in +`openttd-jgrpp.nix`: + +```nix +{ openttd, fetchFromGitHub }: +openttd.overrideAttrs (old: { + src = fetchFromGitHub { + owner = "jgrennison"; + repo = "openttd-patches"; + rev = "jgrpp-0.57.1"; + sha256 = "sha256-mQy+QdhEXoM9wIWvSkMgRVBXJO1ugXWS3lduccez1PQ="; + }; +}) +``` + +then `pkgs.callPackage ./openttd-jgrpp.nix { }`. + +For instance, the following (rather silly) command will build such a file: + +``` + » nix build -L --impure --expr 'with import {}; callPackage ./openttd-jgrpp.nix {}' +``` ## Limitations -go and rust bustedness -link to the architecture issue +Most notably, [overrideAttrs doesn't work][overrideAttrs-busted] on several +significant language ecosystems including Rust and Go, since one almost always +needs to override the arguments of `buildRustPackage` or `buildGoPackage` when +replacing something. For these, either one can do crimes to introduce an +`overrideRust` function (see below), or one can cry briefly and then vendor the +package. The latter is easier. + +```nix +let + pkgs = import { }; + # Give the package a fake buildRustPackage from callPackage that modifies the + # arguments through a function. + overrideRust = f: drv: drv.override (oldArgs: + let rustPlatform = oldArgs.rustPlatform or pkgs.rustPlatform; + in oldArgs // { + rustPlatform = rustPlatform // { + buildRustPackage = args: rustPlatform.buildRustPackage (f args); + }; + }); + + # Take some arguments to buildRustPackage and make new ones. In this case, + # override the version and the hash + evil = oldArgs: oldArgs // { + src = oldArgs.src.override { + rev = "v0.20.9"; + sha256 = "sha256-NxWqpMNwu5Ajffw1E2q9KS4TgkCH6M+ctFyi9Jp0tqQ="; + }; + version = "master"; + # FIXME: if you are actually doing this put a real hash here + cargoSha256 = pkgs.lib.fakeHash; + }; + +in +{ + x = overrideRust evil pkgs.tree-sitter; +} +``` + +[overrideAttrs-busted]: https://github.com/NixOS/nixpkgs/issues/99100 + +Then: `nix build -L -f evil.nix x` + +
+
Difficulty
+
Highly variable, sometimes trivial, sometimes nearly impossible, depending +on architectural flaws of nixpkgs.
+
Rebuilds
+
+For the overlay use case of actually using this overridden package, this will +build the overridden package and anything depending on it. For the direct at +use site case, this will just rebuild the package, and anything depending on it +will get the version in upstream nixpkgs. +
+
# Patch a NixOS module -disable modules thing +If one wants to replace a NixOS module, say, by getting it from a later version +of nixpkgs, see [Replacing Modules] in the NixOS manual. + +[Replacing Modules]: https://nixos.org/manual/nixos/stable/#sec-replace-modules # Patch the base system without a world rebuild -xz etc +It's possible to replace an entire store path with another inside a NixOS +system without rebuilding the world (but wasting some space (by duplicating +things for the rewritten version) and being somewhat evil/potentially unsound +since it is just a text replacement of the hashes). This can be achieved with +the NixOS option +[`system.replaceRuntimeDependencies`][replaceRuntimeDependencies]. + +[replaceRuntimeDependencies]: https://nixos.org/manual/nixos/stable/options#opt-system.replaceRuntimeDependencies + +# Why do we need all of this? + +The primary reason that Nix doesn't allow trivially overriding packages with a +different version is that it is a generalized build system building software +that has non-uniform expectations of how to be built. One can in indeed see +that the "replace one version with some other in some file" idea is *almost* +reality in languages using `mkDerivation` directly, though one might have to +tweak other build properties sometimes. Architectural problems in nixpkgs +prevent this working for several ecosystems. + +Another sort of issue is that nixpkgs tries to provide a mostly [globally +coherent] set of software versions, where, like most Linux distributions, there +is generally one blessed version of a library with some exceptions. This is, in +fact, mandatory to be able to have any cache hits as a hermetic build system: +if everyone was building slightly different versions of libraries, all +downstream packages will have different hashes and thus miss the cache. + +So, in a way, a software distribution based on Nix cannot have separate locking +for every package and simultaneously have functional caches: the moment that +everything is not built together, caches will miss. + +[globally coherent]: https://www.haskellforall.com/2022/05/the-golden-rule-of-software.html