pinning nix things

content/posts/pinning-packages-in-nix.md

@@ -1,24 +1,21 @@
 +++
-date = "2024-04-02"
+date = "2024-05-19"
-draft = true
+draft = false
 path = "/blog/pinning-packages-in-nix"
 tags = ["nix"]
 title = "Pinning packages in Nix"
 +++
 
-Although Nix supposedly makes pinning things easy, it really does not seem so:
+Although Nix supposedly makes pinning things easy, it really does not seem so
-it is not possible to simply write `package = "^5.0.1"` in some file somewhere
+from a perspective of looking at other software using pinning: it is not
-and get *one* package pinned at a specific version. Though this is frustrating,
+possible to simply write `package = "^5.0.1"` in some file somewhere and get
-there is a reason for this, and it primarily speaks to how nixpkgs is a Linux
+*one* package pinned at a specific version. Though this is frustrating, there
-distribution and is unlike a standard language package manager.
+is a reason for this, and it primarily speaks to how nixpkgs is a Linux
+distribution and how Nix is unlike a standard language package manager.
 
 This post will go through the ways to pin a package to some older version and
 why one would use each method.
 
-## FIXME
-mention that these methods can generally be overlayed. mention that overlaying
-*across different nixpkgs* is probably a bad idea
-
 # Simply add an older version of nixpkgs
 
 > Software regressed? No patches in master to fix it? Try 30-40 different
@@ -26,9 +23,11 @@ mention that these methods can generally be overlayed. mention that overlaying
   pinning 30-40 versions of nixpkgs.
 
 Unlike most systems, it is fine to mix versions of nixpkgs, although it will
-likely go wrong if, e.g. libraries are intermingled between versions. But, if
+likely go wrong if, e.g. libraries are intermingled between versions (*in
-one package is all that is necessary, one can in fact simply import another
+particular*, it is inadvisable to replace some program with a version
-version of nixpkgs.
+from a different nixpkgs from within an overlay for this reason). But, if one
+package is all that is necessary, one can in fact simply import another version
+of nixpkgs.
 
 This works because binaries from multiple versions of nixpkgs can coexist
 on a computer and simply work. However, it can go wrong if they are loading
@@ -123,30 +122,189 @@ with the desired package version and then copy the `package.nix` or
 `default.nix` or such into your own project, and then call it with
 `callPackage`.
 
+You can find it with something like:
+
+```
+ » nix eval --raw -f '<nixpkgs>' hello.meta.position
+/nix/store/0qd773b63yg8435w8hpm13zqz7iipcbs-source/pkgs/by-name/he/hello/package.nix:41
+```
+
+Or, equivalently, with `nix repl -f '<nixpkgs>'`, `:e hello` or to do the same
+as above, `hello.meta.position`.
+
+Then, vendor that file into your configurations repository.
+
+Once it is vendored, it can be used either from an overlay:
+
+```nix
+final: prev: {
+  hello = final.callPackage ./hello-vendored.nix { };
+}
+```
+
+or directly in your use site:
+
+```nix
+{ pkgs, ... }: {
+  environment.systemPackages = [
+    (pkgs.callPackage ./vendored-hello.nix { })
+  ];
+}
+```
+
+
 <dl>
 <dt>Difficulty</dt>
 <dd>Slight effort</dd>
 <dt>Rebuilds</dt>
 <dd>
-None, but will bring in another copy of nixpkgs and any dependencies (and
+For the overlay use case, this will build the overridden package and anything
-transitive dependencies).
+depending on it. For the direct at use site case, this will just rebuild the
+package, and anything depending on it will get the version in upstream nixpkgs.
 </dd>
 </dl>
 
 # Patch the package with overrides
 
-maybe explain what .override does
+nixpkgs offers several separate methods to "override" things that mean
+different things. In short:
+
+- [`somePackage.override`][override] replaces the dependencies of a package;
+  more specifically the dependencies injected by `callPackage`. It accepts an
+  attribute set but can also accept a lambda of one argument, providing the
+  previous dependencies of the package.
+- [`somePackage.overrideAttrs`][overrideAttrs] replaces the `stdenv.mkDerivation`
+  arguments of a package. This lets you replace the `src` of a package, in
+  principle.
+- [`overrideCabal`][overrideCabal] replaces the `haskellPackages.mkDerivation`
+  arguments for a Haskell package in a similar way that `overrideAttrs` does for
+  `stdenv.mkDerivation`. This is internally implemented by methods equivalent
+  to the evil crimes below.
+
+[override]: https://nixos.org/manual/nixpkgs/stable/#sec-pkg-override
+[overrideAttrs]: https://nixos.org/manual/nixpkgs/stable/#sec-pkg-overrideAttrs
+[overrideCabal]: https://nixos.org/manual/nixpkgs/stable/#haskell-overriding-haskell-packages
+
+Here are some examples:
+
+Build an openttd with a different upstream source by putting this in
+`openttd-jgrpp.nix`:
+
+```nix
+{ openttd, fetchFromGitHub }:
+openttd.overrideAttrs (old: {
+  src = fetchFromGitHub {
+    owner = "jgrennison";
+    repo = "openttd-patches";
+    rev = "jgrpp-0.57.1";
+    sha256 = "sha256-mQy+QdhEXoM9wIWvSkMgRVBXJO1ugXWS3lduccez1PQ=";
+  };
+})
+```
+
+then `pkgs.callPackage ./openttd-jgrpp.nix { }`.
+
+For instance, the following (rather silly) command will build such a file:
+
+```
+ » nix build -L --impure --expr 'with import <nixpkgs> {}; callPackage ./openttd-jgrpp.nix {}'
+```
 
 ## Limitations
 
-go and rust bustedness
+Most notably, [overrideAttrs doesn't work][overrideAttrs-busted] on several
-link to the architecture issue
+significant language ecosystems including Rust and Go, since one almost always
+needs to override the arguments of `buildRustPackage` or `buildGoPackage` when
+replacing something. For these, either one can do crimes to introduce an
+`overrideRust` function (see below), or one can cry briefly and then vendor the
+package. The latter is easier.
+
+```nix
+let
+  pkgs = import <nixpkgs> { };
+  # Give the package a fake buildRustPackage from callPackage that modifies the
+  # arguments through a function.
+  overrideRust = f: drv: drv.override (oldArgs:
+    let rustPlatform = oldArgs.rustPlatform or pkgs.rustPlatform;
+    in oldArgs // {
+      rustPlatform = rustPlatform // {
+        buildRustPackage = args: rustPlatform.buildRustPackage (f args);
+      };
+    });
+
+  # Take some arguments to buildRustPackage and make new ones. In this case,
+  # override the version and the hash
+  evil = oldArgs: oldArgs // {
+    src = oldArgs.src.override {
+      rev = "v0.20.9";
+      sha256 = "sha256-NxWqpMNwu5Ajffw1E2q9KS4TgkCH6M+ctFyi9Jp0tqQ=";
+    };
+    version = "master";
+    # FIXME: if you are actually doing this put a real hash here
+    cargoSha256 = pkgs.lib.fakeHash;
+  };
+
+in
+{
+  x = overrideRust evil pkgs.tree-sitter;
+}
+```
+
+[overrideAttrs-busted]: https://github.com/NixOS/nixpkgs/issues/99100
+
+Then: `nix build -L -f evil.nix x`
+
+<dl>
+<dt>Difficulty</dt>
+<dd>Highly variable, sometimes trivial, sometimes nearly impossible, depending
+on architectural flaws of nixpkgs.</dd>
+<dt>Rebuilds</dt>
+<dd>
+For the overlay use case of actually using this overridden package, this will
+build the overridden package and anything depending on it. For the direct at
+use site case, this will just rebuild the package, and anything depending on it
+will get the version in upstream nixpkgs.
+</dd>
+</dl>
 
 # Patch a NixOS module
 
-disable modules thing
+If one wants to replace a NixOS module, say, by getting it from a later version
+of nixpkgs, see [Replacing Modules] in the NixOS manual.
+
+[Replacing Modules]: https://nixos.org/manual/nixos/stable/#sec-replace-modules
 
 # Patch the base system without a world rebuild
 
-xz etc
+It's possible to replace an entire store path with another inside a NixOS
+system without rebuilding the world (but wasting some space (by duplicating
+things for the rewritten version) and being somewhat evil/potentially unsound
+since it is just a text replacement of the hashes). This can be achieved with
+the NixOS option
+[`system.replaceRuntimeDependencies`][replaceRuntimeDependencies].
+
+[replaceRuntimeDependencies]: https://nixos.org/manual/nixos/stable/options#opt-system.replaceRuntimeDependencies
+
+# Why do we need all of this?
+
+The primary reason that Nix doesn't allow trivially overriding packages with a
+different version is that it is a generalized build system building software
+that has non-uniform expectations of how to be built. One can in indeed see
+that the "replace one version with some other in some file" idea is *almost*
+reality in languages using `mkDerivation` directly, though one might have to
+tweak other build properties sometimes. Architectural problems in nixpkgs
+prevent this working for several ecosystems.
+
+Another sort of issue is that nixpkgs tries to provide a mostly [globally
+coherent] set of software versions, where, like most Linux distributions, there
+is generally one blessed version of a library with some exceptions. This is, in
+fact, mandatory to be able to have any cache hits as a hermetic build system:
+if everyone was building slightly different versions of libraries, all
+downstream packages will have different hashes and thus miss the cache.
+
+So, in a way, a software distribution based on Nix cannot have separate locking
+for every package and simultaneously have functional caches: the moment that
+everything is not built together, caches will miss.
+
+[globally coherent]: https://www.haskellforall.com/2022/05/the-golden-rule-of-software.html