jade/blog
0
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

pinning nix things

Browse source
This commit is contained in:
Jade Lovelace 2024-05-19 19:10:35 -07:00
parent 1f33d774c2
commit 4d83323451
1 changed files with 179 additions and 21 deletions
Download patch file Download diff file Expand all files Collapse all files

200
content/posts/pinning-packages-in-nix.md
View file

@ -1,24 +1,21 @@
+++
date = "2024-04-02"
draft = true
date = "2024-05-19"
draft = false
path = "/blog/pinning-packages-in-nix"
tags = ["nix"]
title = "Pinning packages in Nix"
+++
Although Nix supposedly makes pinning things easy, it really does not seem so:
it is not possible to simply write `package = "^5.0.1"` in some file somewhere
and get *one* package pinned at a specific version. Though this is frustrating,
there is a reason for this, and it primarily speaks to how nixpkgs is a Linux
distribution and is unlike a standard language package manager.
Although Nix supposedly makes pinning things easy, it really does not seem so
from a perspective of looking at other software using pinning: it is not
possible to simply write `package = "^5.0.1"` in some file somewhere and get
*one* package pinned at a specific version. Though this is frustrating, there
is a reason for this, and it primarily speaks to how nixpkgs is a Linux
distribution and how Nix is unlike a standard language package manager.
This post will go through the ways to pin a package to some older version and
why one would use each method.
## FIXME
mention that these methods can generally be overlayed. mention that overlaying
*across different nixpkgs* is probably a bad idea
# Simply add an older version of nixpkgs
> Software regressed? No patches in master to fix it? Try 30-40 different
@ -26,9 +23,11 @@ mention that these methods can generally be overlayed. mention that overlaying
pinning 30-40 versions of nixpkgs.
Unlike most systems, it is fine to mix versions of nixpkgs, although it will
likely go wrong if, e.g. libraries are intermingled between versions. But, if
one package is all that is necessary, one can in fact simply import another
version of nixpkgs.
likely go wrong if, e.g. libraries are intermingled between versions (*in
particular*, it is inadvisable to replace some program with a version
from a different nixpkgs from within an overlay for this reason). But, if one
package is all that is necessary, one can in fact simply import another version
of nixpkgs.
This works because binaries from multiple versions of nixpkgs can coexist
on a computer and simply work. However, it can go wrong if they are loading
@ -123,30 +122,189 @@ with the desired package version and then copy the `package.nix` or
`default.nix` or such into your own project, and then call it with
`callPackage`.
You can find it with something like:
```
» nix eval --raw -f '<nixpkgs>' hello.meta.position
/nix/store/0qd773b63yg8435w8hpm13zqz7iipcbs-source/pkgs/by-name/he/hello/package.nix:41
```
Or, equivalently, with `nix repl -f '<nixpkgs>'`, `:e hello` or to do the same
as above, `hello.meta.position`.
Then, vendor that file into your configurations repository.
Once it is vendored, it can be used either from an overlay:
```nix
final: prev: {
hello = final.callPackage ./hello-vendored.nix { };
}
```
or directly in your use site:
```nix
{ pkgs, ... }: {
environment.systemPackages = [
(pkgs.callPackage ./vendored-hello.nix { })
];
}
```
<dl>
<dt>Difficulty</dt>
<dd>Slight effort</dd>
<dt>Rebuilds</dt>
<dd>
None, but will bring in another copy of nixpkgs and any dependencies (and
transitive dependencies).
For the overlay use case, this will build the overridden package and anything
depending on it. For the direct at use site case, this will just rebuild the
package, and anything depending on it will get the version in upstream nixpkgs.
</dd>
</dl>
# Patch the package with overrides
maybe explain what .override does
nixpkgs offers several separate methods to "override" things that mean
different things. In short:
- [`somePackage.override`][override] replaces the dependencies of a package;
more specifically the dependencies injected by `callPackage`. It accepts an
attribute set but can also accept a lambda of one argument, providing the
previous dependencies of the package.
- [`somePackage.overrideAttrs`][overrideAttrs] replaces the `stdenv.mkDerivation`
arguments of a package. This lets you replace the `src` of a package, in
principle.
- [`overrideCabal`][overrideCabal] replaces the `haskellPackages.mkDerivation`
arguments for a Haskell package in a similar way that `overrideAttrs` does for
`stdenv.mkDerivation`. This is internally implemented by methods equivalent
to the evil crimes below.
[override]: https://nixos.org/manual/nixpkgs/stable/#sec-pkg-override
[overrideAttrs]: https://nixos.org/manual/nixpkgs/stable/#sec-pkg-overrideAttrs
[overrideCabal]: https://nixos.org/manual/nixpkgs/stable/#haskell-overriding-haskell-packages
Here are some examples:
Build an openttd with a different upstream source by putting this in
`openttd-jgrpp.nix`:
```nix
{ openttd, fetchFromGitHub }:
openttd.overrideAttrs (old: {
src = fetchFromGitHub {
owner = "jgrennison";
repo = "openttd-patches";
rev = "jgrpp-0.57.1";
sha256 = "sha256-mQy+QdhEXoM9wIWvSkMgRVBXJO1ugXWS3lduccez1PQ=";
};
})
```
then `pkgs.callPackage ./openttd-jgrpp.nix { }`.
For instance, the following (rather silly) command will build such a file:
```
» nix build -L --impure --expr 'with import <nixpkgs> {}; callPackage ./openttd-jgrpp.nix {}'
```
## Limitations
go and rust bustedness
link to the architecture issue
Most notably, [overrideAttrs doesn't work][overrideAttrs-busted] on several
significant language ecosystems including Rust and Go, since one almost always
needs to override the arguments of `buildRustPackage` or `buildGoPackage` when
replacing something. For these, either one can do crimes to introduce an
`overrideRust` function (see below), or one can cry briefly and then vendor the
package. The latter is easier.
```nix
let
pkgs = import <nixpkgs> { };
# Give the package a fake buildRustPackage from callPackage that modifies the
# arguments through a function.
overrideRust = f: drv: drv.override (oldArgs:
let rustPlatform = oldArgs.rustPlatform or pkgs.rustPlatform;
in oldArgs // {
rustPlatform = rustPlatform // {
buildRustPackage = args: rustPlatform.buildRustPackage (f args);
};
});
# Take some arguments to buildRustPackage and make new ones. In this case,
# override the version and the hash
evil = oldArgs: oldArgs // {
src = oldArgs.src.override {
rev = "v0.20.9";
sha256 = "sha256-NxWqpMNwu5Ajffw1E2q9KS4TgkCH6M+ctFyi9Jp0tqQ=";
};
version = "master";
# FIXME: if you are actually doing this put a real hash here
cargoSha256 = pkgs.lib.fakeHash;
};
in
{
x = overrideRust evil pkgs.tree-sitter;
}
```
[overrideAttrs-busted]: https://github.com/NixOS/nixpkgs/issues/99100
Then: `nix build -L -f evil.nix x`
<dl>
<dt>Difficulty</dt>
<dd>Highly variable, sometimes trivial, sometimes nearly impossible, depending
on architectural flaws of nixpkgs.</dd>
<dt>Rebuilds</dt>
<dd>
For the overlay use case of actually using this overridden package, this will
build the overridden package and anything depending on it. For the direct at
use site case, this will just rebuild the package, and anything depending on it
will get the version in upstream nixpkgs.
</dd>
</dl>
# Patch a NixOS module
disable modules thing
If one wants to replace a NixOS module, say, by getting it from a later version
of nixpkgs, see [Replacing Modules] in the NixOS manual.
[Replacing Modules]: https://nixos.org/manual/nixos/stable/#sec-replace-modules
# Patch the base system without a world rebuild
xz etc
It's possible to replace an entire store path with another inside a NixOS
system without rebuilding the world (but wasting some space (by duplicating
things for the rewritten version) and being somewhat evil/potentially unsound
since it is just a text replacement of the hashes). This can be achieved with
the NixOS option
[`system.replaceRuntimeDependencies`][replaceRuntimeDependencies].
[replaceRuntimeDependencies]: https://nixos.org/manual/nixos/stable/options#opt-system.replaceRuntimeDependencies
# Why do we need all of this?
The primary reason that Nix doesn't allow trivially overriding packages with a
different version is that it is a generalized build system building software
that has non-uniform expectations of how to be built. One can in indeed see
that the "replace one version with some other in some file" idea is *almost*
reality in languages using `mkDerivation` directly, though one might have to
tweak other build properties sometimes. Architectural problems in nixpkgs
prevent this working for several ecosystems.
Another sort of issue is that nixpkgs tries to provide a mostly [globally
coherent] set of software versions, where, like most Linux distributions, there
is generally one blessed version of a library with some exceptions. This is, in
fact, mandatory to be able to have any cache hits as a hermetic build system:
if everyone was building slightly different versions of libraries, all
downstream packages will have different hashes and thus miss the cache.
So, in a way, a software distribution based on Nix cannot have separate locking
for every package and simultaneously have functional caches: the moment that
everything is not built together, caches will miss.
[globally coherent]: https://www.haskellforall.com/2022/05/the-golden-rule-of-software.html