+++
date = "2022-06-15"
draft = false
path = "/blog/building-nix-derivations-manually"
tags = ["nix"]
title = "Building Nix derivations manually in nix-shell"
+++

When writing, debugging, or otherwise working with Nix expressions, it is often
useful to run a build part of the way through, or to run it manually in a
shell. For instance, perhaps you want to test some development version of a
tool against a package and want to iterate quickly. Or, a build is broken and
you want to look at it more closely than `nix-build --keep-failed` makes
convenient.

Most packages are built with the generic builder, regardless of language. The
language specific builders are then written in terms of it. For instance,
Haskell packages are [built with this builder][haskell-builder], customizing it
by adding `setupCompilerEnvironmentPhase` and overriding many phases.

[haskell-builder]: https://github.com/nixos/nixpkgs/blob/master/pkgs/development/haskell-modules/generic-builder.nix

The way `nix-build` typically builds packages with the generic builder is
documented in the [`stdenv` chapter][stdenv-docs] of the nixpkgs documentation.
More or less, it runs [various "phases"][phases] in order to build the
package. This is roughly analogous to the `build()`, `check()`, etc functions
in an Arch Linux or Alpine Linux `PKGBUILD` file.

[stdenv-docs]: https://nixos.org/manual/nixpkgs/stable/#chap-stdenv
[phases]: https://nixos.org/manual/nixpkgs/stable/#sec-stdenv-phases

The generic builder is made of function definitions in the file
[`pkgs/stdenv/generic/setup.sh`][setup.sh] in the nixpkgs repository, and it is
usually `source`d by the actual build script,
[`pkgs/stdenv/generic/default-builder.sh`][default-builder.sh], which is
trivial:

```sh
source $stdenv/setup
genericBuild
```

[setup.sh]: https://github.com/nixos/nixpkgs/blob/master/pkgs/stdenv/generic/setup.sh
[default-builder.sh]: https://github.com/nixos/nixpkgs/blob/master/pkgs/stdenv/generic/default-builder.sh

`default-builder.sh` is not always used: `setup.sh` is more or less a shell
script library and can equally be used in a custom build script that defines
some functions then calls `genericBuild`.

*By default*, `genericBuild` will run the default phases, the most notable of
which are `unpackPhase`, `configurePhase`, `buildPhase`, `checkPhase` (tests),
and `installPhase`. There are also hooks that can run. These phases are either
the functions of that name, which perform the default C `./configure; make;
make install` thing, environment variables containing scripts overriding
them, or shell functions defined in the file that sources `setup.sh`.

Another feature supported by the generic builder is "hooks" which are basically
little shell functions, possibly defined by environment variables, which can be
run at various points in the build process if present.

More phases can also be added for more complicated things to build. In
practice, all of these extensibility features mean that the generic builder is
reused for most language ecosystems.

## Running a build manually

Because of the complication of possible custom phases, hooks, etc, it is not
really prudent to just run the phase functions directly because it essentially
means you are reimplementing `genericBuild` in your head.

When debugging, what you probably want to do is consult the full phases list
and run `genericBuild` with those phases. We can see that the default phases are
the following (from [setup.sh][setupsh-phases]):

[setupsh-phases]: https://github.com/nixos/nixpkgs/blob/1e2a288f0e84b7064020554cd89415932b458c1b/pkgs/stdenv/generic/setup.sh#L1335-L1340

```sh
if [ -z "${phases:-}" ]; then
    phases="${prePhases:-} unpackPhase patchPhase ${preConfigurePhases:-} \
        configurePhase ${preBuildPhases:-} buildPhase checkPhase \
        ${preInstallPhases:-} installPhase ${preFixupPhases:-} fixupPhase installCheckPhase \
        ${preDistPhases:-} distPhase ${postPhases:-}";
fi
```

A manual build of a package might look like this:

First, open a nix-shell with the derivation of the package in question. For
instance, `nix-shell '<nixpkgs>' -A hello`.

Then do:

```sh
phases="${prePhases:-} unpackPhase patchPhase" genericBuild
```

which will dump you into a shell in the sources directory for the package.

Once you've done the stuff you'll run once, you can run the build as many times
as you'd like:

```sh
out=$(pwd)/out phases="${preConfigurePhases:-} configurePhase ${preBuildPhases:-} buildPhase" genericBuild
```

The `$out` variable has to be specified because it defaults to somewhere in the
nix store, which usually breaks builds since is not available for writing if
you are not the actual sandboxed nix builder. If there are other outputs such
as `doc`, these also need to be specified here.