jade/blog
0
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

rewrite a bunch of the nix post

Browse source
This commit is contained in:
Jade 2021-03-13 03:22:19 -08:00
parent 41812643dd
commit e12ccb5006
1 changed files with 160 additions and 90 deletions
Download patch file Download diff file Expand all files Collapse all files

242
content/posts/nix-and-haskell.md
View file

@ -6,17 +6,14 @@ tags = ["nix", "haskell"]
title = "Using Nix to build multi-package, full stack Haskell apps" title = "Using Nix to build multi-package, full stack Haskell apps"
+++ +++
This post has been updated on June 16 following the finalization of the Nix **UPDATE**: March 13 2021 - rewrote a fair amount of the post
port.
As part of my job working on an [open source logic As part of my job working on an [open source logic
textbook](https://carnap.io), I picked up a Haskell textbook](https://carnap.io), I picked up a Haskell
codebase that was rather hard to build. This was problematic for new codebase that was rather hard to build. This was problematic for new
contributors getting started, so I wanted to come up with a better process. contributors getting started, so I wanted to come up with a better process.
Further, because of legal requirements for public institutions in BC, I need to Further, I was interested in this simplification allowing continuous
be able to host this software in Canada, for which it would be useful to be integration, packaging, and other useful process improvements.
able to have CI and containerization (where it is directly useful to have an
easy to set up build environment).
The value proposition of Nix is that it ensures that regardless of who is The value proposition of Nix is that it ensures that regardless of who is
building the software or where it is being built, it is possible to ensure the building the software or where it is being built, it is possible to ensure the
@ -40,31 +37,33 @@ for one, the user facing documentation seems to be less complete than the
documentation comments on functions, and often it is useful to read the library documentation comments on functions, and often it is useful to read the library
function source alongside the documentation. function source alongside the documentation.
Usually I keep a tab in my neovim open to the nixpkgs source and use either I recommend having a terminal with a `nix repl` and the
[nix-doc](https://github.com/lf-/nix-doc) or [nix-doc](https://github.com/lf-/nix-doc) plugin and an editor with a
[ripgrep](https://github.com/BurntSushi/ripgrep) to search for the function I checkout of the `nixpkgs` source code open while working on Nix stuff.
am interested in.
----- # Implementation
This post summarizes the design decisions that went into implementing Nix for This post summarizes the design decisions that went into implementing Nix for
this existing full stack app. If you'd like to read the source, it is this existing full stack app. If you'd like to read the source, it is
[available on GitHub](https://github.com/lf-/Carnap/tree/nix). [available on GitHub](https://github.com/Carnap/Carnap/).
## `default.nix`
I have a top-level `default.nix` that imports nixpkgs with overlays for each I have a top-level `default.nix` that imports nixpkgs with overlays for each
conceptual part of the application (this could all be done in one but it is conceptual part of the application (this could all be done in one but it is
useful to separate them for maintenance purposes). A simplified version is useful to separate them for maintenance purposes). A simplified version is
below: below:
{% codesample(desc="`default.nix` outline") %}
```nix ```nix
{ compiler ? "ghc865", { compiler ? "ghc865",
ghcjs ? "ghcjs" ghcjs ? "ghcjs"
}: }:
let nixpkgs = import (builtins.fetchTarball { let # I highly recommend using `niv` for managing git versions of things
name = "nixpkgs-20.03-2020-06-28"; # It will generate this sources.nix file for you.
url = "https://github.com/NixOS/nixpkgs/archive/f8248ab6d9e69ea9c07950d73d48807ec595e923.zip"; sources = import ./nix/sources.nix;
sha256 = "009i9j6mbq6i481088jllblgdnci105b2q4mscprdawg3knlyahk"; nixpkgs = sources.nixpkgs {
}) {
config = { config = {
# Use this if you use 'broken' packages that are fixed in an overlay # Use this if you use 'broken' packages that are fixed in an overlay
allowBroken = true; allowBroken = true;
@ -80,23 +79,104 @@ below:
} }
``` ```
{% end %}
In each Haskell package, use `cabal2nix .` to generate nix files for the ## Overlays and `server.nix`
package. These nix files can then be picked up with
[`lib.callPackage`](https://github.com/NixOS/nixpkgs/blob/b63f684/lib/customisation.nix#L96-L121) A brief interlude on overlays: an overlay is a function taking two curried
in an overlay: arguments, `self` and `super`, which are extremely poorly named. They should
be called something like `final` and `current` respectively (but the
convention has been made already). `final` is the package set after all
the overlays have been applied, and `current` is the state after all the
states *before* this overlay have been applied.
Overlays return a new attribute set that will be used to update the parent
package set. However there is a significant footgun that the workaround for
results in annoying amounts of nesting on account of Nix's `//` attribute set
update operator doing shallow updates: if you are overriding subsets such as
the Haskell packages, you need to update each piece at each level. It will
hopefully be clearer in the example below.
There is a [workaround to avoid this nesting in our real Nix expressions](https://github.com/Carnap/Carnap/blob/aec8fab2619f54d2ad7b2c59b2d4d11d6eda09bc/nix/compose-haskell-overlays.nix)
but I've chosen to write it out in full here for the sake of simplicity.
I'll also demonstrate the several types of hacks you will probably have to do
on a sufficiently large codebase to get dependencies' Haskell packages to
build even when they are broken in `nixpkgs`.
In each Haskell package, use `callCabal2nix` to add your own packages:
{% codesample(desc="`server.nix` sample") %}
```nix ```nix
{ ghcjs ? "ghcjs", compiler ? "ghc865" }: { ghcjs ? "ghcjs", compiler ? "ghc865", sources }:
self: super: self: super:
let overrideCabal = super.haskell.lib.overrideCabal; let # import the library functions from super, not self
inherit (super.haskell.lib)
overrideCabal # lets you override various settings of a haskell
# package. doJailbreak, dontCheck, etc are just wrappers
# around individual properties of this
callCabal2nix # calls cabal2nix with some source and makes a nix package
doJailbreak # disables cabal version bounds. it's a big hammer.
dontCheck # disables tests
justStaticExecutables # disables a bunch of unnecessary output + build
# steps for binaries
overrideSrc;
in { in {
haskell = super.haskell // { haskell = super.haskell // {
packages = super.haskell.packages // { packages = super.haskell.packages // {
"${compiler}" = super.haskell.packages."${compiler}".override { "${compiler}" = super.haskell.packages."${compiler}".override {
overrides = newpkgs: oldpkgs: { overrides = newpkgs: oldpkgs: {
Common1 = oldpkgs.callPackage ./Common1/Common1.nix { }; # your stuff
Common1 = oldpkgs.callCabal2nix "Common1" ./Common1 { };
# a note! if you don't use nix for your production builds, you can
# basically not worry about most of the stuff for Server as it
# *exclusively* applies to when you are doing something like
# `nix-build -A server`. If you aren't using Nix for building your
# app, you can just put `callCabal2nix "Server" ./Server {}` here.
Server = justStaticExecutables (overrideCabal
(oldpkgs.callCabal2nix "Server" ./Server { })
(old: let client = oldpkgs."${ghcjs}".packages.Client; in {
# copy the client into the package
preConfigure = ''
cp ${client.out}/bin/Main.jsexe/main.js static/client/
'';
# disabling these saves a pile of build time as it doesn't
# build the app twice
enableLibraryProfiling = false;
enableExecutableProfiling = false;
isExecutable = true;
})
)
# ... # ...
# HACK: you have to downgrade some dependency
hoauth2 = oldpkgs.callHackage "hoauth2" "1.8.9" { };
# HACK: you need a dependency newer than the `all-cabal-hashes`
# used by your version of `nixpkgs`. Put in the version and name
# here, and just zero out part of the hash. It will fail to build
# and tell you the right hash.
yesod-auth-oauth2 = oldpkgs.callHackageDirect {
pkg = "yesod-auth-oauth2";
ver = "0.6.1.3";
sha256 = "1bikn9kfw6mrsais4z1nk07aa7i7hyrcs411kbbfgc7n74k6sd5b";
} { };
# HACK: some dependency has broken tests
tz = dontCheck oldpkgs.tz;
# HACK: some dependency has excessively tight version bounds (this
# is like `allow-newer` in `stack.yaml` but it applies to only one
# package)
yesod-persistent = doJailbreak oldpkgs.yesod-persistent;
# HACK: you have to replace the source of a dependency entirely
# (this also demonstrates how to use a subdirectory from a source,
# in this case "/persistent" under the yesod persistent repository)
persistent = overrideSrc oldpkgs.persistent { src = (sources.persistent + "/persistent"); };
}; };
}; };
}; };
@ -104,58 +184,61 @@ in an overlay:
} }
``` ```
## Shells {% end %}
You could normally use ## Shells and development tools
[`nixpkgs.haskell.packages.${ghcVer}.shellFor`](https://github.com/NixOS/nixpkgs/blob/c565d7c/pkgs/development/haskell-modules/make-package-set.nix#L288)
to construct a shell. However, this is not ideal for multiple package projects
since it will invariably make Nix build some of your projects because they are
"dependencies".
There does not appear to be any built in resolution for this. However, You can construct shells with arbitrary developer tools from `nixpkgs`. I
[reflex-platform](https://github.com/reflex-frp/reflex-platform), has cannot overstate how awesome this feature is: a lot of Haskell developer
integrated a module called tools are supremely frustrating to build oneself, you know everyone has the
[`workOnMulti`](https://github.com/reflex-frp/reflex-platform/blob/20ed151/nix-utils/work-on-multi/default.nix). same version, and Nix means everyone can get them.
I thus took the opportunity to extricate it from its dependencies on the rest
of reflex-platform to be able to use it independently. This extracted version
is [available here](https://github.com/lf-/Carnap/blob/cde2671/nix/work-on-multi.nix).
It can be used thus: Nix will build/retrieve the tools listed in `buildInputs` and put them in
your shell's PATH, and it will *retrieve the dependencies* for the packages
returned by the function passed as `packages` and make those available, but
it will not build the packages given. This lets you use cabal to manage
builds while programming, and have all your dependencies provided so cabal
*only* builds your software.
Put the following in a `default.nix` (here's the [production version for reference](https://github.com/Carnap/Carnap/blob/aec8fab2619f54d2ad7b2c59b2d4d11d6eda09bc/default.nix)):
{% codesample(desc="`default.nix` sample, continued") %}
```nix ```nix
let # import nixpkgs with overlays... let
workOnMulti = import ./nix/work-on-multi.nix { # merge this section so that nixpkgs import is at the same level of the let
inherit nixpkgs; # block
# put whatever tools you want in the shell environments here devtools = { isGhcjs }: with nixpkgs.haskell.packages."${compiler}"; ([
generalDevTools = _: {
inherit (nixpkgs) cabal2nix;
inherit (nixpkgs.haskell.packages."${ghcVer}")
Cabal Cabal
cabal-install cabal-install
ghcid ghcid
hasktags; hasktags
yesod-bin
# hls is disabled for ghcjs shells because it probably will not work on
# pure-ghcjs components.
] ++ (lib.optional (!isGhcjs) haskell-language-server)
) ++ (with nixpkgs; [
cabal2nix
niv
]);
in {
# merge this section into the attribute set with `client` and `server`.
ghcShell = nixpkgs.haskell.packages."${compiler}".shellFor {
packages = p: [ p.Common1 p.Server1 ];
withHoogle = true;
buildInputs = devtools { isGhcjs = false; };
}; };
ghcjsShell = nixpkgs-stable.haskell.packages."${ghcjs}".shellFor {
packages = p: [ p.Common1 p.Client1 ];
withHoogle = true;
buildInputs = devtools { isGhcjs = true; };
}; };
in { }
ghcShell = workOnMulti {
envPackages = [
"Common1"
"Common2"
"Server"
];
env = with nixpkgs.haskell.packages."${ghcVer}"; {
# enable hoogle in the environment
ghc = ghc.override {
override = self: super: {
withPackages = super.ghc.withHoogle;
};
};
inherit Common1 Common2 Server mkDerivation;
};
};
}
``` ```
{% end %}
Then, you can use `nix-shell` with this attribute: `nix-shell -A ghcShell`. Then, you can use `nix-shell` with this attribute: `nix-shell -A ghcShell`.
Build with Cabal as usual (`cabal new-build all`), assuming you've built the Build with Cabal as usual (`cabal new-build all`), assuming you've built the
@ -163,6 +246,12 @@ GHCJS parts already (see below).
## GHCJS ## GHCJS
Create an overlay the same way as with the server side components, but using
`nixpkgs.haskell.packages."${ghcjs}"` instead of `${compiler}`, and including
the client and common packages.
On GHCJS specifically, there is a hack that is very useful to apply:
GHCJS breaks many unit tests such that they freeze the Nix build process. You GHCJS breaks many unit tests such that they freeze the Nix build process. You
can override `mkDerivation` to disable most packages' unit tests. For some, can override `mkDerivation` to disable most packages' unit tests. For some,
this does not work because nixpkgs puts test runs in a conditional already, this does not work because nixpkgs puts test runs in a conditional already,
@ -171,7 +260,8 @@ which causes the `mkDerivation` override to be ignored.
can be used to deal with these cases. can be used to deal with these cases.
```nix ```nix
# inside the config.packageOverrides.haskell.packages.${compiler}.override call # inside the overlay for the client (at the same level as the package
# definitions)
mkDerivation = args: super.mkDerivation (args // { mkDerivation = args: super.mkDerivation (args // {
doCheck = false; doCheck = false;
enableLibraryProfiling = false; enableLibraryProfiling = false;
@ -185,32 +275,12 @@ is used. Namely, `nix-build -o client-out -A client` is used to build the
client and put a symbolic link in a known place, then manually created symbolic links are client and put a symbolic link in a known place, then manually created symbolic links are
placed in the static folder pointing back into this client output link. placed in the static folder pointing back into this client output link.
For package builds, a [`preConfigure` script](https://github.com/lf-/Carnap/blob/cde2671/server.nix#L30-L36) For package builds, a [`preConfigure` script](https://github.com/Carnap/Carnap/blob/cde2671/server.nix#L30-L36)
is used with is used with
[`haskell.lib.overrideCabal`](https://github.com/NixOS/nixpkgs/blob/32c8e79/pkgs/development/haskell-modules/lib.nix#L11-L41) [`haskell.lib.overrideCabal`](https://github.com/NixOS/nixpkgs/blob/32c8e79/pkgs/development/haskell-modules/lib.nix#L11-L41)
to replace these links with paths in the Nix store for the browser JavaScript. to replace these links with paths in the Nix store for the browser JavaScript.
A dependency on the built JavaScript is also added so it gets pulled in. A dependency on the built JavaScript is also added so it gets pulled in.
## Custom dependencies
Larger projects have a higher likelihood of having dependencies on Hackage
packages that are not in nixpkgs, or absolutely need to be a specific version.
It's easy to integrate these into the nix project using `cabal2nix`:
```
$ cabal2nix cabal://your-package-0.1.0.0 | tee nix/your-package.nix
```
These can then be integrated into the project by using
[`lib.callPackage`](https://github.com/NixOS/nixpkgs/blob/b63f684/lib/customisation.nix#L96-L121).
While it is also possible to use
[`callCabal2nix`](https://github.com/NixOS/nixpkgs/blob/f5b6ea1/pkgs/development/haskell-modules/make-package-set.nix#L200-L216),
I choose not to for reasons of initial build performance and reproducibility:
`cabal2nix` is not fast, and inadvertent updates could happen when updates are
made on the Hackage side, whereas checking in `cabal2nix` output ensures that
exactly the same package is used.
## Final thoughts ## Final thoughts
This project was very stimulating and challenging, and I learned a lot about This project was very stimulating and challenging, and I learned a lot about