nixcon2022 talk, blogpost version

content/pages/nixcon2022/index.md

@@ -1,11 +0,0 @@
-+++
-title = "NixCon 2022"
-path = "/nixcon2022"
-[extra]
-isPage = true
-+++
-Hi!
-
-You've reached the NixCon 2022 page. Links:
-* [Slides](./slides)
-* [Video (livestream)](https://youtu.be/l70haNBm1wc?t=4h34m28s)

content/posts/nixcon2022-retrospective.md

@@ -0,0 +1,164 @@
++++
+date = "2022-10-23"
+draft = false
+path = "/blog/nixcon2022-retrospective"
+tags = ["nix", "haskell", "nixcon"]
+title = "NixCon 2022!!"
++++
+
+Well, that sure was a thing. It was more than a thing: it was a really good
+thing.
+
+NixCon 2022 was my first in-person conference, and therefore also my first time
+speaking at an in-person conference. I understand better why people want these.
+It's really nice to be around The People For The Thing and meet new people
+(whether they are wearing cat ears or otherwise).
+
+I met a whole *bunch* of people and made some new friends.
+
+I am not sure my feelings about the main venue, as it was really rather stuffy
+in there, and I mostly avoided it to reduce my exposure to COVID. However the
+rest of the college it was held at was pretty wonderful (modulo the lack of
+WiFi for people without eduroam access, although that was worked around).
+
+## Speaking
+
+I delivered a talk "Debugging Closure Size Graphically". This was a fun
+experience; it's a lot easier to deliver jokes in front of an audience. I'm
+excited to see what people do with the tools I built, and I'm glad they are
+open source now.
+
+In this talk, I show how I found issues with the closure size of Haskell and
+Node.JS programs, then how I fixed the Node.JS issues.
+
+Links:
+* [Livestream video][livestream] (a better recording with slides is coming soon)
+* [Slides]
+* Tools:
+    * nix-closure-graph: https://github.com/lf-/dotfiles/tree/main/programs/nix-closure-graph
+    * Graph viewer: https://mercurytechnologies.github.io/looking-glass-viewer/
+
+[livestream]: https://youtu.be/l70haNBm1wc?t=4h34m10s
+[Slides]: https://jade.fyi/nixcon2022/slides
+
+## The hacking room
+
+I spent most of the conference either in breakout rooms and later, the
+hackathon room. We got the following done:
+
+### Incremental builds for Haskell
+
+This was done after an extremely useful discussion with [Jonas Chevalier
+(@zimbatm)][Jonas Chevalier] and implemented in large part by [Harry Garrood].
+The idea is that we can achieve incremental builds of Haskell by doing an
+impure dependency on the previous build, intentionally breaking the evaluation
+chain.
+
+[Jonas Chevalier]: https://twitter.com/zimbatm
+[Harry Garrood]: https://twitter.com/hdgarrood
+
+In practice, this looks like a setting on the Haskell builder to enable
+a separate output for incremental information (which is the interface and
+object files of the run), and a second setting to copy in incremental
+information from some path.
+
+This approach [requires GHC 9.4][harry-blog] in order to use hash-based
+incremental information rather than entirely relying on file timestamps as
+was previously done.
+
+The reason this is fantastic is that it avoids the following false dichotomy of
+flawed options:
+  * Use one derivation per file. This would pessimize build times by having to
+    invoke GHC for each file as [ghc-nix] does, eating startup costs for each
+    file, and also losing finer grained incremental support than "did the
+    file change".
+
+    That is a completely valid and reasonable approach as it keeps the build
+    hermetic at the cost of more build time, but it relies heavily on
+    content-addressed derivations, and if you used it within Nix, it would
+    also use recursive Nix or [derivations-building-derivations].
+
+  * Change the project structure to use more packages so that Nix can
+    incrementally build at the package level.
+
+    This is hard to do, and [multi-package pessimizes the developer
+    experience][mp-fail] due to lack of multiple home units support as well
+    as slowing down builds in development (as opposed to Nix builds) since
+    Cabal/GHC do not yet know how to build the non-dependent parts of
+    dependent packages at the same time.
+
+    In my view, commercial projects probably should be one package as it
+    currently stands, because splitting packages is bad for development
+    efficiency, requiring duplicating dependency lists and doing a bunch of
+    other housekeeping, on top of the issues with concurrently working on
+    multiple packages described above.
+
+    Note that as in the link above, calling GHC once per file *does* allow
+    solving the dependent-package parallelism problem, as Bazel does, but it
+    also pessimizes build times.
+
+We decided to have our cake and eat it too. We prototyped an approach of
+letting GHC do the incremental builds in the way [Harry describes][harry-blog],
+and then convincing Nix to let us do it. This leaves dev completely alone, and
+constitutes only a minor impurity crime (since at least it is reproducible
+given some effort!), while not pessimizing compile times at all.
+
+[harry-blog]: https://harry.garrood.me/blog/easy-incremental-haskell-ci-builds-with-ghc-9.4/
+[ghc-nix]: https://github.com/matthewbauer/ghc-nix
+[derivations-building-derivations]: https://github.com/NixOS/rfcs/blob/master/rfcs/0092-plan-dynamism.md
+[mp-fail]: ./cabal-test-dev-trick
+
+### [nix-otel] grows up
+
+Another really nice achievement of the conference is that, with the help of
+[Linus Heckemann] and [Jean-François Roche], we have improved data quality in
+nix-otel and poked at making it work in the daemon, after which it could be
+integrated into Nix itself.
+
+Following discussions, it's likely that future structured logging will be
+OpenTelemetry based as it avoids reinventing the wheel.
+
+The data quality improvement is that previously, changing phases in the builder
+would not emit spans, which it now does. There is one span for each phase. This
+was achieved by effectively postprocessing the log data from Nix. Also, there
+was a memory corruption I fixed, which was somehow only causing missing data
+rather than crashes. Yikes.
+
+We also now report 100% of the information that Nix gives us via the logger,
+which means that further improvements will be in Nix itself, improving
+logging for everyone.
+
+Linus worked on getting nix-otel to work in the daemon, which is still a work
+in progress. We want to use settings for the API keys and endpoint to avoid
+needing environment variables for them. On that account, we arrived at hunting
+a bug in our settings handling in which the settings were not getting their
+values properly.
+
+Another thing that was discussed is making the daemon and client cooperate
+since the client knows some things the daemon doesn't, and the daemon has
+better timing information. This can be done pretty easily by having the client
+do propagation of trace IDs to the daemon, and keeping track of whether log
+data was forwarded (and in that case, not sending it to OpenTelemetry since it
+already was sent).
+
+Once this foundational work is done, nix-otel can be integrated into Nix
+itself, potentially representing the first Rust in Nix since the previous
+attempt that had significantly more complicated foreign-function-interface
+usage, leading to its failure.
+
+Looking forward, another thing we can probably do is to use the `$NIX_LOG_FD`
+infrastructure that already exists to build an OpenTelemetry
+propagator/exporter that can be used within builds to also instrument the
+inside of builds.
+
+[nix-otel]: https://github.com/lf-/nix-otel
+[Linus Heckemann]: https://twitter.com/linux_hackerman
+[Jean-François Roche]: https://github.com/jfroche
+
+## oops, feelings
+
+Paris is beautiful. I really appreciate having a practical transit system and
+being able to walk everywhere.
+
+Certainly this trip has renewed my motivation for looking into possibly moving
+to Europe in the next few years.

content/posts/nixcon2022/index.md

@@ -0,0 +1,354 @@
++++
+title = "NixCon 2022: Debugging Closure Sizes Graphically with nix-closure-graph"
+path = "/nixcon2022"
+draft = false
+tags = ["nix", "haskell", "nixcon", "javascript"]
+date = "2022-10-20"
++++
+
+I delivered a talk "Debugging Closure Size Graphically" at NixCon 2022. This
+blog post is intended as a textual version of the talk although it is not a
+script.
+
+Here are links to the live version:
+
+* [Livestream video][livestream] (a better recording with slides is coming soon)
+* [Slides]
+* Tools:
+    * nix-closure-graph: https://github.com/lf-/dotfiles/tree/main/programs/nix-closure-graph
+    * Graph viewer: https://mercurytechnologies.github.io/looking-glass-viewer/
+
+[nix-closure-graph]: https://github.com/lf-/dotfiles/tree/main/programs/nix-closure-graph
+[looking-glass]: https://mercurytechnologies.github.io/looking-glass-viewer/
+[looking-glass-github]: https://github.com/MercuryTechnologies/looking-glass-viewer
+[livestream]: https://youtu.be/l70haNBm1wc?t=4h34m10s
+[Slides]: https://jade.fyi/nixcon2022/slides
+
+## Closure size
+
+Nix's smallest unit of building is a so-called "derivation". Generally there is
+one of these per package or per part of a NixOS system. The derivation is a
+concept at the layer of the Nix store.
+
+Xe has said that [Nix is a compiler from Haskell to Bash][xe-why], which is a
+quite good if flippant explanation of how the Nix language relates to the build
+system: the "haskell" part is the Nix language, and the "bash" is the full
+invocation of some builder (typically written in bash) to actually build the
+thing, produced by `builtins.derivation`.
+
+You can see the content of a derivation created by a Nix expression like so:
+
+{% codesample(desc="`nix show-derivation nixpkgs#hello`") %}
+
+```json
+ » nix show-derivation nixpkgs#hello
+{
+  "/nix/store/vdqm71n5bkfydjwjwmx2rkj18jrpxlmx-hello-2.12.1.drv": {
+    "outputs": {
+      "out": {
+        "path": "/nix/store/m2wybrc3grfn76hxcv4gyiyncng17z30-hello-2.12.1"
+      }
+    },
+    "inputSrcs": [
+      "/nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh"
+    ],
+    "inputDrvs": {
+      "/nix/store/004b0bvpjng4l23kahn6vzawlpr6dx75-bash-5.1-p16.drv": [
+        "out"
+      ],
+      "/nix/store/gajvqygc937wsfjdkpfpqrrk3mlpyy6z-stdenv-linux.drv": [
+        "out"
+      ],
+      "/nix/store/j80mkmlf7xv7cqn3chh7ghi1r96406m2-hello-2.12.1.tar.gz.drv": [
+        "out"
+      ]
+    },
+    "system": "x86_64-linux",
+    "builder": "/nix/store/dd3713mm8wql4r2d5jxx0f58g16nfm4h-bash-5.1-p16/bin/bash",
+    "args": [
+      "-e",
+      "/nix/store/9krlzvny65gdc8s7kpb6lkx8cd02c25b-default-builder.sh"
+    ],
+    "env": {
+      "buildInputs": "",
+      "builder": "/nix/store/dd3713mm8wql4r2d5jxx0f58g16nfm4h-bash-5.1-p16/bin/bash",
+      "cmakeFlags": "",
+      "configureFlags": "",
+      "depsBuildBuild": "",
+      "depsBuildBuildPropagated": "",
+      "depsBuildTarget": "",
+      "depsBuildTargetPropagated": "",
+      "depsHostHost": "",
+      "depsHostHostPropagated": "",
+      "depsTargetTarget": "",
+      "depsTargetTargetPropagated": "",
+      "doCheck": "1",
+      "doInstallCheck": "",
+      "mesonFlags": "",
+      "name": "hello-2.12.1",
+      "nativeBuildInputs": "",
+      "out": "/nix/store/m2wybrc3grfn76hxcv4gyiyncng17z30-hello-2.12.1",
+      "outputs": "out",
+      "patches": "",
+      "pname": "hello",
+      "propagatedBuildInputs": "",
+      "propagatedNativeBuildInputs": "",
+      "src": "/nix/store/pa10z4ngm0g83kx9mssrqzz30s84vq7k-hello-2.12.1.tar.gz",
+      "stdenv": "/nix/store/vsw40d3nb6mpjf770iv58czisf65lzzl-stdenv-linux",
+      "strictDeps": "",
+      "system": "x86_64-linux",
+      "version": "2.12.1"
+    }
+  }
+}
+```
+
+{% end %}
+
+[xe-why]: https://twitter.com/leftpaddotpy/status/1578914185857372165
+
+The "closure" of some Nix store path, for instance, of the "out" output of the
+"hello" derivation above, refers to all the store paths referenced in that
+store path, and the store paths referenced in such references recursively. If
+you would like a mathy definition, see [Eelco's PhD thesis][phd-thesis], page
+96.
+
+[phd-thesis]: https://edolstra.github.io/pubs/phd-thesis.pdf
+
+Put another way, the closure is "your thing and everything it depends on,
+recursively".
+
+The way that runtime dependencies are created in Nix is that Nix will
+effectively `grep` for the hash part of any inputs of a derivation, which will
+form the runtime dependencies.
+
+This is unlike most other systems, which require explicit specification of
+runtime dependencies, potentially allowing forgetting some (which will work on
+the maintainer's machine that has the software but not on a fresh install). Nix
+has the opposite problem: it's easy to accidentally *create* runtime
+dependencies, since any reference to the build inputs in the outputs can create
+one.
+
+Closure sizes *largely* don't matter if you are building on the same system
+that you are running the software on, besides leaving some stuff around that
+the garbage collector can't delete due to pointless references. However, the
+entire closure needs to be shipped around whenever you are deploying Nix-built
+software to another machine, using up space (prod generally has less space than
+your laptop!) and bandwidth.
+
+### spaghetti monsters
+
+Let the spaghetti monster be a stand-in for your choice of large thing; perhaps
+a compiler, a build system, some interpreter you don't actually need at
+runtime, or anything else.
+
+If the spaghetti monster is in `buildInputs` of your derivation:
+
+![silly drawing of mkDerivation with a build input of a drawing of a spaghetti
+monster](slides/img/fsm-drv-1.svg)
+
+... *and* its store path appears in the output:
+
+![continuation of the previous image, with a buildPhase echoing spaghetti monster into $out](slides/img/fsm-drv-2.svg)
+
+then the derivation will depend on the spaghetti monster at runtime, and thus
+you will be transporting a big spaghetti monster along with your software,
+anywhere it goes (such as in a Docker image or a NixOS deployment):
+
+![little container ship with a container mostly containing spaghetti monster and insignificantly containing myapp](slides/img/fsm-docker.svg)
+
+## Figuring out what your software depends on
+
+You can use `nix path-info` to list the dependencies of a store path
+`-r`ecursively, with `-s`izes, and closure `-S`izes, in `-h`uman readable form.
+
+```text
+~ » nix path-info -rsSh nixpkgs#hello
+/nix/store/c8aj6kzv5h7c5vympiy7l1p3aw626yqy-libobjc-11.0.0                               199.7K  199.7K
+/nix/store/y5cp9q9h6p80mzbsijs00zxpi7g0lc9d-apple-framework-CoreFoundation-11.0.0        667.1K  866.8K
+/nix/store/xbqj64vdr3z13nlf8cvl1lf5lxa16mha-hello-2.12.1                                 126.9K  993.7K
+```
+
+This is, however, unhelpful, since this is actually structurally a graph, and
+this view does not make it obvious what depends on what.
+
+Nix actually can output GraphViz with `nix-store --query --graph` but it
+doesn't show sizes so it is unhelpful.
+
+### Just render a graph then!
+
+I wrote a program in `jq`, everyone's favourite [language that, like Nix, is
+also allegedly Haskell][xe-why], which generates GraphViz with good quality
+output. It can be invoked with [`nix-closure-graph nixpkgs#python3 >
+img.svg`][nix-closure-graph].
+
+This works pretty well for Python (open the image in a new tab if you want to
+look at it more closely):
+
+<img src="slides/img/python3Closure.svg" style="object-fit: cover; object-position: center;" width="100%" height="300px" alt="graphviz graph showing nodes for each of the packages Python depends on">
+
+But not so for NixOS closures:
+
+<img src="slides/img/nixosClosure.png" style="object-fit: cover; object-position: center;" width="700px" height="400px" alt="incomprehensible graphviz graph absolutely covered in lines for a nixos image">
+
+
+### Just render a graph, but not with GraphViz
+
+Writing a better graph viewer sounds hard, but it turns out
+[Sigma.js](https://www.sigmajs.org/) already did the hard part of writing a
+graph renderer that does well with big graphs, and I had already built a
+Sigma-based graph viewer as a prototype at work as part of my internship
+project.
+
+I believe that it's not practical to show large graphs non-interactively: since
+GraphViz contains the results of several papers on graph layout and it is slow
+while also having poor output for large graphs, it follows that the only
+practical option is to do something worse and faster for layout and make up for
+it with interactivity. Sigma seems to be one of the better options for building
+a viewer that does that.
+
+[Looking Glass][looking-glass], my graph viewer, consists of a simple JSON
+format (intended to be simple enough you can generate it directly out of
+PostgreSQL) and a properties panel to list arbitrary data of nodes.
+
+It's [open source][looking-glass-github] under the MIT license, if you want to
+hack on it.
+
+![Looking glass showing an extremely busy, yet somewhat usable view of a NixOS closure](./slides/img/nixos-closure.png)
+
+To use it, you can do a command like `nix-closure-graph --lg nixpkgs#python3 |
+xsel -b` (pick your store path and clipboard tool of choice), then paste the
+JSON into the box in the viewer.
+
+[Here is the JSON from my NixOS closure, for example](./slides/graphs/nixos.json)
+
+#### Why is GHC in my NixOS closure?
+
+This is possibly a known issue, or possibly fixed in newer GHC, or possibly me
+forgetting to use `pkgs.haskell.lib.justStaticExecutables`. Regardless, it's
+not on purpose.
+
+## Practical closure size problems
+
+I was packaging [actual-budget], a budgeting program, to run it on fly.io. So I
+figured out how to build a docker image for it with Nix. You can get the
+[final version here](https://github.com/lf-/actual-server/tree/flake).
+
+[actual-budget]: https://actualbudget.com/
+
+However, I had a problem:
+
+```text
+ » ls -lah $(readlink result)
+-r--r--r-- 1 root root 219M Dec 31  1969 /nix/store/6r0nslg23w2sa6a2zril3g0fvd6mnp1q-actual-server.tar.gz
+```
+
+200MB compressed is not good. It's just some JavaScript, what happened there?
+
+![screenshot of the graph viewer showing that actual-server depends on actual-server-modules, a store path of suspiciously similar size](./slides/img/actual-stage0.png)
+
+It appears that for some reason `actual-server` and `actual-server-modules` are
+both there, which is not quite right.
+
+Also, it appears that Python got in there somehow, which is not necessary:
+
+![screenshot of the graph viewer showing that nodejs depends on
+python](./slides/img/actual-stage0-python.png)
+
+### What went wrong?
+
+Python is in there because of an exceedingly poorly named legacy build system
+inherited from Chromium called "Generate Your Projects", which compiles weird
+Python into Makefiles.
+
+Unfortunately Actual uses some native extensions, but *after* building those
+with "Generate Your Projects", there's no reason at all to have Python around.
+
+It turns out, nixpkgs ships a version of NodeJS that does not include Python
+or npm, called [`nodejs-slim`][nodejs-slim].
+
+[nodejs-slim]: https://github.com/nixos/nixpkgs/blob/4567b99d17d3e0f96e0022c18f92a08a03e91f3e/pkgs/top-level/all-packages.nix#L8586-L8588
+
+The app is there twice because of a symlink:
+
+```text
+» nix why-depends -a --precise \
+    /nix/store/qq1zblyp6ysx-actual-server \
+    /nix/store/i778ag8s7jf8-actual-sync-modules-1.0.1
+
+/nix/store/qq1zblyp6ysx96vd8havhw0wcsihcidf-actual-server
+└───libexec/actual-sync/deps/actual-sync/node_modules
+        -> /nix/store/i778ag8s7jf8b-actual-sync-mo>
+    → /nix/store/i778ag8s7jf8-actual-sync-modules-1.0.1
+```
+
+However, if we look at what's actually in that symlink, we will find nothing
+necessary at all.
+
+```text
+» ls -a result/libexec/actual-sync/deps/actual-sync/node_modules
+.  ..  .bin
+ » ls -a result/libexec/actual-sync/deps/actual-sync/node_modules/.bin
+.  ..  eslint  prettier  tsc  tsserver  uuid
+```
+
+![Scooby Doo 'let's see who this is anyway' meme, unmasking 'closure size' to
+be 'totally pointless stuff'](./slides/img/lets-see-who-this-really-is.png)
+
+### Fixing it
+
+The solution to Python being there is to remove all references to `nodejs` and
+replace the relevant ones with `nodejs-slim`. This was easiest to do by just
+doing a silly find command to delete the ones in unimportant scripts, then
+manually patching the shebang of the entry point script using `sed`.
+
+The app being included twice was fixed by deleting the symlink and replacing it
+with a new symlink to the right relative path (since apparently it was not only
+a closure size problem but also wrong).
+
+It's a good idea to put the package you want to exclude in
+[`disallowedRequisites`](https://nixos.org/manual/nix/stable/language/advanced-attributes.html#adv-attr-disallowedRequisites)
+or `disallowedReferences` so that Nix will fail the build if it appears again.
+
+All in all it looks like the following: ([original
+source](https://github.com/lf-/actual-server/blob/flake/flake.nix#L40-L51))
+
+```nix
+prev.mkYarnPackage rec {
+    # ...
+
+    # we don't need to have the full-fat nodejs with python
+    # (for gyp) and so on except to build. It is undesirably
+    # referenced by binaries in dependencies, and also would be
+    # patchShebang'd into bin/actual-server as well if we
+    # didn't disable that and do it manually.
+    dontPatchShebangs = true;
+    extraBuildInputs = [ final.removeReferencesTo ];
+    disallowedRequisites = [ final.nodejs-16_x ];
+
+    distPhase = ''
+      # redundant symlink that introduces a 150mb runtime dep
+      # on the actual-server-modules derivation
+      rm $out/libexec/actual-sync/deps/actual-sync/node_modules
+      # .. and replace it with a relative symlink inside the
+      # package so the server can find its web files
+      ln -s $out/libexec/actual-sync/node_modules $out/libexec/actual-sync/deps/actual-sync/node_modules
+
+      # manually patchelf actual-server
+      sed -i '1c #!${final.nodejs-slim-16_x}/bin/node' "$(readlink -f "$out/bin/actual-server")"
+
+      # break unnecessary dependency binaries
+      find "$out" -type f -exec remove-references-to -t ${final.nodejs-16_x} '{}' +
+    '';
+}
+```
+
+## Conclusion
+
+Don't ship around spaghetti monsters if it can be avoided: closure size matters
+whenever you are shipping around Nix build output.
+
+Most often, closure size is down to accidentally depending on build tools and
+other packages, and it can be solved case-by-case by adding fixup scripts to
+remove pointless references.
+