# Debugging closure sizes graphically

Jade Lovelace <jadel@mercury.com>

NixCon 2022
October 20, 2022
https://jade.fyi

Slides here: https://jade.fyi/nixcon2022/slides

# who am I? * Computer Engineering student at UBC in Vancouver, BC, Canada * Presently interning in backend dev for Mercury, a company offering banking<fnref>1</fnref> designed for the unique needs of startups * Haskell + PostgreSQL + TypeScript + Nix stack * Working on internal tools for risk management <div class="footnotes"> <ol> <li>Mercury is a financial technology company, not a bank. Banking services provided by Choice Financial Group and Evolve Bank & Trust, Members FDIC.</li> </ol> </div>
# disclaimer Views represented in this talk are my own and do not necessarily represent the views of Mercury.
# what's closure size? * Size of some derivation and everything it depends on * Accidental dependencies are the major contributor * Other systems don't have this problem: forgetting about runtime dependencies may silently fail (or work if you have them installed!) at runtime
# how do I create a dependency?
<img src="./img/fsm-drv-1.svg" alt="silly drawing of mkDerivation with a build input of a drawing of a spaghetti monster">
<img src="./img/fsm-drv-2.svg" alt="continuation of the previous slide, with a buildPhase echoing spaghetti monster into $out">
<img src="./img/fsm-runtime-dep.svg" alt="little graph showing myapp-0.0.0 depending on spaghetti monster">
# issues with excess closure size <img src="./img/fsm-docker.svg" alt="little container ship with a container mostly containing spaghetti monster and insignificantly containing myapp">
# what does my stuff depend on? <pre><code class="text"><script type="text/template">~ » nix path-info -rsSh nixpkgs#hello /nix/store/c8aj6kzv5h7c5vympiy7l1p3aw626yqy-libobjc-11.0.0 199.7K 199.7K /nix/store/y5cp9q9h6p80mzbsijs00zxpi7g0lc9d-apple-framework-CoreFoundation-11.0.0 667.1K 866.8K /nix/store/xbqj64vdr3z13nlf8cvl1lf5lxa16mha-hello-2.12.1 126.9K 993.7K </script></code></pre>
# that's a graph though Yeah. I can't tell what the relationship is between these. No tree or graph mode for `nix path-info` unlike `nix-store --query`
# what if..? haha, jk, unless? There is a JSON output mode, what if we just used that? and then did some stuff to it So I wrote a program in jq: <pre><code class="text">nix-closure-graph nixpkgs#python3 > python3Closure.svg</code></pre>
# cool <div class="img-container"> <img src="./img/python3Closure.svg" style="object-fit: cover; object-position: center;" width="100%" height="100%" alt="graphviz graph showing nodes for each of the packages Python depends on"> </div>
# what about a nixos image maybe not <div class="img-container"> <img src="./img/nixosClosure.png" style="object-fit: cover; object-position: center;" width="700px" height="400px" alt="incomprehensible graphviz graph absolutely covered in lines for a nixos image"> </div>
# well that didn't work I built this graph viewer prototype at work, I wonder if I could stick a NixOS closure into it https://mercurytechnologies.github.io/looking-glass-viewer/ <pre><code class="text"><script type="text/template"> » nix-closure-graph --lg .#nixosConfigurations.micro.config.system.build.toplevel | goo copy </script></code></pre>
# it doesn't fall over? The rendering has some room for improvement but it is usable! <img class="r-stretch" src="img/nixos-closure.png" alt="Looking glass showing an extremely busy, perhaps unusable view on a NixOS closure">
# incidentally, why is half of hackage on there? GHC puts dependency paths into executables for fun sometimes.<fnref>1</fnref> <pre><code class="text"><script type="text/template"> » nix why-depends --precise /nix/store/abcde-hsutils-0.0.0 /nix/store/defgh-semigroupoids-5.3.7 /nix/store/abcde-hsutils-0.0.0 └───lib/ghc-9.2.4/x86_64-linux-ghc-9.2.4/libHShsutils-0.0.0-aaaa-ghc9.2.4.so: …6_64-linux-ghc-9.2.4:/nix/store/defgh-semigroupoids-5.3.7/lib/ghc-9.2… → /nix/store/defgh-semigroupoids-5.3.7 </script></code></pre> <div class="footnotes"> <ol> <li>I've <a href="https://github.com/Carnap/Carnap/blob/5e82366fd7d8804566558de189d59f40a704b1a8/server.nix#L86-L98">fixed this before</a></li> </ol> </div>
# why would haskell put that path in there?! Unfortunately a known issue; help wanted. <img src="img/github-issues.png" alt="screenshot of the github issues page for nixpkgs, showing a bug about large closures">
# practical closure size problems I was packaging [actual-budget], a budgeting program, to run it on fly.io. So I figured out how to build a docker image for it with Nix. [actual-budget]: https://actualbudget.com/
# and then it's way too big <pre><code class="text"><script type="text/template"> » nix build .#dockerImage.aarch64-linux » ls -lah $(readlink result) -r--r--r-- 1 root root 219M Dec 31 1969 /nix/store/6r0nslg23w2sa6a2zril3g0fvd6mnp1q-actual-server.tar.gz </script></code></pre> 200MB compressed is not good. It's just some JavaScript, what happened there?
# let's bring out the big guns ```text nix-closure-graph --lg .# | goo copy ```
# what went wrong here? <img class="r-stretch" src="./img/actual-stage0.png" alt="screenshot of the graph viewer showing that actual-server depends on actual-server-modules, a derivation of suspiciously similar size"> <!-- FIXME: embed the graph viewer -->
# python? <img class="r-stretch" src="./img/actual-stage0-python.png" alt="screenshot of the graph viewer showing that nodejs depends on python"> <!-- FIXME: embed the graph viewer -->
# what went wrong here? * Python * Mysteriously includes the app twice
# python?! * Node has an obsolescent build system "Generate Your Projects" for C++ extensions that generates makefiles * Originally inherited from Chromium
# Generaten't My Projects nodejs-slim to the rescue!
# why is the app there twice? <pre><code class="text"><script type="text/template">» nix why-depends -a --precise \ /nix/store/qq1zblyp6ysx-actual-server \ /nix/store/i778ag8s7jf8-actual-sync-modules-1.0.1 /nix/store/qq1zblyp6ysx96vd8havhw0wcsihcidf-actual-server └───libexec/actual-sync/deps/actual-sync/node_modules -> /nix/store/i778ag8s7jf8b-actual-sync-mo> → /nix/store/i778ag8s7jf8-actual-sync-modules-1.0.1 </script> </code></pre>
# what's in there anyway? <pre><code class="text"><script type="text/template"> » ls -a result/libexec/actual-sync/deps/actual-sync/node_modules . .. .bin » ls -a result/libexec/actual-sync/deps/actual-sync/node_modules/.bin . .. eslint prettier tsc tsserver uuid </script> </code></pre>
# yeah I don't know why the builder did that <img class="r-stretch" src="img/lets-see-who-this-really-is.png" alt="Scooby Doo 'let's see who this is anyway' meme, unmasking 'closure size' to be 'totally pointless stuff'">
# actually fixing it To recap, our goals are: 1. Getting rid of Python via nodejs-slim 2. Fixing the app being included twice
<video class="r-stretch" src="img/livedemo.mp4" controls>
# summary * Accidental dependencies cause closure size bloat * Shipping suspiciously sentient spaghetti slurps time and bandwidth * Fix closure size issues by deleting references
# Mercury is hiring! * Frontend * Backend<fnref>1</fnref> * Full stack * Design * Data engineering * Data science * Engineering management See open positions at https://mercury.com/jobs <div class="footnotes"> <ol><li>No Haskell experience required or expected; training is provided!</li></ol> </div>
# fin Tools: - nix-closure-graph: https://github.com/lf-/dotfiles/tree/main/programs/nix-closure-graph - Graph viewer: https://mercurytechnologies.github.io/looking-glass-viewer/ [@leftpaddotpy on Twitter](https://twitter.com/leftpaddotpy) <https://jade.fyi> my name [at] jade [dot] fyi