372 lines
12 KiB
HTML
372 lines
12 KiB
HTML
<!doctype html>
|
|
<html>
|
|
<head>
|
|
<meta charset="utf-8">
|
|
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no">
|
|
|
|
<title>reveal.js</title>
|
|
|
|
<link rel="stylesheet" href="dist/reset.css">
|
|
<link rel="stylesheet" href="dist/reveal.css">
|
|
<link rel="stylesheet" href="dist/theme/black.css">
|
|
<link rel="stylesheet" href="styles.css">
|
|
|
|
<!-- Theme used for syntax highlighted code -->
|
|
<link rel="stylesheet" href="plugin/highlight/monokai.css">
|
|
</head>
|
|
<body>
|
|
<div class="reveal">
|
|
<div class="slides">
|
|
<section class="title-slide">
|
|
<h1> Debugging closure sizes graphically </h1>
|
|
<div class="meta">
|
|
<p>
|
|
Jade Lovelace <jadel@mercury.com>
|
|
</p>
|
|
<p>
|
|
NixCon 2022<br>
|
|
October 20, 2022<br>
|
|
https://jade.fyi
|
|
</p>
|
|
<p>
|
|
Slides here: <a href="https://jade.fyi/nixcon2022/slides">https://jade.fyi/nixcon2022/slides</a>
|
|
</p>
|
|
</div>
|
|
</section>
|
|
|
|
<section data-markdown><textarea data-template>
|
|
# who am I?
|
|
* Computer Engineering student at UBC in Vancouver, BC, Canada
|
|
* Presently interning in backend dev for Mercury, a company offering banking<fnref>1</fnref> designed for the unique needs of startups
|
|
* Haskell + PostgreSQL + TypeScript + Nix stack
|
|
* Working on internal tools for risk management
|
|
|
|
<div class="footnotes">
|
|
<ol>
|
|
<li>Mercury is a financial technology company, not a bank. Banking services provided by Choice Financial Group and Evolve Bank & Trust, Members FDIC.</li>
|
|
</ol>
|
|
</div>
|
|
</textarea></section>
|
|
|
|
|
|
<section data-markdown><textarea data-template>
|
|
# disclaimer
|
|
|
|
Views represented in this talk are my own and do not necessarily represent the views of Mercury.
|
|
</textarea></section>
|
|
|
|
<section data-markdown><textarea data-template>
|
|
# what's closure size?
|
|
|
|
* Size of some derivation and everything it depends on
|
|
* Accidental dependencies are the major contributor
|
|
* Other systems don't have this problem: forgetting about runtime dependencies may silently fail (or work if you have them installed!) at runtime
|
|
</textarea></section>
|
|
|
|
<section data-markdown><textarea data-template>
|
|
# how do I create a dependency?
|
|
</textarea></section>
|
|
|
|
<section data-markdown><textarea data-template>
|
|
<img src="./img/fsm-drv-1.svg" alt="silly drawing of mkDerivation with a build input of a drawing of a spaghetti monster">
|
|
</textarea></section>
|
|
|
|
<section data-markdown><textarea data-template>
|
|
<img src="./img/fsm-drv-2.svg" alt="continuation of the previous slide, with a buildPhase echoing spaghetti monster into $out">
|
|
</textarea></section>
|
|
|
|
<section data-markdown><textarea data-template>
|
|
<img src="./img/fsm-runtime-dep.svg" alt="little graph showing myapp-0.0.0 depending on spaghetti monster">
|
|
</textarea></section>
|
|
|
|
<section data-markdown><textarea data-template>
|
|
# issues with excess closure size
|
|
<img src="./img/fsm-docker.svg" alt="little container ship with a container mostly containing spaghetti monster and insignificantly containing myapp">
|
|
</textarea></section>
|
|
|
|
<!--
|
|
<section data-markdown><textarea data-template>
|
|
# digression: "what went in?"
|
|
|
|
Nix strings are magic: they keep track of the derivations that they've referenced as "string context", and propagate that information into any derivations they land in.
|
|
</textarea></section>
|
|
-->
|
|
|
|
<section data-markdown><textarea data-template>
|
|
# what does my stuff depend on?
|
|
|
|
<pre><code class="text"><script type="text/template">~ » nix path-info -rsSh nixpkgs#hello
|
|
/nix/store/c8aj6kzv5h7c5vympiy7l1p3aw626yqy-libobjc-11.0.0 199.7K 199.7K
|
|
/nix/store/y5cp9q9h6p80mzbsijs00zxpi7g0lc9d-apple-framework-CoreFoundation-11.0.0 667.1K 866.8K
|
|
/nix/store/xbqj64vdr3z13nlf8cvl1lf5lxa16mha-hello-2.12.1 126.9K 993.7K
|
|
</script></code></pre>
|
|
</textarea></section>
|
|
|
|
<section data-markdown><textarea data-template>
|
|
# that's a graph though
|
|
|
|
Yeah. I can't tell what the relationship is between these.
|
|
|
|
No tree or graph mode for `nix path-info` unlike `nix-store --query`
|
|
</textarea></section>
|
|
|
|
<section data-markdown><textarea data-template>
|
|
# what if..? haha, jk, unless?
|
|
|
|
There is a JSON output mode, what if we just used that? and then did some stuff to it
|
|
|
|
So I wrote a program in jq:
|
|
|
|
<pre><code class="text">nix-closure-graph nixpkgs#python3 > python3Closure.svg</code></pre>
|
|
</textarea></section>
|
|
|
|
<section data-markdown><textarea data-template>
|
|
# cool
|
|
|
|
<div class="img-container">
|
|
<img src="./img/python3Closure.svg" style="object-fit: cover; object-position: center;" width="100%" height="100%" alt="graphviz graph showing nodes for each of the packages Python depends on">
|
|
</div>
|
|
|
|
</textarea></section>
|
|
|
|
<section data-markdown><textarea data-template>
|
|
# what about a nixos image
|
|
|
|
maybe not
|
|
|
|
<div class="img-container">
|
|
<img src="./img/nixosClosure.png" style="object-fit: cover; object-position: center;" width="700px" height="400px" alt="incomprehensible graphviz graph absolutely covered in lines for a nixos image">
|
|
</div>
|
|
|
|
</textarea></section>
|
|
|
|
<section data-markdown><textarea data-template>
|
|
# well that didn't work
|
|
|
|
I built this graph viewer prototype at work, I wonder if I could stick a NixOS closure into it
|
|
|
|
https://mercurytechnologies.github.io/looking-glass-viewer/
|
|
|
|
<pre><code class="text"><script type="text/template"> » nix-closure-graph --lg .#nixosConfigurations.micro.config.system.build.toplevel | goo copy
|
|
</script></code></pre>
|
|
|
|
</textarea></section>
|
|
|
|
<section data-markdown><textarea data-template>
|
|
# it doesn't fall over?
|
|
|
|
The rendering has some room for improvement but it is usable!
|
|
|
|
<img class="r-stretch" src="img/nixos-closure.png" alt="Looking glass showing an extremely busy, perhaps unusable view on a NixOS closure">
|
|
|
|
</textarea></section>
|
|
|
|
<section data-markdown><textarea data-template>
|
|
# incidentally, why is half of hackage on there?
|
|
|
|
GHC puts dependency paths into executables for fun sometimes.<fnref>1</fnref>
|
|
|
|
<pre><code class="text"><script type="text/template"> » nix why-depends --precise /nix/store/abcde-hsutils-0.0.0 /nix/store/defgh-semigroupoids-5.3.7
|
|
|
|
/nix/store/abcde-hsutils-0.0.0
|
|
└───lib/ghc-9.2.4/x86_64-linux-ghc-9.2.4/libHShsutils-0.0.0-aaaa-ghc9.2.4.so: …6_64-linux-ghc-9.2.4:/nix/store/defgh-semigroupoids-5.3.7/lib/ghc-9.2…
|
|
→ /nix/store/defgh-semigroupoids-5.3.7
|
|
</script></code></pre>
|
|
|
|
<div class="footnotes">
|
|
<ol>
|
|
<li>I've <a href="https://github.com/Carnap/Carnap/blob/5e82366fd7d8804566558de189d59f40a704b1a8/server.nix#L86-L98">fixed this before</a></li>
|
|
</ol>
|
|
</div>
|
|
|
|
</textarea></section>
|
|
|
|
<section data-markdown><textarea data-template>
|
|
# why would haskell put that path in there?!
|
|
|
|
Unfortunately a known issue; help wanted.
|
|
|
|
<img src="img/github-issues.png" alt="screenshot of the github issues page for nixpkgs, showing a bug about large closures">
|
|
|
|
</textarea></section>
|
|
|
|
<section data-markdown><textarea data-template>
|
|
# practical closure size problems
|
|
|
|
I was packaging [actual-budget], a budgeting program, to run it on fly.io. So I
|
|
figured out how to build a docker image for it with Nix.
|
|
|
|
[actual-budget]: https://actualbudget.com/
|
|
|
|
</textarea></section>
|
|
|
|
<section data-markdown><textarea data-template>
|
|
# and then it's way too big
|
|
|
|
<pre><code class="text"><script type="text/template"> » nix build .#dockerImage.aarch64-linux
|
|
» ls -lah $(readlink result)
|
|
-r--r--r-- 1 root root 219M Dec 31 1969 /nix/store/6r0nslg23w2sa6a2zril3g0fvd6mnp1q-actual-server.tar.gz
|
|
</script></code></pre>
|
|
|
|
200MB compressed is not good. It's just some JavaScript, what happened there?
|
|
</textarea></section>
|
|
|
|
<section data-markdown><textarea data-template>
|
|
# let's bring out the big guns
|
|
|
|
```text
|
|
nix-closure-graph --lg .# | goo copy
|
|
```
|
|
|
|
</textarea></section>
|
|
|
|
<section data-markdown><textarea data-template>
|
|
# what went wrong here?
|
|
|
|
<img class="r-stretch" src="./img/actual-stage0.png" alt="screenshot of the graph viewer showing that actual-server depends on actual-server-modules, a derivation of suspiciously similar size">
|
|
|
|
<!-- FIXME: embed the graph viewer -->
|
|
|
|
</textarea></section>
|
|
|
|
<section data-markdown><textarea data-template>
|
|
# python?
|
|
|
|
<img class="r-stretch" src="./img/actual-stage0-python.png" alt="screenshot of the graph viewer showing that nodejs depends on python">
|
|
|
|
<!-- FIXME: embed the graph viewer -->
|
|
|
|
</textarea></section>
|
|
|
|
<section data-markdown><textarea data-template>
|
|
# what went wrong here?
|
|
|
|
* Python
|
|
* Mysteriously includes the app twice
|
|
|
|
</textarea></section>
|
|
|
|
<section data-markdown><textarea data-template>
|
|
# python?!
|
|
|
|
* Node has an obsolescent build system "Generate Your Projects" for
|
|
C++ extensions that generates makefiles
|
|
* Originally inherited from Chromium
|
|
|
|
</textarea></section>
|
|
|
|
<section data-markdown><textarea data-template>
|
|
# Generaten't My Projects
|
|
|
|
nodejs-slim to the rescue!
|
|
|
|
</textarea></section>
|
|
|
|
<section data-markdown><textarea data-template>
|
|
# why is the app there twice?
|
|
|
|
<pre><code class="text"><script type="text/template">» nix why-depends -a --precise \
|
|
/nix/store/qq1zblyp6ysx-actual-server \
|
|
/nix/store/i778ag8s7jf8-actual-sync-modules-1.0.1
|
|
|
|
/nix/store/qq1zblyp6ysx96vd8havhw0wcsihcidf-actual-server
|
|
└───libexec/actual-sync/deps/actual-sync/node_modules
|
|
-> /nix/store/i778ag8s7jf8b-actual-sync-mo>
|
|
→ /nix/store/i778ag8s7jf8-actual-sync-modules-1.0.1
|
|
</script>
|
|
</code></pre>
|
|
</textarea></section>
|
|
|
|
<section data-markdown><textarea data-template>
|
|
# what's in there anyway?
|
|
|
|
<pre><code class="text"><script type="text/template"> » ls -a result/libexec/actual-sync/deps/actual-sync/node_modules
|
|
. .. .bin
|
|
» ls -a result/libexec/actual-sync/deps/actual-sync/node_modules/.bin
|
|
. .. eslint prettier tsc tsserver uuid
|
|
</script>
|
|
</code></pre>
|
|
</textarea></section>
|
|
|
|
<section data-markdown><textarea data-template>
|
|
# yeah I don't know why the builder did that
|
|
|
|
<img class="r-stretch" src="img/lets-see-who-this-really-is.png" alt="Scooby Doo 'let's see who this is anyway' meme, unmasking 'closure size' to be 'totally pointless stuff'">
|
|
|
|
</textarea></section>
|
|
|
|
<section data-markdown><textarea data-template>
|
|
# actually fixing it
|
|
|
|
To recap, our goals are:
|
|
1. Getting rid of Python via nodejs-slim
|
|
2. Fixing the app being included twice
|
|
|
|
</textarea></section>
|
|
|
|
<section data-markdown><textarea data-template>
|
|
<video class="r-stretch" src="img/livedemo.mp4" controls>
|
|
</textarea></section>
|
|
|
|
<section data-markdown><textarea data-template>
|
|
# summary
|
|
|
|
* Accidental dependencies cause closure size bloat
|
|
* Shipping suspiciously sentient spaghetti slurps time and bandwidth
|
|
* Fix closure size issues by deleting references
|
|
</textarea></section>
|
|
|
|
<section data-markdown><textarea data-template>
|
|
# Mercury is hiring!
|
|
|
|
* Frontend
|
|
* Backend<fnref>1</fnref>
|
|
* Full stack
|
|
* Design
|
|
* Data engineering
|
|
* Data science
|
|
* Engineering management
|
|
|
|
See open positions at https://mercury.com/jobs
|
|
<div class="footnotes">
|
|
<ol><li>No Haskell experience required or expected; training is provided!</li></ol>
|
|
</div>
|
|
</textarea></section>
|
|
|
|
<section data-markdown><textarea data-template>
|
|
# fin
|
|
|
|
Tools:
|
|
- nix-closure-graph: https://github.com/lf-/dotfiles/tree/main/programs/nix-closure-graph
|
|
- Graph viewer: https://mercurytechnologies.github.io/looking-glass-viewer/
|
|
|
|
[@leftpaddotpy on Twitter](https://twitter.com/leftpaddotpy)
|
|
|
|
<https://jade.fyi>
|
|
|
|
my name [at] jade [dot] fyi
|
|
</textarea></section>
|
|
|
|
</div>
|
|
</div>
|
|
|
|
<script src="dist/reveal.js"></script>
|
|
<script src="plugin/notes/notes.js"></script>
|
|
<script src="plugin/markdown/markdown.js"></script>
|
|
<script src="plugin/highlight/highlight.js"></script>
|
|
<script>
|
|
// More info about initialization & config:
|
|
// - https://revealjs.com/initialization/
|
|
// - https://revealjs.com/config/
|
|
Reveal.initialize({
|
|
hash: true,
|
|
|
|
// Learn about plugins: https://revealjs.com/plugins/
|
|
plugins: [ RevealMarkdown, RevealHighlight, RevealNotes ],
|
|
transitionSpeed: 'fast',
|
|
transition: 'none',
|
|
slideNumber: true,
|
|
});
|
|
</script>
|
|
</body>
|
|
</html>
|