nixcon2022 talk, blogpost version
This commit is contained in:
Jade Lovelace
parent
f854ef75ca
commit
b333d2d023
131 changed files with 518 additions and 11 deletions
|
|
@ -1,372 +0,0 @@
|
|||
<!doctype html>
|
||||
<html>
|
||||
<head>
|
||||
<meta charset="utf-8">
|
||||
<meta name="viewport" content="width=device-width, initial-scale=1.0, maximum-scale=1.0, user-scalable=no">
|
||||
|
||||
<title>reveal.js</title>
|
||||
|
||||
<link rel="stylesheet" href="dist/reset.css">
|
||||
<link rel="stylesheet" href="dist/reveal.css">
|
||||
<link rel="stylesheet" href="dist/theme/black.css">
|
||||
<link rel="stylesheet" href="styles.css">
|
||||
|
||||
<!-- Theme used for syntax highlighted code -->
|
||||
<link rel="stylesheet" href="plugin/highlight/monokai.css">
|
||||
</head>
|
||||
<body>
|
||||
<div class="reveal">
|
||||
<div class="slides">
|
||||
<section class="title-slide">
|
||||
<h1> Debugging closure sizes graphically </h1>
|
||||
<div class="meta">
|
||||
<p>
|
||||
Jade Lovelace <jadel@mercury.com>
|
||||
</p>
|
||||
<p>
|
||||
NixCon 2022<br>
|
||||
October 20, 2022<br>
|
||||
https://jade.fyi
|
||||
</p>
|
||||
<p>
|
||||
Slides here: <a href="https://jade.fyi/nixcon2022/slides">https://jade.fyi/nixcon2022/slides</a>
|
||||
</p>
|
||||
</div>
|
||||
</section>
|
||||
|
||||
<section data-markdown><textarea data-template>
|
||||
# who am I?
|
||||
* Computer Engineering student at UBC in Vancouver, BC, Canada
|
||||
* Presently interning in backend dev for Mercury, a company offering banking<fnref>1</fnref> designed for the unique needs of startups
|
||||
* Haskell + PostgreSQL + TypeScript + Nix stack
|
||||
* Working on internal tools for risk management
|
||||
|
||||
<div class="footnotes">
|
||||
<ol>
|
||||
<li>Mercury is a financial technology company, not a bank. Banking services provided by Choice Financial Group and Evolve Bank & Trust, Members FDIC.</li>
|
||||
</ol>
|
||||
</div>
|
||||
</textarea></section>
|
||||
|
||||
|
||||
<section data-markdown><textarea data-template>
|
||||
# disclaimer
|
||||
|
||||
Views represented in this talk are my own and do not necessarily represent the views of Mercury.
|
||||
</textarea></section>
|
||||
|
||||
<section data-markdown><textarea data-template>
|
||||
# what's closure size?
|
||||
|
||||
* Size of some derivation and everything it depends on
|
||||
* Accidental dependencies are the major contributor
|
||||
* Other systems don't have this problem: forgetting about runtime dependencies may silently fail (or work if you have them installed!) at runtime
|
||||
</textarea></section>
|
||||
|
||||
<section data-markdown><textarea data-template>
|
||||
# how do I create a dependency?
|
||||
</textarea></section>
|
||||
|
||||
<section data-markdown><textarea data-template>
|
||||
<img src="./img/fsm-drv-1.svg" alt="silly drawing of mkDerivation with a build input of a drawing of a spaghetti monster">
|
||||
</textarea></section>
|
||||
|
||||
<section data-markdown><textarea data-template>
|
||||
<img src="./img/fsm-drv-2.svg" alt="continuation of the previous slide, with a buildPhase echoing spaghetti monster into $out">
|
||||
</textarea></section>
|
||||
|
||||
<section data-markdown><textarea data-template>
|
||||
<img src="./img/fsm-runtime-dep.svg" alt="little graph showing myapp-0.0.0 depending on spaghetti monster">
|
||||
</textarea></section>
|
||||
|
||||
<section data-markdown><textarea data-template>
|
||||
# issues with excess closure size
|
||||
<img src="./img/fsm-docker.svg" alt="little container ship with a container mostly containing spaghetti monster and insignificantly containing myapp">
|
||||
</textarea></section>
|
||||
|
||||
<!--
|
||||
<section data-markdown><textarea data-template>
|
||||
# digression: "what went in?"
|
||||
|
||||
Nix strings are magic: they keep track of the derivations that they've referenced as "string context", and propagate that information into any derivations they land in.
|
||||
</textarea></section>
|
||||
-->
|
||||
|
||||
<section data-markdown><textarea data-template>
|
||||
# what does my stuff depend on?
|
||||
|
||||
<pre><code class="text"><script type="text/template">~ » nix path-info -rsSh nixpkgs#hello
|
||||
/nix/store/c8aj6kzv5h7c5vympiy7l1p3aw626yqy-libobjc-11.0.0 199.7K 199.7K
|
||||
/nix/store/y5cp9q9h6p80mzbsijs00zxpi7g0lc9d-apple-framework-CoreFoundation-11.0.0 667.1K 866.8K
|
||||
/nix/store/xbqj64vdr3z13nlf8cvl1lf5lxa16mha-hello-2.12.1 126.9K 993.7K
|
||||
</script></code></pre>
|
||||
</textarea></section>
|
||||
|
||||
<section data-markdown><textarea data-template>
|
||||
# that's a graph though
|
||||
|
||||
Yeah. I can't tell what the relationship is between these.
|
||||
|
||||
No tree or graph mode for `nix path-info` unlike `nix-store --query`
|
||||
</textarea></section>
|
||||
|
||||
<section data-markdown><textarea data-template>
|
||||
# what if..? haha, jk, unless?
|
||||
|
||||
There is a JSON output mode, what if we just used that? and then did some stuff to it
|
||||
|
||||
So I wrote a program in jq:
|
||||
|
||||
<pre><code class="text">nix-closure-graph nixpkgs#python3 > python3Closure.svg</code></pre>
|
||||
</textarea></section>
|
||||
|
||||
<section data-markdown><textarea data-template>
|
||||
# cool
|
||||
|
||||
<div class="img-container">
|
||||
<img src="./img/python3Closure.svg" style="object-fit: cover; object-position: center;" width="100%" height="100%" alt="graphviz graph showing nodes for each of the packages Python depends on">
|
||||
</div>
|
||||
|
||||
</textarea></section>
|
||||
|
||||
<section data-markdown><textarea data-template>
|
||||
# what about a nixos image
|
||||
|
||||
maybe not
|
||||
|
||||
<div class="img-container">
|
||||
<img src="./img/nixosClosure.png" style="object-fit: cover; object-position: center;" width="700px" height="400px" alt="incomprehensible graphviz graph absolutely covered in lines for a nixos image">
|
||||
</div>
|
||||
|
||||
</textarea></section>
|
||||
|
||||
<section data-markdown><textarea data-template>
|
||||
# well that didn't work
|
||||
|
||||
I built this graph viewer prototype at work, I wonder if I could stick a NixOS closure into it
|
||||
|
||||
https://mercurytechnologies.github.io/looking-glass-viewer/
|
||||
|
||||
<pre><code class="text"><script type="text/template"> » nix-closure-graph --lg .#nixosConfigurations.micro.config.system.build.toplevel | goo copy
|
||||
</script></code></pre>
|
||||
|
||||
</textarea></section>
|
||||
|
||||
<section data-markdown><textarea data-template>
|
||||
# it doesn't fall over?
|
||||
|
||||
The rendering has some room for improvement but it is usable!
|
||||
|
||||
<img class="r-stretch" src="img/nixos-closure.png" alt="Looking glass showing an extremely busy, perhaps unusable view on a NixOS closure">
|
||||
|
||||
</textarea></section>
|
||||
|
||||
<section data-markdown><textarea data-template>
|
||||
# incidentally, why is half of hackage on there?
|
||||
|
||||
GHC puts dependency paths into executables for fun sometimes.<fnref>1</fnref>
|
||||
|
||||
<pre><code class="text"><script type="text/template"> » nix why-depends --precise /nix/store/abcde-hsutils-0.0.0 /nix/store/defgh-semigroupoids-5.3.7
|
||||
|
||||
/nix/store/abcde-hsutils-0.0.0
|
||||
└───lib/ghc-9.2.4/x86_64-linux-ghc-9.2.4/libHShsutils-0.0.0-aaaa-ghc9.2.4.so: …6_64-linux-ghc-9.2.4:/nix/store/defgh-semigroupoids-5.3.7/lib/ghc-9.2…
|
||||
→ /nix/store/defgh-semigroupoids-5.3.7
|
||||
</script></code></pre>
|
||||
|
||||
<div class="footnotes">
|
||||
<ol>
|
||||
<li>I've <a href="https://github.com/Carnap/Carnap/blob/5e82366fd7d8804566558de189d59f40a704b1a8/server.nix#L86-L98">fixed this before</a></li>
|
||||
</ol>
|
||||
</div>
|
||||
|
||||
</textarea></section>
|
||||
|
||||
<section data-markdown><textarea data-template>
|
||||
# why would haskell put that path in there?!
|
||||
|
||||
Unfortunately a known issue; help wanted.
|
||||
|
||||
<img src="img/github-issues.png" alt="screenshot of the github issues page for nixpkgs, showing a bug about large closures">
|
||||
|
||||
</textarea></section>
|
||||
|
||||
<section data-markdown><textarea data-template>
|
||||
# practical closure size problems
|
||||
|
||||
I was packaging [actual-budget], a budgeting program, to run it on fly.io. So I
|
||||
figured out how to build a docker image for it with Nix.
|
||||
|
||||
[actual-budget]: https://actualbudget.com/
|
||||
|
||||
</textarea></section>
|
||||
|
||||
<section data-markdown><textarea data-template>
|
||||
# and then it's way too big
|
||||
|
||||
<pre><code class="text"><script type="text/template"> » nix build .#dockerImage.aarch64-linux
|
||||
» ls -lah $(readlink result)
|
||||
-r--r--r-- 1 root root 219M Dec 31 1969 /nix/store/6r0nslg23w2sa6a2zril3g0fvd6mnp1q-actual-server.tar.gz
|
||||
</script></code></pre>
|
||||
|
||||
200MB compressed is not good. It's just some JavaScript, what happened there?
|
||||
</textarea></section>
|
||||
|
||||
<section data-markdown><textarea data-template>
|
||||
# let's bring out the big guns
|
||||
|
||||
```text
|
||||
nix-closure-graph --lg .# | goo copy
|
||||
```
|
||||
|
||||
</textarea></section>
|
||||
|
||||
<section data-markdown><textarea data-template>
|
||||
# what went wrong here?
|
||||
|
||||
<img class="r-stretch" src="./img/actual-stage0.png" alt="screenshot of the graph viewer showing that actual-server depends on actual-server-modules, a derivation of suspiciously similar size">
|
||||
|
||||
<!-- FIXME: embed the graph viewer -->
|
||||
|
||||
</textarea></section>
|
||||
|
||||
<section data-markdown><textarea data-template>
|
||||
# python?
|
||||
|
||||
<img class="r-stretch" src="./img/actual-stage0-python.png" alt="screenshot of the graph viewer showing that nodejs depends on python">
|
||||
|
||||
<!-- FIXME: embed the graph viewer -->
|
||||
|
||||
</textarea></section>
|
||||
|
||||
<section data-markdown><textarea data-template>
|
||||
# what went wrong here?
|
||||
|
||||
* Python
|
||||
* Mysteriously includes the app twice
|
||||
|
||||
</textarea></section>
|
||||
|
||||
<section data-markdown><textarea data-template>
|
||||
# python?!
|
||||
|
||||
* Node has an obsolescent build system "Generate Your Projects" for
|
||||
C++ extensions that generates makefiles
|
||||
* Originally inherited from Chromium
|
||||
|
||||
</textarea></section>
|
||||
|
||||
<section data-markdown><textarea data-template>
|
||||
# Generaten't My Projects
|
||||
|
||||
nodejs-slim to the rescue!
|
||||
|
||||
</textarea></section>
|
||||
|
||||
<section data-markdown><textarea data-template>
|
||||
# why is the app there twice?
|
||||
|
||||
<pre><code class="text"><script type="text/template">» nix why-depends -a --precise \
|
||||
/nix/store/qq1zblyp6ysx-actual-server \
|
||||
/nix/store/i778ag8s7jf8-actual-sync-modules-1.0.1
|
||||
|
||||
/nix/store/qq1zblyp6ysx96vd8havhw0wcsihcidf-actual-server
|
||||
└───libexec/actual-sync/deps/actual-sync/node_modules
|
||||
-> /nix/store/i778ag8s7jf8b-actual-sync-mo>
|
||||
→ /nix/store/i778ag8s7jf8-actual-sync-modules-1.0.1
|
||||
</script>
|
||||
</code></pre>
|
||||
</textarea></section>
|
||||
|
||||
<section data-markdown><textarea data-template>
|
||||
# what's in there anyway?
|
||||
|
||||
<pre><code class="text"><script type="text/template"> » ls -a result/libexec/actual-sync/deps/actual-sync/node_modules
|
||||
. .. .bin
|
||||
» ls -a result/libexec/actual-sync/deps/actual-sync/node_modules/.bin
|
||||
. .. eslint prettier tsc tsserver uuid
|
||||
</script>
|
||||
</code></pre>
|
||||
</textarea></section>
|
||||
|
||||
<section data-markdown><textarea data-template>
|
||||
# yeah I don't know why the builder did that
|
||||
|
||||
<img class="r-stretch" src="img/lets-see-who-this-really-is.png" alt="Scooby Doo 'let's see who this is anyway' meme, unmasking 'closure size' to be 'totally pointless stuff'">
|
||||
|
||||
</textarea></section>
|
||||
|
||||
<section data-markdown><textarea data-template>
|
||||
# actually fixing it
|
||||
|
||||
To recap, our goals are:
|
||||
1. Getting rid of Python via nodejs-slim
|
||||
2. Fixing the app being included twice
|
||||
|
||||
</textarea></section>
|
||||
|
||||
<section data-markdown><textarea data-template>
|
||||
<video class="r-stretch" src="img/livedemo.mp4" controls>
|
||||
</textarea></section>
|
||||
|
||||
<section data-markdown><textarea data-template>
|
||||
# summary
|
||||
|
||||
* Accidental dependencies cause closure size bloat
|
||||
* Shipping suspiciously sentient spaghetti slurps time and bandwidth
|
||||
* Fix closure size issues by deleting references
|
||||
</textarea></section>
|
||||
|
||||
<section data-markdown><textarea data-template>
|
||||
# Mercury is hiring!
|
||||
|
||||
* Frontend
|
||||
* Backend<fnref>1</fnref>
|
||||
* Full stack
|
||||
* Design
|
||||
* Data engineering
|
||||
* Data science
|
||||
* Engineering management
|
||||
|
||||
See open positions at https://mercury.com/jobs
|
||||
<div class="footnotes">
|
||||
<ol><li>No Haskell experience required or expected; training is provided!</li></ol>
|
||||
</div>
|
||||
</textarea></section>
|
||||
|
||||
<section data-markdown><textarea data-template>
|
||||
# fin
|
||||
|
||||
Tools:
|
||||
- nix-closure-graph: https://github.com/lf-/dotfiles/tree/main/programs/nix-closure-graph
|
||||
- Graph viewer: https://mercurytechnologies.github.io/looking-glass-viewer/
|
||||
|
||||
[@leftpaddotpy on Twitter](https://twitter.com/leftpaddotpy)
|
||||
|
||||
<https://jade.fyi>
|
||||
|
||||
my name [at] jade [dot] fyi
|
||||
</textarea></section>
|
||||
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<script src="dist/reveal.js"></script>
|
||||
<script src="plugin/notes/notes.js"></script>
|
||||
<script src="plugin/markdown/markdown.js"></script>
|
||||
<script src="plugin/highlight/highlight.js"></script>
|
||||
<script>
|
||||
// More info about initialization & config:
|
||||
// - https://revealjs.com/initialization/
|
||||
// - https://revealjs.com/config/
|
||||
Reveal.initialize({
|
||||
hash: true,
|
||||
|
||||
// Learn about plugins: https://revealjs.com/plugins/
|
||||
plugins: [ RevealMarkdown, RevealHighlight, RevealNotes ],
|
||||
transitionSpeed: 'fast',
|
||||
transition: 'none',
|
||||
slideNumber: true,
|
||||
});
|
||||
</script>
|
||||
</body>
|
||||
</html>
|
||||
Loading…
Add table
Add a link
Reference in a new issue