+++ date = "2021-02-19" draft = false path = "/blog/finding-functions-in-nixpkgs" tags = ["nix"] title = "Finding functions in nixpkgs" +++ It is a poorly guarded secret that the most effective way to get documentation on nixpkgs or nix functions is to read the source code. The hard part of that is *finding* the right source code. I'll document a few static and dynamic analysis strategies to find the documentation or definition of things in nixpkgs going from trivial to some effort. ## Simple These work on functions that have no wrappers around them, which account for most library functions in nixpkgs. The option for `ctags` is a little bit better because it will just show you the source code which you can subsequently trace through. ### Static #### ctags As of version 0.5.0 (released 2021-07-03), `nix-doc` supports emitting ctags files with `nix-doc tags .` from a nixpkgs checkout. This lets you `:tag` things in vim or other editor supporting ctags and instantly jump to them by name, as well as `CTRL-]` to jump to the symbol under the cursor. It's clever enough to distinguish functions from other values at a syntax level, but like every ctags tool that's about where it stops. #### Search tools Static analysis is, in my view, slightly slower, since you can't be sure you're getting the function you're seeing in the interactive environment in `nix repl` or elsewhere. There are two tools for this that are capable of parsing nix source, `nix-doc` (my project) and `manix`, both of which are in nixpkgs. Note that `nix-doc` will only work from within a nixpkgs checkout or if you pass it a second parameter of where your nixpkgs is. This is mostly because `nix-doc`'s main focus has switched to being mainly on providing a function in the REPL. There is also a [fork of `rnix-lsp`](https://github.com/elkowar/rnix-lsp) which provides `manix` based documentation from within editors. ``` # your nixpkgs checkout $ cd ~/dev/nixpkgs; nix-shell -p manix nix-doc [nix-shell:~/dev/nixpkgs]$ manix foldl ```

manix output


Here's what I found in nixpkgs: lib.lists.foldl'
lib.lists.foldl lib.foldl
lib.foldl' haskellPackages.foldl-statistics
haskellPackages.foldl-transduce-attoparsec
haskellPackages.foldl-incremental haskellPackages.foldl-exceptions
haskellPackages.foldl haskellPackages.foldl-transduce

Nixpkgs Comments
────────────────────
# foldl (lib/lists.nix)
 “left fold”, like `foldr`, but from the left:
     `foldl op nul [x_1 x_2 ... x_n] == op (... (op (op nul x_1) x_2) ... x_n)`.

     Type: foldl :: (b -> a -> b) -> b -> [a] -> b

     Example:
       lconcat = foldl (a: b: a + b) "z"
       lconcat [ "a" "b" "c" ]
       => "zabc"
       # different types
       lstrange = foldl (str: int: str + toString (int + 1)) "a"
       lstrange [ 1 2 3 4 ]
       => "a2345"



NixOS Documentation
────────────────────
# lib.lists.foldl' (foldl' :: (b -> a -> b) -> b -> [a] -> b)
Strict version of `foldl`.

NixOS Documentation
────────────────────
# lib.lists.foldl (foldl :: (b -> a -> b) -> b -> [a] -> b)
“left fold”, like `foldr`, but from the left:
`foldl op nul [x_1 x_2 ... x_n] == op (... (op (op nul x_1) x_2) ... x_n)`.

Arguments:
  op: Function argument
  nul: Function argument
  list: Function argument

Example:

  lconcat = foldl (a: b: a + b) "z"
  lconcat [ "a" "b" "c" ]
  => "zabc"
  # different types
  lstrange = foldl (str: int: str + toString (int + 1)) "a"
  lstrange [ 1 2 3 4 ]
  => "a2345"

  lconcat = foldl (a: b: a + b) "z"
  lconcat [ "a" "b" "c" ]
  => "zabc"
  # different types
  lstrange = foldl (str: int: str + toString (int + 1)) "a"
  lstrange [ 1 2 3 4 ]
  => "a2345"

`manix` includes the file path with the documentation from the nixpkgs sources but no line number. It also includes NixOS manual documentation, which I appreciate. ``` [nix-shell:~/dev/nixpkgs]$ nix-doc foldl ```

nix-doc output


   “left fold”, like `foldr`, but from the left:
       `foldl op nul [x_1 x_2 ... x_n] == op (... (op (op nul x_1) x_2) ... x_n)`.

       Type: foldl :: (b -> a -> b) -> b -> [a] -> b

       Example:
         lconcat = foldl (a: b: a + b) "z"
         lconcat [ "a" "b" "c" ]
         => "zabc"
   different types
         lstrange = foldl (str: int: str + toString (int + 1)) "a"
         lstrange [ 1 2 3 4 ]
         => "a2345"
foldl = op: nul: list: ...
# ./lib/lists.nix:80

`nix-doc` basically gets you the same thing, but it is missing `foldl'`, which is a bug in the `nix-doc` command line interface, I think. It does, however, give you a source path with line number so you can use middle click or `C-w F` or similar to go directly to the function's source in your editor. ### Dynamic This is the wheelhouse of `nix-doc`. It adds the two functions demonstrated below (added by the `nix-doc` Nix plugin, see [the README][1] for installation instructions): [1]: https://github.com/lf-/nix-doc/#nix-plugin

nix-repl> n = import <nixpkgs> {}

nix-repl> builtins.unsafeGetLambdaPos n.lib.foldl
{ column = 11; file = "/nix/store/...-nixpkgs-.../nixpkgs/lib/lists.nix"; line = 80; }

nix-repl> builtins.doc n.lib.foldl
   “left fold”, like `foldr`, but from the left:
       `foldl op nul [x_1 x_2 ... x_n] == op (... (op (op nul x_1) x_2) ... x_n)`.

       Type: foldl :: (b -> a -> b) -> b -> [a] -> b

       Example:
         lconcat = foldl (a: b: a + b) "z"
         lconcat [ "a" "b" "c" ]
         => "zabc"
   different types
         lstrange = foldl (str: int: str + toString (int + 1)) "a"
         lstrange [ 1 2 3 4 ]
         => "a2345"
func = op: nul: list: ...
# /nix/store/...-nixpkgs-.../nixpkgs/lib/lists.nix:80
null

You can also get this information using `builtins.unsafeGetAttrPos`, an undocumented built-in function in Nix itself:


nix-repl> builtins.unsafeGetAttrPos "foldl" n.lib
{ column = 25; file = "/nix/store/...-nixpkgs-.../nixpkgs/lib/default.nix"; line = 82; }

## Functions without documentation nixpkgs has a few of these. Let's pick on Haskell infrastructure because I am most familiar with it. First let's try some static analysis to try to find the signature or source for `nixpkgs.haskell.lib.disableLibraryProfiling`:


[nix-shell:~/dev/nixpkgs]$ manix disableLibraryProfiling

[nix-shell:~/dev/nixpkgs]$ nix-doc disableLibraryProfiling

[nix-shell:~/dev/nixpkgs]$ rg disableLibraryProfiling
pkgs/development/haskell-modules/configuration-common.nix
50:  ghc-heap-view = disableLibraryProfiling super.ghc-heap-view;
51:  ghc-datasize = disableLibraryProfiling super.ghc-datasize;
1343:  graphql-engine = disableLibraryProfiling( overrideCabal (super.graphql-engine.override {

pkgs/development/haskell-modules/lib.nix
177:  disableLibraryProfiling = drv: overrideCabal drv (drv: { enableLibraryProfiling = false; });

pkgs/development/haskell-modules/configuration-nix.nix
97:  hercules-ci-agent = disableLibraryProfiling super.hercules-ci-agent;

Oh dear! That's not good. Neither the `manix` nor `nix-doc` command line tools found the function. This leaves `rg`, which is not based on the Nix abstract syntax tree, and for functions that are used a lot of times, the definition of a function will get buried among its uses. This is not ideal. I believe that in the case of `nix-doc` it may have found it but ignored the function since it had no documentation. Let's test that.

Results of adding a comment to see what happens


[nix-shell:~/dev/nixpkgs]$ nix-doc disableLibraryProfiling
   disable library profiling
disableLibraryProfiling = drv: ...
# ./pkgs/development/haskell-modules/lib.nix:178

Yep. Well, time to pull out the dynamic analysis again. Like in the [simple case](#simple), you can get the source code with `builtins.unsafeGetAttrPos` or the functions added by `nix-doc`. On my system nixpkgs where there is no documentation comment for the function, this is what I get:


nix-repl> builtins.doc n.haskell.lib.disableLibraryProfiling

func = drv: ...
# /nix/store/...-nixpkgs-.../nixpkgs/pkgs/development/haskell-modules/lib.nix:177

Although there is no documentation, `nix-doc` has pulled out the signature, which may already be enough to guess what the function does. If not, there is a source code reference. ## Indirection The hardest class of functions to find documentation for are ones which are wrapped by some other function. These can be frustrating since the AST pattern matching for functions as used by `nix-doc` and `manix` will fall apart on them. An example function like this is `nixpkgs.fetchFromGitLab`, but any arbitrary package created via `callPackage` will also work for this, as they are not really functions, but you do want to find them. `manix` knows of the function, but does not know from whence it came, whereas `nix-doc`'s CLI does not see it at all:


[nix-shell:~/dev/nixpkgs]$ manix fetchFromGitLab
Here's what I found in nixpkgs: pkgsMusl.fetchFromGitLab
fetchFromGitLab fetchFromGitLab.override
fetchFromGitLab.__functor fetchFromGitLab.__functionArgs
pkgsHostTarget.fetchFromGitLab pkgsBuildBuild.fetchFromGitLab
pkgsStatic.fetchFromGitLab pkgsTargetTarget.fetchFromGitLab
targetPackages.fetchFromGitLab gitAndTools.fetchFromGitLab
__splicedPackages.fetchFromGitLab buildPackages.fetchFromGitLab
pkgsHostHost.fetchFromGitLab pkgsBuildHost.fetchFromGitLab
pkgsBuildTarget.fetchFromGitLab pkgsi686Linux.fetchFromGitLab


[nix-shell:~/dev/nixpkgs]$ nix-doc fetchFromGitLab

Time to get out the dynamic analysis again!


nix-repl> n = import <nixpkgs> {}

nix-repl> builtins.doc n.fetchFromGitLab
error: (string):1:1: value is a set while a lambda was expected

nix-repl> builtins.typeOf n.fetchFromGitLab
"set"

nix-repl> n.fetchFromGitLab
{ __functionArgs = { ... }; __functor = «lambda @ /nix/store/...-nixpkgs-.../nixpkgs/lib/trivial.nix:324:19»; override = { ... }; }

That didn't work! It's a set with the `__functor` attribute that makes it callable. Even if we try pointing `nix-doc` at the `__functor`, it will tell us about `setFunctionArgs`, which is not what we were looking for. From what I understand of the Nix internals from writing the plugin, there is not really a nice way of getting the source of a function wrapped like this, since the information is already lost by the time the value enters the dumping function as Nix only stores lambda and attribute definition locations so once you have taken the value of the attribute that information is no longer available. This could be resolved with a new REPL command as those take strings of source which could be split to get the attribute name and attribute set, but custom REPL commands are not supported so some kind of modification would have to be made to Nix itself to add this feature. Therefore, I have to use the last trick up my sleeve, `unsafeGetAttrPos`, to find the definition of the attribute:


nix-repl> builtins.unsafeGetAttrPos "fetchFromGitLab" n
{ column = 3; file = "/nix/store/...-nixpkgs-.../nixpkgs/pkgs/top-level/all-packages.nix"; line = 466; }

This tells me, albeit in an annoying to copy format, that the next breadcrumb is at `pkgs/top-level/all-packages.nix:466`, which is ```nix fetchFromGitLab = callPackage ../build-support/fetchgitlab {}; ``` Then, I can look in `pkgs/build-support/fetchgitlab` and find a `default.nix` which will have the definition I want.