wip post

content/posts/debugging-rr-children/.gitignore

@@ -0,0 +1,2 @@
+/crasher
+/caller

content/posts/debugging-rr-children/Makefile

@@ -0,0 +1,5 @@
+CFLAGS = -g
+all: caller crasher
+.PHONY: all
+caller: caller.o
+crasher: crasher.o

content/posts/debugging-rr-children/caller.c

@@ -0,0 +1,21 @@
+#include <stdio.h>
+#include <unistd.h>
+#include <sys/wait.h>
+
+int main(void) {
+    int pid = fork();
+    if (pid == -1) {
+        perror("fork");
+        return 1;
+    } else if (pid == 0) {
+        execl("./crasher", "./crasher", NULL);
+        return 1;
+    } else {
+        // parent
+        int status;
+        printf("[caller] spawned pid %d\n", pid);
+        int ret = waitpid(pid, &status, 0);
+        printf("[caller] waitpid: %d, exited? %d status %d, signaled? %d signal %d\n", ret, WIFEXITED(status), WEXITSTATUS(status), WIFSIGNALED(status), WTERMSIG(status));
+        return 0;
+    }
+}

content/posts/debugging-rr-children/crasher.c

@@ -0,0 +1,7 @@
+#include <stdio.h>
+#include <stdlib.h>
+
+int main(void) {
+    printf("[crasher] about to crash\n");
+    abort();
+}

content/posts/debugging-rr-children/index.md

@@ -0,0 +1,80 @@
++++
+date = "2022-02-24"
+draft = false
+path = "/blog/debugging-rr-children"
+tags = []
+title = "Debugging: using rr to deal with unruly children (processes)"
++++
+
+I have done multiple rounds of debugging blobs of processes that start together
+and then something bad happens in one of the children several forks down.
+Although gdb claims to support child processes with `set follow-fork-mode`
+([docs][gdb-follow-fork-mode]), in practice, this is extremely painful since
+you may have to set it to multiple different things in one reproduction.
+
+To deal with these, I've done such hacks as writing wrapper scripts for the
+executable at fault that run it in a gdbserver. However, by far the worst one
+I've done is printing out the PID of the misbehaving process then waiting, to
+give me time to attach the debugger ([this is even suggested in the gdb
+documentation][gdb-sleep]).
+
+[gdb-follow-fork-mode]: https://docs.jade.fyi/gnu/gdb/gdb.html#index-set-follow_002dfork_002dmode
+[gdb-sleep]: https://docs.jade.fyi/gnu/gdb/gdb.html#Forks
+
+## Using rr to do it the easy way
+
+For this demo, I am using two programs I wrote:
+
+- `crasher` just prints out that it's about to crash, then aborts.
+- `caller` forks and executes `crasher`, then prints its return value once it
+   exits.
+
+These are written in C but their source is not super interesting. Nevertheless,
+you can find their source code [at the bottom of the post](#source).
+
+Here they are in action:
+
+```
+» ./caller
+[caller] spawned pid 158938
+[crasher] about to crash
+[caller] waitpid: 158938, exited? 0 status 0, signaled? 1 signal 6
+```
+
+Signal 6, if you consult the table in `man 'signal(7)'`, is `SIGABRT` as
+expected.
+
+We want to figure out why the crasher is crashing. It's possible to do with
+gdb, but that's unnecessarily hard because of gdb, even moreso if it forks
+multiple times.
+
+Let's use `rr` to do this more easily. First, record a run:
+
+```
+» rr record ./caller
+rr: Saving execution to trace directory `/home/lf/.local/share/rr/caller-0'.
+[caller] spawned pid 159146
+[crasher] about to crash
+[caller] waitpid: 159146, exited? 0 status 0, signaled? 1 signal 6
+```
+
+Then find the process ID of the crashing process:
+
+```
+» rr ps
+PID     PPID    EXIT    CMD
+159145  --      0       ./caller
+159146  159145  -6      ./crasher
+```
+
+Next, use either `--onfork=<PID>` or `--onprocess=<PID>` to get a debugger on
+the problem process:
+
+```
+
+
+## Demo source {#source}
+
+{{ codefile(path="caller.c", code_lang="c", colocated=true, hide=true) }}
+{{ codefile(path="crasher.c", code_lang="c", colocated=true, hide=true) }}
+{{ codefile(path="Makefile", code_lang="make", colocated=true, hide=true) }}