jade/blog
0
0
Fork 0
Code Issues Pull requests Projects Releases Packages Wiki Activity Actions

more tracing stuff i guess?

Browse source
This commit is contained in:
Jade Lovelace 2023-06-24 09:51:09 +02:00
parent fdef491dae
commit 3a1d2710d6
1 changed files with 62 additions and 11 deletions
Download patch file Download diff file Expand all files Collapse all files

109
content/posts/tracing-ideas.md
View file

@ -1,109 +0,0 @@
+++
date = "2022-12-26"
draft = true
path = "/blog/tracing-ideas"
tags = ["haskell", "opentelemetry"]
title = "Make tracing easy easily! Solving more problems with tracing"
+++
I interned at Mercury for several months and built out some ideas for making
OpenTelemetry tracing the first choice to investigate something by making it
the easiest and most useful option available. This blog post catalogues the
ideas that I implemented, how much work they were, and whether I think they're
worth it.
## Put a link to traces in a header
I made the back-end emit a header `trace-link`, which contains a link to the
Honeycomb trace for the request.
#### How easy was it?
1 afternoon of work (plus a couple days work later once we had to start hitting
the API due to the new data model with environments). Most of this work is open
source and reusable for Haskell apps.
#### What did it accomplish?
This was probably the best tracing adoption improvement I made because it lets
devs directly look at misbehaving requests in browser dev tools and then open
the trace in one click. It singlehandedly got a handful of people to start
using tracing.
It doesn't really give any capability that isn't available by copying the trace
ID out of the second component of the `traceparent` header you're already
sending if you're using the [w3c trace propagator], however, doing that is very
arduous and manual.
#### How to do it
If you're using the hs-opentelemetry ecosystem for Haskell, the relevant code
is here, in the package `hs-opentelemetry-vendor-honeycomb`:
https://github.com/iand675/hs-opentelemetry/tree/main/vendors/honeycomb
What this package does is:
1. Find where data is going using the [Honeycomb Auth API]: you need to know
the dataset, tenancy name, and environment that the API key is going into.
In my design, this data is acquired at startup time so trace link generation
is just string concatenation thereafter.
2. Create [Direct Trace Links] using the trace ID then put them in a header.
[Honeycomb Auth API]: https://docs.honeycomb.io/api/auth/
[Direct Trace Links]: https://docs.honeycomb.io/api/direct-trace-links/
[w3c trace propagator]: https://www.w3.org/TR/trace-context/
## Instrument the test suite
#### How easy was it?
Implementing the hspec stuff originally took about half a week since it involved reading
substantial amounts of hspec internals. I assume probably similar times for
initially adding instrumentation to any other test framework/language.
However, once the integration to your test framework of choice exists, it takes
a few minutes to add it to a new codebase.
#### What did it accomplish?
I was initially surprised at this having as big an impact as it did, but
Honeycomb wound up being the easiest and cleanest way to view test suite runs
and get database logs, exceptions and other useful debugging info to fix broken
tests. This was a very worthwhile project and saved a handful of people
probably a couple of hours each debugging thorny test failures.
#### How to do it
I wrote a Haskell library that starts spans for each test case in hspec:
[hs-opentelemetry-instrumentation-hspec]. Plug this in per the example in the
sources, and then you're done.
Bonus points if you print out a trace link at the end, since you can just reuse
the trace link infrastructure from above for this.
You may also need to modify the way you do database interactions in tests to
use instrumentation, for example.
[hs-opentelemetry-instrumentation-hspec]: https://github.com/iand675/hs-opentelemetry/tree/main/instrumentation/hspec
## Instrument scheduled tasks
#### How easy was it?
20 minutes to initialize tracing that already existed for the app, but in the
scheduled tasks system.
#### What did it accomplish?
This one achieved ridiculously good results basically immediately: it's
significantly easier to debug scheduled task misbehaviour and performance.
#### How to do it
Initialize tracing in your scheduled task runner, then create a context/root
span for the task execution. Bonus points if you propagate the trace ID context
from whatever invoked the scheduled task so it can be referenced.