clipper post some more

This commit is contained in:
Jade Lovelace 2023-07-20 01:20:50 -07:00
parent bea2d15b6b
commit 0cfb543828
5 changed files with 52 additions and 26 deletions

View file

@ -6,18 +6,24 @@ tags = ["clipper", "rust"]
title = "Announcing Clipper: TLS-transparent HTTP debugging for native apps"
+++
<!-- FIXME: wire_blahaj.jpg -->
[Repo link][repo]
[repo]: https://github.com/lf-/clipper
{% image(name="wire_blahaj.jpg", colocated=true) %}
little blahaj wrapped in ethernet cable
{% end %}
Sometimes, I get jealous of web developers for having real network debugging
tools for watching HTTP traffic. Other times, I decide I can have nice things
too and spend a few weeks writing a tool.
A while ago I was debugging an issue with the Rust OpenTelemetry library and it
was immensely frustrated by not being able to get the actual requests out of
it. Eventually it yielded when I forked `h2` to add some logging that would be
exceptionally inadmissible in production. This is (probably) not the fault of
either rust-opentelemetry or the HTTP library that the observability system was
not observable, but it sure was frustrating.
A while ago I was debugging an issue with the Rust OpenTelemetry library on my
computer and it was immensely frustrated by not being able to get the actual
requests out of it. Eventually it yielded when I forked `h2` to add some
logging that would be exceptionally inadmissible in production. This is
(probably) not the fault of either rust-opentelemetry or the HTTP library that
the observability system was not observable, but it sure was frustrating.
Perhaps HTTP libraries should have a standard interface for dumping the raw
requests they perform, but this is going to be filtered through what the
@ -27,8 +33,14 @@ quite variable.
What if there *was* one tool that was universal and could read any HTTP traffic
with no configuration, regardless of the app? What if debugging a modern native
app's HTTP traffic (HTTP3 coming soon) could be (almost) as easy as opening dev
tools in a browser?
app's HTTP traffic could be (almost) as easy as opening dev tools in a browser?
I picked up the hammer and built one:
<div class="image">
<video controls alt="video of clipper showing curl requests in chrome devtools to the side" src="./chrome-devtools-live.webm">
</video>
</div>
If only modern computers had a [Clipper chip] and you happened to be the NSA so
you could just decrypt everything. Thankfully that is not the case. I suppose
@ -49,21 +61,27 @@ supports:
## Wire shark doo doo doo doo doo doo
<!-- FIXME: tls added and removed here slide -->
{% image(name="ssl-added-and-removed-here.png", colocated=true) %}
Notorious NSA PRISM slide titled "Current Efforts - Google", showing Google
Front End stripping off TLS before sending cleartext into Google datacenters.
{% end %}
But it's encrypted! Sadly, "TLS added and removed here" is an architecture of
the past, and for good reason. However, what if we had the session keys and
decrypted the TLS so we could read it?
<!-- FIXME: link -->
It turns out that Wireshark can actually do this, if you have the keys already.
However, getting the keys is the hard part.
It turns out that Wireshark [can actually do this][wireshark-tls], if you have
the keys already. However, getting the keys is the hard part.
[wireshark-tls]: https://wiki.wireshark.org/TLS
<!-- FIXME: link to SSLKEYLOGFILE spec -->
Also, it turns out that we *did* collectively standardize getting the keys out
of TLS, and practically every TLS implementation implements it. Specifically,
there is a standard format called `SSLKEYLOGFILE`, originally implemented in
Mozilla NSS, for logging the decryption keys from TLS libraries.
there is a [standard format called `SSLKEYLOGFILE`][sslkeylogfile-spec],
originally implemented in Mozilla NSS, for logging the decryption keys from TLS
libraries.
[sslkeylogfile-spec]: https://www.ietf.org/archive/id/draft-thomson-tls-keylogfile-00.html
Cool, so we're done? Well, as much as it is implemented in the TLS libraries,
code changes to client code are required to actually use it. This is probably
@ -72,19 +90,20 @@ for good reason, since it does break TLS:
- `curl` gates it behind a compile flag that is off by default. Not sure about
libcurl.
- NSS gates it behind a compile flag which is off by default.
- Firefox only enables the NSS flag on nightly or other dev builds.
- Firefox only enables that NSS flag on nightly or other dev builds.
- rustls requires you set a special field on the ClientSettings and
ServerSettings structures, which downstream users do not do by default.
- Go `crypto/tls` requires a similar method to rustls.
- OpenSSL requires you call `SSL_CTX_set_keylog_callback` when initializing it.
- Chromium implements the `SSLKEYLOGFILE` environment variable by default, yay.
<!-- FIXME: link -->
Alarmingly, there is a bug report of this feature being abused by antivirus
Alarmingly, [there is a bug report of this feature being
abused][no-good-deed-goes-unpunished] by antivirus
vendors to do TLS decryption, leading the Chromium developers to remove the
"your traffic is being intercepted" banner. So perhaps everyone else removing
it was a good idea.
"dangerous environment variable" banner. So perhaps everyone else removing
support was a good idea.
[no-good-deed-goes-unpunished]: https://crbug.com/991290
## Fiddler2, OWASP ZAP, mitmproxy
@ -128,7 +147,6 @@ As for getting the keys, either you can patch the binary at compile time
runtime (fun, indistinguishable from malware behaviour). Since I want a magical
user experience, runtime patching it is.
<!-- FIXME: link -->
Some absolute sickos at [mirrord] made a system which allows a local process on
a developer computer to interact with the network as if it is on another
machine in a staging environment. This was achieved by using the excellent
@ -136,6 +154,7 @@ machine in a staging environment. This was achieved by using the excellent
along with `LD_PRELOAD` to get their code running inside processes.
[Frida]: https://frida.re
[mirrord]: https://metalbear.co/blog/mirrord-internals-hooking-libc-functions-in-rust-and-fixing-bugs/
Clipper also uses Frida GUM. The actual patches to extract keys aren't much:
- In OpenSSL, hook `SSL_new` to first call `SSL_CTX_set_keylog_callback` on the
@ -143,10 +162,17 @@ Clipper also uses Frida GUM. The actual patches to extract keys aren't much:
- In rustls, they use vtable dispatch over a field set in the client/server
settings structure. This seems like a pain in the ass, so we instead hook the
no-op key log functions to not be no-ops.
- In Go `crypto/tls` they use a similar mechanism to rustls and we do the same
thing.
- In Go `crypto/tls` they use a similar mechanism to rustls and we can probably
do the same thing to rustls.
Once we have the keys, we can add universal `SSLKEYLOGFILE` support, or send
the keys over a socket to the capturing Clipper instance. Both of these are
implemented.
# Thanks
- [edef] for getting mad at this with me
- [puck] for helping me with several complex Linux and Rust issues
[puck]: https://twitter.com/puckipedia
[edef]: https://twitter.com/edefic

Binary file not shown.

After

Width:  |  Height:  |  Size: 313 KiB

Binary file not shown.

After

Width:  |  Height:  |  Size: 2.7 MiB

View file

@ -207,7 +207,7 @@ details {
margin-bottom: 2em;
}
img {
img, video {
// make the alt text extremely obnoxious and harder to miss if the image
// is missing
font-size: 3rem;