7.9 KiB
+++ date = "2023-06-24" draft = true path = "/blog/announcing-clipper" tags = ["clipper", "rust"] title = "Announcing Clipper: TLS-transparent HTTP debugging for native apps" +++
{% image(name="wire_blahaj.jpg", colocated=true) %} little blahaj wrapped in ethernet cable {% end %}
Sometimes, I get jealous of web developers for having real network debugging tools for watching HTTP traffic. Other times, I decide I can have nice things too and spend a few weeks writing a tool.
A while ago I was debugging an issue with the Rust OpenTelemetry library on my
computer and it was immensely frustrated by not being able to get the actual
requests out of it. Eventually it yielded when I forked h2 to add some
logging that would be exceptionally inadmissible in production. This is
(probably) not the fault of either rust-opentelemetry or the HTTP library that
the observability system was not observable, but it sure was frustrating.
Perhaps HTTP libraries should have a standard interface for dumping the raw requests they perform, but this is going to be filtered through what the library thought it did, rather than necessarily what it actually did. Plus, there's hundreds of HTTP libraries, so the experience would be undoubtedly quite variable.
What if there was one tool that was universal and could read any HTTP traffic with no configuration, regardless of the app? What if debugging a modern native app's HTTP traffic could be (almost) as easy as opening dev tools in a browser?
I picked up the hammer and built one:
If only modern computers had a Clipper chip and you happened to be the NSA so
you could just decrypt everything. Thankfully that is not the case. I suppose
the next best thing is to have code execution, a binary patching library, and
insufficient fear of LD_PRELOAD.
What is this thing?
Clipper is a suite of tools for doing TLS-transparent HTTP interception. It supports:
- Unprivileged packet capture that also catches keys, storing to PCAPNG files.
- Attaching Chrome DevTools to unmodified native processes and viewing HTTP activity.
- Viewing PCAPNG files in Chrome DevTools.
- Extracting keys from unmodified applications.
Wire shark doo doo doo doo doo doo
{% image(name="ssl-added-and-removed-here.png", colocated=true) %} Notorious NSA PRISM slide titled "Current Efforts - Google", showing Google Front End stripping off TLS before sending cleartext into Google datacenters. {% end %}
But it's encrypted! Sadly, "TLS added and removed here" is an architecture of the past, and for good reason. However, what if we had the session keys and decrypted the TLS so we could read it?
It turns out that Wireshark can actually do this, if you have the keys already. However, getting the keys is the hard part.
Also, it turns out that we did collectively standardize getting the keys out
of TLS, and practically every TLS implementation implements it. Specifically,
there is a standard format called SSLKEYLOGFILE,
originally implemented in Mozilla NSS, for logging the decryption keys from TLS
libraries.
Cool, so we're done? Well, as much as it is implemented in the TLS libraries, code changes to client code are required to actually use it. This is probably for good reason, since it does break TLS:
-
curlgates it behind a compile flag that is off by default. Not sure about libcurl. -
NSS gates it behind a compile flag which is off by default.
-
Firefox only enables that NSS flag on nightly or other dev builds.
-
rustls requires you set a special field on the ClientSettings and ServerSettings structures, which downstream users do not do by default.
-
Go
crypto/tlsrequires a similar method to rustls. -
OpenSSL requires you call
SSL_CTX_set_keylog_callbackwhen initializing it. -
Chromium implements the
SSLKEYLOGFILEenvironment variable by default, yay.Alarmingly, there is a bug report of this feature being abused by antivirus vendors to do TLS decryption, leading the Chromium developers to remove the "dangerous environment variable" banner. So perhaps everyone else removing support was a good idea.
Fiddler2, OWASP ZAP, mitmproxy
These are all fine tools and perhaps better suited to this use case. However, in order to use any of these proxies, one needs to execute an actual woman-in-the-middle attack, which has Consequences:
- Proxy support is required
- Support for adding trusted CAs is required
- Key pinning must be disabled
- It's possible that certain HTTP library compatibility bugs may be concealed by virtue of the proxy decoding and reencoding the data.
I would like to look at the actual traffic and not put anything relevant in the data plane to the greatest extent possible.
What the hell crimes did you do to achieve that, jade
Glad you asked. Numerous!
Let's start at capturing packets. Normally this requires having high privilege
on Linux in order to be able to bind a AF_PACKET socket to capture packets.
However, CAP_NET_RAW privilege is in the eye of the beholder. Thanks to the
technologies used by unprivileged containers, this is possible and fairly easy.
Specifically, by entering a new user namespace and network namespace, Clipper
jails processes away from the host network onto a virtual network over which it
has root due to the user namespace. Then it can open a AF_PACKET socket and
send a handle out of the namespace to the host process over a unix(7) socket
and capture everything going on in the namespace.
However, that's not sufficient, since the inner process cannot actually connect
to anything from a new network namespace and host root would be required to
attach virtual interfaces to the host network stack. Fortunately, again, the
missing pieces were built for rootless containers: slirp4netns is a userspace
NAT daemon that NATs a network namespace onto the host network, creating a
tap interface with internet access inside the namespace.
As for getting the keys, either you can patch the binary at compile time (boring, may require patching dependencies), or you can patch the binary at runtime (fun, indistinguishable from malware behaviour). Since I want a magical user experience, runtime patching it is.
Some absolute sickos at mirrord made a system which allows a local process on
a developer computer to interact with the network as if it is on another
machine in a staging environment. This was achieved by using the excellent
Frida GUM Rust bindings, which allow hooking arbitrary code in executables,
along with LD_PRELOAD to get their code running inside processes.
Clipper also uses Frida GUM. The actual patches to extract keys aren't much:
- In OpenSSL, hook
SSL_newto first callSSL_CTX_set_keylog_callbackon the passed in context. - In rustls, they use vtable dispatch over a field set in the client/server settings structure. This seems like a pain in the ass, so we instead hook the no-op key log functions to not be no-ops.
- In Go
crypto/tlsthey use a similar mechanism to rustls and we can probably do the same thing to rustls.
Once we have the keys, we can add universal SSLKEYLOGFILE support, or send
the keys over a socket to the capturing Clipper instance. Both of these are
implemented.