There are few tools that feel like a permanent uphill battle like OpenPGP, at least to me. It is a critical specification in FOSS and even so I frequently find myself banging my head against it in use cases that one would argue are basic or should work more easily. It isn’t just GnuPG, which has become a monstruous behemoth, as I will touch below, but also the whole of OpenPGP regardless of implementation used. It’s frustrating to use, to a level I would equate to the frustration of using XMPP (to this day!), but the difference being, as I said above, that OpenPGP is a critical specification that we depend on for the integrity and security of our systems.

Where do I even start? First, there is the well-known problem of key sharing and publication, especially after the debacle of key servers running SKS back in 2019. The reason why those attacks rendered the popular MIT key server unusable and also forced Ubuntu to switch theirs to Hockeypuck was, simply put, that public trust in keys is enforced by signatures that are appended to the key itself. Also, keys allow for arbitrary data such as comments, even an image file, so signing a key with a key specially crafted to be excessively long would effectively render that key unusable as clients would have a hard time retrieving or refreshing it. SKS servers federate by default, so the attack on a key would propagate to any other key server running SKS. Therefore, modern key server software strips keys of a lot of information or, like in the case of the now very popular Hagrid-based OpenPGP.org key server, they don’t federate with anyone… because they favor something called the Web Key Directory protocol or WKD.

The WKD protocol allows for keys be found via the domain names of the email addresses defining identities. In short, you can find my key by telling your OpenPGP client to reach my domain, instead of looking for it on a key server. OpenPGP.org’s key server provides support for WKD and is as easy to set up as pointing a CNAME record in your domain’s DNS Zone File to a subdomain of theirs. Alternatively, you can set up WKD in your own web server if you run one. Some email providers make your keys available via WKD automatically.

The idea is brilliant and, in my opinion, brought life back to an infrastructure that was falling apart. The problem is that the whole of our OpenPGP “culture” still assumes key servers to be the “real” default, even though GnuPG’s --locate-external-keys has defaulted to WKD for a long time now… but how many people are still using the --recv-keys option instead? The latter defaults to key servers and is the option that is usually stated as “the one” to use to “get someone’s key.”

This leads directly to an absurd problem I had for weeks on my Arch Linux laptop. For historical reasons, GnuPG implements a very peculiar network stack of its own called dirmngr. It deals both with key server and WKD requests. It’s also one of the most picky and fragile pieces of networking software in FOSS, operating with an overall philosophy that any network failure is a critical security risk and should be dealt with promptly aborting the operation requested, no matter how mundane the error was. Sequoia, the new OpenPGP implementation that has popped up in the last decade, just uses the OS’s network stack. However, although Sequoia has gained a lot of traction as a more user-friendly OpenPGP client, especially regarding its CLI, also for historical reasons, GnuPG is still the implementation used in background services or in infrastructure in general. So, due to an issue with my ISP at my new home (now solved), the archlinux-keyring-wkd-sync service, which updates the signatures of keys already existing in the distro’s keyring, failed miserably when started by the corresponding timer, wasting CPU and polluting the logs ever time systemd restarted it until it reached the limit of service failures.

We should blame my ISP for keeping us on a low-bandwith plan due to some serious miscommunication, but GnuPG’s draconian handling of network issues didn’t help at all: the key lookup failed because packages were dropped when hitting the bandwidth limit, so instead of retrying, it decided to pull the plug. As this was being called by a service, the speed of the whole operation kept the bandwith rate high. If I looked up for keys manually, a second try would work as a charm because I’m not as fast as my CPU, so the bandwith rate per seconds lowered between manual calls to GnuPG.

The WKD service that Arch enables to update the local keyring isn’t critical. It comes enabled by default at the time I’m writing this (it didn’t some years ago), but disabling it won’t render your system unstable and it won’t cut your access to the repositories either. New keys are always added by updating the archlinux-keyring package instead, so the worst that could happen if the weekly WKD service doesn’t work is that you keep old keys around that should be revoked or that you miss some signatures on some keys. There is a security risk with the former, though, but it would only be relevant if the repos had been succesfully attacked. In any case the keyring package will always update everything that should be.

However, it was annoying because systemd was complaining that my system’s state was degraded. This was a boy that cried wolf situation: I got used to the fact that degraded meant that it was the non-critical WKD service that had failed, but this isn’t the proper way to maintain a system. Degraded means something is wrong and you should check the error even though chances are that it’s that pesky WKD service acting up again. What if this time was something completely different? False positives are bad in all situations and need to be fixed. However I find these I had to have their origin in unnecessarily ungraceful handling of common network issues that pose no real security risk by a piece of software that assumes everything is a potential MITM attack, while Sequoia does fine without being such a hardliner.

You might be able to fine-tune dirmngr’s policies but GnuPG is huge. It’s got a level of complexity much alike Apache’s when it comes to configuration files.

And then Thunderbird’s own OpenPGP implementation decided to corrupt itself.

I don’t know what happened and I don’t really know how Thunderbird implements the specification. All I know is that it runs independently of your GnuPG installation, with a keyring of its own, with its own key lookup system, etc. I think I partially know what could have been the problem: from time to time, if your Firefox or Thunderbird user profile was created many versions ago, incompatibilities start to pop up and these applications start to fail in funny ways. Last time something like this happened was on Firefox and I only noticed it because the spacing between elements in the UI was odd and the customization screen refused to accept any of my changes. Purging the profile solved the problem back then on Firefox and it also did now on Thunderbird… and after solving it I found out that the UI has also been corrupted there (this time, the positioning of some elements). I wonder why this happens on Mozilla’s software when, on the other hand, LibreOffice happily migrates decades old configuration files to their new versions without a flinch. Maybe the answer is precisely “Mozilla.”

While I was figuring out what was going on, I temporarily switched to a different email client, aerc. I use aerc mostly for development-related stuff such as git patches and mailing lists because I know aerc won’t mess the formatting up and, in general, it suits that kind of job much better than Thunderbird in my opinion. It’s a good thing that email is a sane protocol that allows you to use as many clients as you like without any danger. What I’m about to tell now makes sense the way it is, but it did make me realize how broken OpenPGP is.

When encrypting an email, aerc requires the recipients’ keys to be trusted. You do that manually using your OpenPGP client of choice when you get someone’s keys. This is how OpenPGP is meant to be used, but all other OpenPGP-capable software I’ve ever used do without that requirement because trusting a key should be done by authenticating the owner by a means different than the one you got the key from. Ideally, you’d do that via a phone call or by meeting the owner in person and checking the key’s fingerprint. This is security 101. Of course, no one does this unless it’s for highly critical keys like the master signing keys of a Linux distro. A “normal” user that just wants to have some decent E2E encryption for their email or just wants to check the signature of an ISO they downloaded won’t bother with going the full mile of proper security clearance protocols. So, even GnuPG lets you use keys you haven’t given any trust level whatsoever. So does Thunderbird. aerc, though, enforces what in theory should be enforced at least by you manually setting up some low level of trust for the key.

The latter made me feel like we’re performing too much security theater. The feeling was not due to aerc, but everything else. Then I realized aerc doesn’t support encrypting the subject line of emails, but Thunderbird does by default and aerc replaced them with “…”, so I had a hard time finding an email I needed to look up. Encrypting subject lines looks like that kind of feature that requires abusing the email specification in some very shady way, but I haven’t dug into this any further. I felt tired: We’re not even properly authenticating keys, why are we pretending we’re doing security? I know, what most of us want and need is just some E2E encryption scheme to make email behave like most messaging applications do nowadays: with some encryption layer that makes it harder for the opportunistic attacker… This is OpenPGP used like the envelope I put my letters when sending them by mail.1 That’s the philosophy behind OMEMO in XMPP or Matrix encryption keys: to provide an envelope. I don’t fool myself, of course: powerful actors probably can run complex attacks that break most consumer-grade E2E encryption schemes.

When it comes to package or ISO signing, the annoyance comes from how keys are managed. It always feels messy and half-baked. Sometimes you get the key as an attachment by email or it’s stored as an armored text file on a website. Sometimes you get it by WKD. Sometimes you get it from a key server, but then, is that key available on all popular servers? And if it’s yours, are you keeping it updated everywhere you uploaded it? (Some services, very notably code forges, ask you to upload your public key manually) What if someone’s key was revoked and you haven’t update your keyring in months?

Speaking of code forges, signing commits and tags is another point of contention. Commit signatures aren’t preserved after rebases (logically), are impossible to preserve in email patches, but GitHub has burned into our brains that signing commits is recommended, as their UI happily shows a Verified badge along the commit hash when a signature is found. Yes, they do work with pull requests and merge commits, but that’s it. I think tag signatures are always preserved as long as the tag exists, namely, as long as the commit the tag points to exists. Sourcehut, very wisely, hides commit signatures from its UI. I would stop signing my commits (and switch to a sign-off model) but a part of me screams: If you do that, the log will show that your commits are no longer signed! Someone might think you’ve been impersonated! Will they? Does anybody care? I can always write an “announcement” here on my blog explaining the situation (this ain’t it, though!) and cross-post it to my projects’ mailing lists, right? Is there even a theoretical way to make OpenPGP more “resilient” in git? Do we want that? Should I give up?

Again, this boxing match between me and OpenPGP has rendered me exhausted. I just wish I could ignore OpenPGP or someone implemented a totally new specification from scratch for the modern age, but all we have is alternative OpenPGP clients and workarounds. I sigh, tired.

  1. Yes, I use mail! ↩︎