Hi everyone! Nick here. I figured it was time I finally write a blog post. If you’ve seen me around on the Vox Pupuli Slack, you know that I’ve been deep in the trenches of OpenVox build code for some time now, getting the various bits to a secure and stable place where it’s easily accessible to community contributors.
Today’s story is about wrestling with MacOS’s Gatekeeper so we can release a production-ready MacOS agent build.
No unsigned code, ever
Perforce recently dropped a LinkedIn post about their latest Puppet Core release, bragging about how secure and enterprise-ready Puppet Core is. The one that stuck out to me: Fully signed binaries and agents - no unsigned code, ever
They’re right - unsigned binaries are like showing up to a formal dinner party in a crab costume. In OpenVox-land, all of our Linux binaries are fully signed. However, just like how Perforce’s internal dev builds are not always signed, our prerelease, non-final MacOS and Windows agents were unsigned (Windows remains so until we navigate the extortion racket that is paying for an Extended Validation code signing certificate from a vendor).
I applaud Perforce on their journey towards creating the safest, most secure Puppet environment possible, so I thought I’d write this little guide to help them take the next step. In order to reach the utopia of a fully signed and notarized MacOS agent that Gatekeeper is happy with, we have to not only sign the right things, but notarize it as well.
Gatekeeper: The ever vigilant security guard of MacOS
Apple’s Gatekeeper ensures that the software you download and run is trusted and safe. If your software isn’t both signed and notarized in the correct way, it will flat out refuse to open it (without extra shenanigans), warning that you potentially downloaded a radioactive piece of malware and you should be ashamed of yourself.
To make Gatekeeper happy, all you used to have to do in the halcyon days of pre-Sonoma was to sign the specific binary entrypoints into your app with an “application developer identity” (certificate + private key). Let’s dive into the GitHub mines and see how this is currently done.
Warning: There be dragons in these mines
The way that Perforce does this signing process for their current MacOS agents (or at least they did before taking everything private, but I don’t believe it has changed) is by sending the three executable files in the puppet-agent package to a signing server, which signs those files, and then sends them back to continue the build process. You can see how this is defined in the puppet-agent repo. This utilizes Vanagon’s ExtraFilesSigner to lay down a bash file to run the command, rsync the file over, run the command, then rsync the file back. This process is embedded in Vanagon’s MacOS platform build process. This code ends up with a dmg that contains a pkg installer, which contains all the files, including those three signed binaries. Finally, the pkg file must be signed with a different “installer developer identity”. Perforce does this by mounting the created dmg, signing the pkg with that identity, then recreating the dmg file with that signed pkg.
Signing and Notarization: Everyone’s favorite migraines
While this worked fine before, as of MacOS 15 Sonoma, the requirements are much more stringent. Not only must you sign the these binaries, but you must sign every single binary, dylib, and bundle file, no exceptions. And on top of that, you have to sign all of them specifying it should use the hardened runtime via the --options runtime flag to the codesign tool, as well a secure timestamp via the --timestamp flag. Doesn’t matter if the files are already signed in a different manner. MacOS 15 has a refined palate and will only consume the finest artisinal binaries. Yes, it’s probably for the best, but I digress.
Additionally, you must notarize either the dmg or pkg file. This involves sending your file to Apple, where their system scans for any hidden nasties and verifies everything that should be signed, is signed in the correct way. Once approved, you then must staple the notarization to the dmg or pkg so Gatekeeper can stop judging us.
Bringing a first pass of order to chaos
Right now in OpenVox, unlike Linux agents where we use containers to build them, we’re building the MacOS agent in a VM that is configured in a particular way with the appropriate bits and bobs installed. This is similar to how Perforce does it, except that this particular VM template also includes the bits needed for signing and notarization. This is not highly portable and accessible to people yet, but we wanted to get this agent into your hands as soon as possible. Soon, we’ll be moving the process into GitHub Actions and hopefully finding a way to build it locally without a Mac as well. We’ll take this awkward Ford Pinto and turn it into a Ferrari. Or at least a Honda Accord.
For this first pass at MacOS agent signing and notarizing, I moved the process entirely into Vanagon. While this may not be where we want to leave it since it reduces potential future flexibility, it makes sense for now since the only MacOS thing we sign is the agent, and this keeps the signing code from being spread across three different repos. This also streamlines the process, since we can do the signing and notarizing as part of the build, rather than having to unpack the build, sign the pkg, then repackage it into a dmg.
This new code hard-codes where to look for binaries that need signing within the agent package. This is not at all ideal, and is ripe for future improvements like moving this definition back into the vanagon project itself. But because it isn’t obvious which files need signing (i.e. binaries don’t all live in the same place, don’t all have a particular suffix), this is an easy way for us to utilize find -exec to sign all the pieces. Another improvement made here is that at every step, we verify the signing/notarizing worked correctly. This gives us an extra layer of confidence that the delivered agent will not shout obscenities at you.
As before, we sign the pkg file, but we do it before building the dmg. Then, we build the dmg and sign that, after which we notarize using the notarytool in Xcode. Usually, the notary process takes less than 2 minutes. On our initial submissions, it took days as they were being manually reviewed. Now that the system recognizes our submission, it is much faster. Once notarization is approved, we staple that notarization to the dmg and then check with Gatekeeper that it isn’t going to complain.
The Bureaucracy: D-U-N-S numbers and Overlook InfraTech, Inc.
Figuring out this whole process was quite a bear. Apple’s documentation mostly assumes you are developing with the Xcode GUI app, and not trying to do everything from the command line. It isn’t terribly clear exactly what needs to be signed and notarized and how without some more digging and trial and error.
Another thing that isn’t terribly ideal is that in order to have a developer account tied to an organization, that organization has to be a full fledged legal entity with a D-U-N-S number. If you ever start a business, you’ll find that you need a franky ridiculous amount of different identification numbers that tell people your business is, in fact, a real business, and this number is just one of many. Vox Pupuli doesn’t have this, so the agent is signed with the Overlook InfraTech, Inc. account. This has the side effect that our company name shows as the one that owns the puppet and pxp-agent background tasks. This should absolutely say Vox Pupuli and hopefully one of these days we can change that.
Final thoughts: Let’s work together
To my knowledge, Perforce does not have their MacOS 15 agent notarized, nor signed with MacOS 15’s newer stringent requirements (repeating my disclaimer: I do not know for sure, as none of us has visibility into their current build process or access to their current agents without signing a EULA which prevents you from inspecting any of the code anyway). I’d love to submit some PRs to their repos, but unfortunately, they have seemingly completely abandoned them in favor of their internal forks. That’s a bummer, because security is a team sport, and we all benefit when the ecosystem is safe. Nevertheless, I invite them to take a look at our work here to assist in getting their MacOS agents notarized, and I’d be more than happy to work with their engineers to assist. Thrilled, in fact. And I promise I’m not being sarcastic about that.
If you’re curious for more details about how we did this or how we do any of the OpenVox things, drop by the Vox Pupuli Slack or IRC, both of which you can find in the Connect menu of voxpupuli.org. I’m always happy to chat, even while I’m debugging why a binary isn’t signing properly at 2AM. If you have any other topics you’d be interested in hearing more about in this blog, drop me a line any time.