Migrate MacOS Jenkins agents to Azure DevOps agents

Information and motivation

I am working right now on a migration from a Jenkins build pipeline to an Azure DevOps pipeline.

Some details on the environment:

• On premise Azure DevOps Server 2020 bundled with on premise Azure Pipeline agents are used
• Azure Pipeline agents were already used in the past for build of non-C++ and rather small projects
• Existing Azure Pipeline agents are Linux or Docker based only

The product to be migrated is a big C++ project.

Some details on the project:

• Commercial product
• private git repository hosted in Azure DevOps Server
• Pull requests with quality gates to multiple production branches
• Dedicated builds for Windows, Debian, RedHat EL/Oracle Linux, Android, iOS
• Distributed object cache (sccache stored into redis in memory db) is enabled to have reasonable build times

iOS has maybe caught the eye. To migrate this pipeline an Azure Pipeline agent for MacOS is required.

I came up with a rather small commandset to download and manually run the agent and a first test was successful (build and sign the iOS app):

# Download and extract the agent

# Find the correct URL from https://github.com/Microsoft/azure-pipelines-agent/releases

curl --output /tmp/vsts.tar.gz \

https://vstsagentpackage.azureedge.net/agent/2.204.0/vsts-agent-osx-x64-2.204.0.tar.gz

cd /azagent && tar xvf /tmp/vsts.tar.gz

# configure the agent

./config.sh --unattended --replace --url <azureurl> --auth pat --token <azuretoken> \

--agent <agentname> --pool <poolname> --acceptTeeEula

./run.sh

Unattended startup

When working with Jenkins, Jenkins started it´s agent by opening an SSH session to the MacOS host and Jenkins took care to start the agent via that SSH connection.

With Azure Pipeline agents this is not possible and one needs to start those agents somehow yourself, but I have not seen a good example on Microsoft documentation about it, just some hint about Launch Agents.

Upon taking a deeper look at that, I noticed, that Launch Agents are only started once a user logs in. This is not suitable as direct/graphical access to those machines is not desired.

The alternative are LaunchDaemons so after googling around I created my first launch daemon plist file and saved it to /Library/LaunchDaemons/com.microsoft.azagent.plist:

<?xml version="1.0" encoding="UTF-8"?>

<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">

<plist version="1.0">

    <key>OnDemand</key>

    <false/>

    <key>UserName</key>

    <string>yourusername</string>

    <key>RunAtLoad</key>

    <true/>

    <key>WorkingDirectory</key>

    <string>/azagent/</string>

    <key>EnvironmentVariables</key>

    <!-- need to modify the path since some tools are installed there

         and not everything is set the same way than in a regular shell -->

    <dict>

           <key>PATH</key>

           <string>/usr/local/bin:/usr/bin:/bin:/usr/sbin:/sbin</string>

    </dict>

    <key>ProgramArguments</key>

    <array>

        <string>/azagent/run.sh</string>

    </array>

</dict>

</plist>

By running

sudo launchctl load /Library/LaunchDaemons/com.microsoft.azagent.plist

the daemon started and connection was successful, also after a reboot.

It does not work

After running a build some problems were observed in comparison to manually running the agent.

Unlock keychain

In general, to sign an iOS app a certificate needs to be available in a keychain, this is done by the following command before signing:

security -v unlock-keychain -p <password>

When running the agent via daemon, an error was thrown during above command:

security: SecKeychainUnlock <NULL>: The user name or passphrase you entered is not correct.

To solve this error the a property was added to the plist file:

    <key>SessionCreate</key>

    <true/>

Certificate access

With access to the keychain the build presented and error during signing which is widely known:

Command /usr/bin/codesign failed with exit code 1 (errSecInternalComponent)

MacOS has some additional check enabled for command line access to those signing certificates.

This error message is typically mitigated by login into the GUI session and sign something in that session via command line. A popup asks for permission which you "Always allow":

After that, at least for the Jenkins agent or a remote SSH connection, signing from the command line works after unlocking the keychain.

But this was not the case for the Azure Pipeline agent started via LaunchDaemon.

Tests using custom keychains did not work either resulted in the same error message or others like:

error: No signing certificate "xxx Development" found: No "iOS Development" signing certificate matching team ID "xxxxxxxxxx" with a private key was found. (in target 'xx' from project 'xxxxxxxx')

But this was not the case for the Azure Pipeline agent started via LaunchDaemon.

A different approach (which works)

During my research and tests I came across a post suggesting to let the LaunchDaemon run an SSH connection to localhost to get a proper session. I knew already that it works when running the agent manually via SSH, but how to automate that?

The idea is to have a password-less SSH connection to localhost and automatically run the agent. This can be achieved by the following commands:

# generate a new keypair, just press enter 3 times to use the defaults

ssh-keygen -t rsa

# authorize ourselves to login without password, but only from localhost and then

# only the agent will be started

{ printf 'from="::1",command="AZP_AGENT_DOWNGRADE_DISABLED=true /azagent/run.sh" ' && \

cat ~/.ssh/id_rsa.pub; } >> ~/.ssh/authorized_keys

# remove rights for other users

chmod og-wx ~/.ssh/authorized_keys

# connect once to test and to store the host in the known hosts lists (confirm with yes)

ssh localhost

Also the plist file needs to be adapted slightly

<?xml version="1.0" encoding="UTF-8"?>

<!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">

<plist version="1.0">

<dict>

    <key>Label</key>

    <string>com.microsoft.azagent</string>

    <key>OnDemand</key>

    <false/>

    <key>UserName</key>

    <string>yourusername</string>

    <key>RunAtLoad</key>

    <true/>

    <key>ProgramArguments</key>

    <array>

        <string>ssh</string>

        <!-- double -t so tty is forced even if the daemon does not have tty

             otherwise SIGTERM will not be forwarded when the daemon is unloaded -->

        <string>-t</string>

        <string>-t</string>

        <string>localhost</string>

    </array>

</dict>

</plist>

With those adaptations finally everything worked on MacOS build agents. Hooray!