Go Modules from the private repository in the pipeline

Go Modules is a significant step forward in making the Golang dependency management “humanly natural”. However, when I recently tried to convert a project to Go 1.12 forcing it to use the Go Modules I came across a nasty issue. My problem was that some of the “in house” built modules are kept in the private Github repositories.

Before 1.12 it was not an issues since all we had to do to solve the problem is to clone the required repositories to the $GOPATH/src directory. It is not a huge issue in the development environment either since we can do the same locally.

However, when it comes to the building the code in the CI/CD pipeline it becomes a bit more complicated. Of course, the same approach could be used but then it is not really using the ability of the latest Golang version to resolve dependencies itself.

By default, Golang uses https to access the dependant repository, but it does not look like it provides means to send the credentials with the request. Two discussions of the similar issue here and here offer a good solution instructing git to use ssh protocol instead, which implies that the user that runs the command will have their ssh private key setup and the corresponding public key is added to the authorised keys set in Github settings.

All you need to do then is to execute the following command in the pipeline:

git config --global url.ssh://git@github.com/.insteadOf https://github.com/

For the standalone CI agent it is enough to proceed, however the issue gets more complicated in the non-standard setup.

In our build environment we use Jenkins to run Continuous Integration and Delivery. Jenkins in its turn uses the Kubernetes plugin that executes jobs in the dynamically created pods. So, in essence when the job is started we always have a vanilla Golang environment in one of the pod containers, hence Go has to resolve dependencies somehow.

We won’t entertain the idea of keeping the private keys inside of any container image: it is too big of the security risk.

Instead, as it is a standard practice for Kubernetes to keep sensitive information such as private keys or passwords in a secret. Then we use a Jenkins Kuberentes plugin secret volume to mount content of the private key to the pods.

The issue with this approach is that up until recently there was no way to control permissions of the mounted secret volume in the Pod Template. That is the problem for the ssh private key as it must only be accessible by the owner and not the ‘group’ or the ‘world’, i.e. effective permission must be set to 0600 or 0400.

To handle this situation we would normally use the following approach. The private key is mounted to the temporary location for example like this:

podTemplate(label: 'jpod', cloud: 'OpenShift', serviceAccount: 'jenkins',
  containers: [
    containerTemplate(name: 'golang', image: 'golang:1.12.5', ttyEnabled: true, command: 'cat'),
    containerTemplate(name: 'sonarqube', image: 'iktech/sonarqube-scanner', ttyEnabled: true, command: 'cat'),
  ],
  volumes: [
    secretVolume(mountPath: '/etc/.ssh', secretName: 'ssh-home'),
  ]
)

And then in the preparation stage we would copy the key to the correct place with correct permissions:

stage('Prepare') {
    checkout scm
    // Set up private key to access GitHub
    sh "cat /etc/.ssh/id_rsa > ~/.ssh/id_rsa"
    sh "chmod 400 ~/.ssh/id_rsa"
}

The problem is ‘Prepare’ stage is executed in the ‘jnlp’ container which is running as user ‘jenkins’. However, the golang container in the above setup will run as a user ‘root’ and other containers can run as other users.

On approach to handle this is to modify the containers that are used by the Jenkins pod forcing them to run as the ‘jenkins’ user. It is probably the cleanest solution but this means that we have to maintain more images.

Alternative is to copy the /home/jenkins/.ssh directory cloning it to respective user home directory. So, if the container is executed by the ‘root’ user we run the following command in the container in question:

cp -R /home/jenkins/.ssh /root

So, the dependency resolution stage will look like the following:

container('golang') {

        stage('Install Dependencies') {
            sh 'git config --global url.ssh://git@github.com/.insteadOf https://github.com/'
            sh 'cp -R /home/jenkins/.ssh /root'
            sh 'go test'
            // Install dependencies
            sh 'curl -sfL https://install.goreleaser.com/github.com/golangci/golangci-lint.sh | sh -s -- -b $(go env GOPATH)/bin latest'
            sh 'go get -u github.com/jstemmer/go-junit-report'
            sh 'go get github.com/axw/gocov/...'
            sh 'go get github.com/AlekSi/gocov-xml'
        }
}

The ‘go test’ command in this case will download the project dependencies as per go.mod file.

Alternatively, the Kubernetes Jenkins plugin recently introduced the ability to set the permission of the mounted secret volume so the volume in the above pod template example could be instead defined as the following:

volumes: [
    secretVolume(mountPath: '/root/.ssh', secretName: 'ssh-home', defaultMode: '256'),
  ]

Please note that ‘defaultMode’ here is a decimal number, hence the standard octal access e.g. 0400 must be converted to the decimal number. For 0400 access mode the corresponding decimal number is 256, for 0600 it is 384. Also, keep in mind that defaultMode value must be a string.