Building containers in Jenkins agent with rootless buildah on latest Alpine v3.24

Jenkins setup:

  • Jenkins agent on Alpine Linux
  • The Jenkins agent image also contains buildah
  • buildah runs in rootless mode as jenkins user

This question might not be strictly related to Jenkins (agent) only.

`buildah bud` (and `buildah info` / `buildah containers`) commands failed with this error:

Error: mkdir /run/containers: permission denied
WARN[0000] failed to shutdown storage: "mkdir /run/containers: permission denied"

I first thought it relates to the Jenkins agent (remoting) version (Index of public/org/jenkins-ci/main/remoting), but it resolved by picking an older buildah version.

The error could be observed with these versions:

  • buildah 1.44.0
  • Alpine 3.24

See Package index - Alpine Linux packages

The error was resolved when I downgraded buildah to the latest version of the alpine 3.23 release, Package index - Alpine Linux packages

  • buildah 1.42.2
  • Alpine 3.24 (or 3.23)

I was unable to find any references in the buildah issues that would directly relate the error above.

Do you have similar setup or experiences with building containers in rootless buildah on a Jenkins agent?

My workaround is to stay on the old buildah version 1.42.2 until newer Alpine Linux releases publish more recent buildah packages (which might fix the issue). I would be very interested in finding out the root cause though.