Remoting Agent with Java 17 & Alpine causing incorrect Java version to be picked up

Using Jenkins Ask a question
1

Background:

We run AMAZON ECS Fargate based agents in combination with remoting jars. Basically a customized inbound agent image.

Recently we were trying to upgrade Jenkins Controller and Build agents to Java 17. Since we have several (1000+) projects which still use Java 8, we decided to add a copy of JDK 17 in agent image and keep Java 8 as the installed version

Java 8 is installed with apk add openjdk8

Java 17 zip [Temurin] is downloaded and extracted to /opt/jre17/ which the remoting agent uses.

Issue:

When using Java 17 in Alpine OS with Remoting agent, it fails to recognize the installed Java 8 version.

That is if we run a test job, we will get below output:

+ java -version
openjdk version "17.0.10" 2024-01-16 LTS
OpenJDK Runtime Environment Corretto-17.0.10.7.1 (build 17.0.10+7-LTS)
OpenJDK 64-Bit Server VM Corretto-17.0.10.7.1 (build 17.0.10+7-LTS, mixed mode, sharing)

+ which java
/usr/lib/jvm/java-1.8-openjdk/bin/java

That is, even when java binary is pointing to Java 8, we are still getting version as 17.

But if we set Java 11 version for the Remoting Agent, this works as expected, see below

+ java -version
openjdk version "1.8.0_392"
OpenJDK Runtime Environment (IcedTea 3.29.0) (Alpine 8.392.08-r1)
OpenJDK 64-Bit Server VM (build 25.392-b08, mixed mode)
+ which java
/usr/lib/jvm/java-8-openjdk/bin/java

This issue is not coming, when the base image is Ubuntu 22.

Steps to Replicate:

Start a local container instance of Alpine

sudo docker run -it alpine:latest /bin/sh

Install wget, curl, tar, xz, openjdk8

apk update && apk add tar xz openjdk8 wget curl

Verify Java verison

java -version
javac -version

Download & Extract Latest Temurin JRE 17 to a seperate folder (lets say /opt/jre17)

mkdir /opt/jre17 

wget -q -O - "https://github.com/adoptium/temurin17-binaries/releases/download/jdk-17.0.10%2B7/OpenJDK17U-jre_x64_alpine-linux_hotspot_17.0.10_7.tar.gz" | tar -xz --strip=1 -C /opt/jre17

Download latest remoting JAR (As of today) and save it to a folder, lets say /agent

(either from your Jenkins Instance or latest version from Remoting Artifactory Repo)

mkdir /agent

curl -sO https://<JenkinsURL>/jnlpJars/agent.jar

Create a Jenkins Permanent agent in Controller & configure the agent with Websocket

Start the newly configured agent

/opt/jre17/bin/java agent.jar <rest of the secret and workdir configuration>

Run a test job with newly configured agent, run java -version

Repeat the same with JRE 11 ( Can be in same container instance)

mkdir /opt/jre11 
wget -q -O - "https://github.com/adoptium/temurin11-binaries/releases/download/jdk-11.0.21%2B9/OpenJDK11U-jre_x64_alpine-linux_hotspot_11.0.21_9.tar.gz" | tar -xz --strip=1 -C /opt/jre11

/opt/jre11/bin/java agent.jar <rest of the secret and workdir configuration>

Run a test job with newly configured agent, run java -version

I am not sure if I am doing some wrong, or if this is a bug. Any insights will be most appreciated !!

Jenkins Setup

Jenkins: 2.426.3
OS: Linux - 4.14.334-252.552.amzn2.x86_64
Java: 17.0.10 - Amazon.com Inc. (OpenJDK 64-Bit Server VM)
2

I can reproduce the problem with openjdk 17 and 21, with the difference that I’m using ssh to connect to the agent.
I don’t think that this is a Jenkins specific problem but more a problem with openjdk in alpine that is using musl. I can see an environment variable injected
LD_LIBRARY_PATH=/opt/jre17/lib/server:/opt/sapmachine-jdk-21.0.2/lib:/opt/sapmachine-jdk-21.0.2/../lib
which probably comes from the jdk as it is already in the env of the java process itself.
When connecting an agent via ssh, the current environment is printed before starting the Java process, there LD_LIBRARY_PATH is not listed. But when looking that the environment of the agent java process by going to System information of the agent, the it is listed with above values.
I guess that this causes an interception of all calls to processes with name java so that they get replaced by the corresponding java17 executable.
Unsetting the env variable with
unset LD_LIBRARY_PATH in a sh step before calling java -version solves the problem.
Unfortunately it is not possible to set empty variables in the env variables section of a node, but you could set it to a dummy value there.

Toolchain feature detects Java 8 JDK as a different Java version when inside an Alpine Linux container. · Issue #24300 · gradle/gradle · GitHub reports the same issue for gradle

1 Like