We have a freestyle job that is configured to use JDK installed from Global tools configuration for JDK. When the job runs it seems to unpack and install and takes forever and the job never seems to complete.
This used to run fine as the global tool had already installed the JDK in the tools directory for the agent and this job has run fine for months until now. It seems to be deleting and installing the JDK at the same time…?
We are also looking to any possible security changes from our organization that might be flagging the JDK version.
We are having a very very similar issue but on linux and also with an azul openjdk. It starts running and simply just keeps on downloading, deleting and unpacking the JDK over and over again. When I follow this process on the agent, You see the JDK appear and getting unpacked and after a few moments it just gets removed and the process starts again.
The timeline for OPs post and ours is just too coincidental. We havent changed anything to these pipelines in weeks. I tried finding logging but havent found anything useful. Id like to post more data but not sure what is needed. Is there a way to get more verbose logging on the pipeline?
Same problem here, on linux, and happy to have found this thread. Builds worked fine until including Wednesday evening CET. Then starting on Thursday morning we started to see build failures. It seems that every single build re-unpacks the jdk binary, although the folder actually exists on the file system:
The screenshot in my previous post is confusing since it happens to be from a build where we tried a newer JDK minor version. But it looks the same without that, here is an example:
We found a workaround: Remove the “Install automatically” option for this JDK installation under Manage Jenkins > Tools > JDK installations. That will prevent any re-download of the JDK and “solves” the problem for us for now.
It looks like something happened between Dec 11th and Dec 15th (that’s when we started seeing the “Unpacking” message repeatedly).
If you have multiple jobs running in parallel it can cause crashes too, not just a slowdown.
The following check from jdk-tool-plugin does not seem to work anymore. And it doesn’t see the already existing installation.
Running ls -al /opt/jenkins/tools/hudson.model.JDK/jdk18 shows only the .timestamp file and the extracted jdk folder … just like in the screenshot above.
It looks like the .installedByHudson marker does not get saved to disk anymore.
Not sure how to fix it, but disabling the “Install Automatically” option is not a long-term solution
We are facing the same issue since last week. But i dont understand it. We’re running on an old Jenkins LTS 2.346.3 Version, we did not changed anything this month, Even no Plugin Updates. But we see that jenkins tries to unpack the JDK Tools on every run, even if they are already on the machine.
Currently some pipelines do not work because they overwrite each other’s JDK during the run.
I’m not sure if it only faces zulujdk or also the other jdk versions we are using. I’ll let you know.
interesting why all of the sudden it started on all of my Jenkins controllers at the same time when I only updated one of them and I do see that my plugin version for JDK-tool is on version 73.vddf737284550. So, this must be bug introduced in some recent release of the plugin.
I can’t duplicate the issue as described here. I made my best guess of the configuration that was being used and created a repository that attempts to duplicate the configuration that I tested. If others can provide more details that will allow the problem to be duplicated, we may be able to fix the issue.
The repository is:
Background
On Dec 15, 2023, the Artifactory repository configuration was changed to reduce its bandwidth consumption. Instead of providing a full cache of Apache Maven Central, Jenkins developers now download dependencies from Apache Maven Central. Dependencies that are published by the Jenkins project continue to be downloaded from the Jenkins artifact repository. The details and history of that change are included in Jenkins infra help desk ticket 3842.
As a result of the Artifactory bandwidth reduction, the crawler job that generates the list of downloadable tool installers was broken. A fix was provided by adding Maven Central to the Grape search list in pull request 134.
Further improvements were made to the crawler by upgrading to the latest HTMLUnit in pull request 135 and upgrading to Java 17 in pull request 136.
The period when crawler updates were delayed might have caused an issue. The content of the crawler data files might have changed. There might be no relation between the crawler updates of Dec 15, 2023 and this report. I can’t tell without being able to duplicate the problem.
Installed Plugins:
Design Language (jenkins-design-language): 1.26.0
Azure App Service Plugin (azure-app-service): 1.0.2
MSBuild Plugin (msbuild): 1.30
JaCoCo plugin (jacoco): 3.3.2.1
Javadoc Plugin (javadoc): 226.v71211feb_e7e9
Handy Uri Templates 2.x API Plugin (handy-uri-templates-2-api): 2.1.8-22.v77d5b_75e6953
Favorite (favorite): 2.3.3.1
Blue Ocean Pipeline Editor (blueocean-pipeline-editor): 1.24.7
Config API for Blue Ocean (blueocean-config): 1.26.0
Maven Repository Scheduled Cleanup Plugin (maven-repo-cleaner): 1.2
Apache HttpComponents Client 4.x API Plugin (apache-httpcomponents-client-4-api): 4.5.13-138.v4e7d9a_7b_a_e61
Cloud Statistics Plugin (cloud-stats): 254.v47891b_b_5b_6f6
OWASP ZAP Plugin (zapper): 1.0.7
Rebuilder (rebuild): 1.32
Jira plugin (jira): 3.3
JavaScript GUI Lib: ACE Editor bundle plugin (ace-editor): 1.1
Script Security Plugin (script-security): 1229.v4880b_b_e905a_6
Azure Commons Plugin (azure-commons): 1.1.3
jQuery plugin (jquery): 1.12.4-1
Pipeline Graph Analysis Plugin (pipeline-graph-analysis): 202.va_d268e64deb_3
JQuery3 API Plugin (jquery3-api): 3.6.1-2
Pipeline: Model API (pipeline-model-api): 2.2121.vd87fb_6536d1e
Global Variable String Parameter (global-variable-string-parameter): 1.2
PowerShell plugin (powershell): 1.8
SSH plugin (ssh): 2.6.1
GitHub plugin (github): 1.33.1
JSch dependency plugin (jsch): 0.1.55.2
Short Workspace Path Plugin (short-workspace-path): 0.3
SnakeYAML API Plugin (snakeyaml-api): 1.33-90.v80dcb_3814d35
Kubernetes CLI Plugin (kubernetes-cli): 1.10.3
PTC Windchill RV&S CM - Jenkins Plugin (integrity-plugin): 2.5
Pipeline: REST API Plugin (pipeline-rest-api): 2.24
Copy Artifact Plugin (copyartifact): 1.46.4
OWASP Dependency-Track Plugin (dependency-track): 4.3.1
Pipeline: Multibranch (workflow-multibranch): 716.vc692a_e52371b_
Maven Integration plugin (maven-plugin): 3.10
Mask Passwords Plugin (mask-passwords): 3.3
Pipeline: Nodes and Processes (workflow-durable-task-step): 1217.v38306d8fa_b_5c
REST Implementation for Blue Ocean (blueocean-rest-impl): 1.24.7
Hidden Parameter plugin (hidden-parameter): 0.0.5
Trilead API Plugin (trilead-api): 1.67.vc3938a_35172f
Qualys Web App Scanning Connector (qualys-was): 2.0.11
GitHub API Plugin (github-api): 1.303-400.v35c2d8258028
Pipeline: Groovy (workflow-cps): 3536.vb_8a_6628079d5
Badge (badge): 1.9.1
Pipeline: Declarative Extension Points API (pipeline-model-extensions): 1.8.4
Azure Credentials (azure-credentials): 182.v3ccd4a755864
Job Configuration History Plugin (jobConfigHistory): 2.31-rc1107.2354f08725a_8
Bootstrap 5 API Plugin (bootstrap5-api): 5.2.0-1
WMI Windows Agents Plugin (windows-slaves): 1.8.1
MATLAB Plugin (matlab): 2.11.0
Config File Provider Plugin (config-file-provider): 3.8.0
Subversion Plug-in (subversion): 2.14.2
Jackson 2 API Plugin (jackson2-api): 2.14.2-319.v37853346a_229
Robot Framework plugin (robot): 3.0.1
Infrastructure plugin for Publish Over X (publish-over): 0.22
GitHub Pipeline for Blue Ocean (blueocean-github-pipeline): 1.24.7
NodeJS Plugin (nodejs): 1.5.1
Mercurial plugin (mercurial): 2.15
Git Pipeline for Blue Ocean (blueocean-git-pipeline): 1.24.7
Extended Read Permission Plugin (extended-read-permission): 3.2
Pipeline: Shared Groovy Libraries (workflow-cps-global-lib): 2.19
Authentication Tokens API Plugin (authentication-tokens): 1.4
Pipeline: Input Step (pipeline-input-step): 466.v6d0a_5df34f81
Git client plugin (git-client): 3.7.1
Pipeline: Stage Tags Metadata (pipeline-stage-tags-metadata): 2.2121.vd87fb_6536d1e
Variant Plugin (variant): 59.vf075fe829ccb
GitHub Branch Source Plugin (github-branch-source): 2.10.4
Analysis Model API Plugin (analysis-model-api): 10.17.0
Pipeline: Job (workflow-job): 1207.ve6191ff089f8
OctoPerf Load Testing Plugin. (octoperf): 4.3.1
Pipeline: Step API (workflow-step-api): 639.v6eca_cd8c04a_a_
Azure Key Vault Plugin (azure-keyvault): 122.v72c88cfc0c2b
Generic Webhook Trigger Plugin (generic-webhook-trigger): 1.86.4
Docker Commons Plugin (docker-commons): 1.19
JAXB plugin (jaxb): 2.3.7-1
JUnit Plugin (junit): 1.49
Flyway Runner (flyway-runner): 1.9
Pipeline Utility Steps (pipeline-utility-steps): 2.8.0
Timestamper (timestamper): 1.21
Branch API Plugin (branch-api): 2.1071.v1a_188a_562481
Events API for Blue Ocean (blueocean-events): 1.24.7
Font Awesome API Plugin (font-awesome-api): 6.1.2-1
Maven Release Plug-in Plug-in (m2release): 0.16.3
OWASP Markup Formatter Plugin (antisamy-markup-formatter): 159.v25b_c67cd35fb_
Pipeline: Declarative (pipeline-model-definition): 1.8.4
Display URL API (display-url-api): 2.3.6
Popper.js API Plugin (popper-api): 1.16.1-3
Audit Trail (audit-trail): 3.10
Publish Over FTP (publish-over-ftp): 1.17
ChuckNorris Plugin (chucknorris): 1.4
Ranorex Test Execution Plugin (ranorex-integration): 1.0.2
Parameter Separator Plugin (parameter-separator): 1.3
Performance Plugin (performance): 3.20
Plain Credentials Plugin (plain-credentials): 1.8
Resource Disposer Plugin (resource-disposer): 0.20
NAnt Plugin (nant): 1.4.3
Pipeline Remote Loader Plugin (workflow-remote-loader): 1.5
Xcode integration (xcode-plugin): 2.0.15
GIT server Plugin (git-server): 1.9
JWT for Blue Ocean (blueocean-jwt): 1.26.0
Git Parameter Plug-In (git-parameter): 0.9.13
JIRA Integration for Blue Ocean (blueocean-jira): 1.26.0
Dashboard for Blue Ocean (blueocean-dashboard): 1.26.0
ECharts API Plugin (echarts-api): 5.4.0-1
Translation Assistance plugin (translation): 1.16
JavaMail API (javax-mail-api): 1.6.2-8
Blue Ocean (blueocean): 1.24.7
Pipeline: SCM Step (workflow-scm-step): 408.v7d5b_135a_b_d49
Display URL for Blue Ocean (blueocean-display-url): 2.4.1
Filesystem List Parameter Plug-in (filesystem-list-parameter-plugin): 0.0.8
Checks API plugin (checks-api): 1.7.0
Pipeline: Stage Step (pipeline-stage-step): 305.ve96d0205c1c6
Metrics Plugin (metrics): 4.2.13-420.vea_2f17932dd6
OkHttp Plugin (okhttp-api): 4.9.3-108.v0feda04578cf
Run Condition Plugin (run-condition): 1.5
SCM API Plugin (scm-api): 631.v9143df5b_e4a_a
Email Extension Plugin (email-ext): 2.83
Pipeline SCM API for Blue Ocean (blueocean-pipeline-scm-api): 1.26.0
Matrix Authorization Strategy Plugin (matrix-auth): 3.1.5
JavaScript GUI Lib: Moment.js bundle plugin (momentjs): 1.1.1
Kubernetes Credentials Plugin (kubernetes-credentials): 0.9.0
SiteMonitor Plugin (sitemonitor): 0.6
Embeddable Build Status Plugin (embeddable-build-status): 304.vdcf48d6b_d2eb
Job Restrictions Plugin (job-restrictions): 0.8
Plugin Utilities API Plugin (plugin-util-api): 2.17.0
Credentials Binding Plugin (credentials-binding): 1.27.1
Active Directory plugin (active-directory): 2.31
Azure VM Agents (azure-vm-agents): 780.v50d067d02f76
Gatling Jenkins Plugin (gatling): 1.3.0
File Operations Plugin (file-operations): 1.11
GitLab Plugin (gitlab-plugin): 1.5.20
Gradle Plugin (gradle): 1.39.4
Caffeine API Plugin (caffeine-api): 2.9.3-65.v6a_47d0f4d1fe
Folders Plugin (cloudbees-folder): 6.722.v8165b_a_cf25e9
Ansible plugin (ansible): 1.1
Warnings Next Generation Plugin (warnings-ng): 9.1.0
Matrix Project Plugin (matrix-project): 1.20
Pipeline: API (workflow-api): 1200.v8005c684b_a_c6
Zentimestamp plugin (zentimestamp): 4.2 DataTables.net API Plugin (data-tables-api): 1.12.1-4
Pipeline implementation for Blue Ocean (blueocean-pipeline-api-impl): 1.24.7
Structs Plugin (structs): 324.va_f5d6774f3a_d
CVS Plug-in (cvs): 2.19.1
Ant Plugin (ant): 481.v7b_09e538fcca
commons-lang3 v3.x Jenkins API Plugin (commons-lang3-api): 3.12.0-36.vd97de6465d5b_
JavaBeans Activation Framework (JAF) API (javax-activation-api): 1.2.0-5
Groovy Postbuild (groovy-postbuild): 2.5
Server Sent Events (SSE) Gateway Plugin (sse-gateway): 1.25
Forensics API Plugin (forensics-api): 1.0.0
Kubernetes Client API Plugin (kubernetes-client-api): 6.4.1-208.vfe09a_9362c2c
Pub-Sub “light” Bus (pubsub-light): 1.16
MapDB API Plugin (mapdb-api): 1.0.9-28.vf251ce40855d
Validating String Parameter Plugin (validating-string-parameter): 2.8
Active Choices Plug-in (uno-choice): 2.6.5
Extended Choice Parameter Plugin (extended-choice-parameter): 359.v35dcfdd0c20d
jQuery UI plugin (jquery-ui): 1.0.2
SonarQube Scanner for Jenkins (sonar): 2.14
Ionicons API (ionicons-api): 31.v4757b_6987003
SSH Pipeline Steps (ssh-steps): 2.0.0
Role-based Authorization Strategy (role-strategy): 587.588.v850a_20a_30162
description setter plugin (description-setter): 1.10
Credentials Plugin (credentials): 2.6.1.1
PAM Authentication plugin (pam-auth): 1.10
Workspace Cleanup Plugin (ws-cleanup): 0.42
Plot plugin (plot): 2.1.11
OWASP Dependency-Check Plugin (dependency-check-jenkins-plugin): 5.1.2
Durable Task Plugin (durable-task): 507.v050055d0cb_dd
TraceTronic ECU-TEST Plugin (ecutest): 2.32
Popper.js 2 API Plugin (popper2-api): 2.11.6-2
Java JSON Web Token (JJWT) Plugin (jjwt-api): 0.11.5-77.v646c772fddb_0
i18n for Blue Ocean (blueocean-i18n): 1.26.0
Pipeline: Stage View Plugin (pipeline-stage-view): 2.24
Jakarta Activation API (jakarta-activation-api): 2.0.1-2
Bootstrap 4 API Plugin (bootstrap4-api): 4.6.0-6
External Monitor Job Type Plugin (external-monitor-job): 203.v683c09d993b_9
Azure Storage plugin (windows-azure-storage): 380.va3a027b784f4
Bitbucket Pipeline for Blue Ocean (blueocean-bitbucket-pipeline): 1.24.7
Pipeline: Supporting APIs (workflow-support): 839.v35e2736cfd5c
AnsiColor (ansicolor): 1.0.2
SSH server (sshd): 3.0.3
Web for Blue Ocean (blueocean-web): 1.26.0
Personalization for Blue Ocean (blueocean-personalization): 1.26.0
Jakarta Mail API (jakarta-mail-api): 2.0.1-2
JavaScript GUI Lib: jQuery bundles (jQuery and jQuery UI) plugin (jquery-detached): 1.2.1
Command Agent Launcher Plugin (command-launcher): 90.v669d7ccb_7c31
JavaScript GUI Lib: Handlebars bundle plugin (handlebars): 3.0.8
Lockable Resources plugin (lockable-resources): 2.10
Nexus Artifact Uploader (nexus-artifact-uploader): 2.13
Token Macro Plugin (token-macro): 321.vd7cc1f2a_52c8
Parameterized Trigger plugin (parameterized-trigger): 2.40
Crowd 2 Integration (crowd2): 3.2.1
Pipeline: Milestone Step (pipeline-milestone-step): 111.v449306f708b_7
Build Timeout (build-timeout): 1.24
Datadog Plugin (datadog): 3.5.2
Docker API Plugin (docker-java-api): 3.1.5.2
Azure SDK API Plugin (azure-sdk): 118.v43f74dd9ca_dc
Pipeline (workflow-aggregator): 2.6
Common API for Blue Ocean (blueocean-commons): 1.26.0
Oracle Java SE Development Kit Installer Plugin (jdk-tool): 63.v62d2fd4b_4793
Pipeline: GitHub Groovy Libraries (pipeline-github-lib): 1.0
Autofavorite for Blue Ocean (blueocean-autofavorite): 1.2.5
Schedule Build Plugin (schedule-build): 345.vc2d8f6ef9182
Docker Pipeline (docker-workflow): 1.26
Cucumber reports (cucumber-reports): 5.7.5
External Workspace Manager Plugin (external-workspace-manager): 1.2.3
Conditional BuildStep (conditional-buildstep): 1.4.1
Pipeline: Basic Steps (workflow-basic-steps): 994.vd57e3ca_46d24
HTML Publisher plugin (htmlpublisher): 1.25
Mailer Plugin (mailer): 435.438.v5b_81173f5b_a_1
bouncycastle API Plugin (bouncycastle-api): 2.27
Pipeline: Build Step (pipeline-build-step): 488.v8993df156e8d
Blue Ocean Core JS (blueocean-core-js): 1.26.0
LDAP Plugin (ldap): 659.v8ca_b_a_fe79fa_d
SSH Credentials Plugin (ssh-credentials): 1.18.2
Git plugin (git): 4.7.1
Bitbucket Branch Source Plugin (cloudbees-bitbucket-branch-source): 2.9.9
HTTP Request Plugin (http_request): 1.16
REST API for Blue Ocean (blueocean-rest): 1.26.0
SSH Build Agents plugin (ssh-slaves): 1.32.0
Problem:
On every run, jenkins unpacks the zuluJDK Version again and again.
I checked before a run whether the jdk version is already in the tools folder on the node, and yes the files are all there and also have the correct authorization for the jenkins service user.
We did not make any changes to our Jenkins systems in December. I therefore don’t understand why this problem is occurring all of a sudden. The problem occurs on all of our Linux nodes (we have 4)
I also don’t understand why this issue is resolved. I have updated to Jenkins 2.437 and i still have this issue. My builds fail as soon as two builds run in parallel due to the JDK being overwritten. Where can i find the ID of JDK, so that i can create the .installedByHudson file by hand and have a workaround for now?
It seems the cdn.azul.com is not setting the last-modified header in the http response. This field is used by the tool installer to check whether a tool needs to be updated or not.
When I download the jdk in Jenkins a file .timestamp is create that has 1.1.1970 as last modified.
If using an adoptium jdk that downloads from github, the problem should not be there.
Downloading a tool installer from outside your own Jenkins environment is not recommended. It makes your Jenkins environment reliant on the network and on the provider of that download. The Jenkins JDK tool installer plugins (Eclipse Temurin and Oracle) include the following usage recommendations:
Usage recommendations
We want to warn that this plugin is NOT a good practice for production environments. As it relies on the Oracle’s website to do the job, it’s highly likely to stop working. It could happen because Oracle’s website change or even if Oracle bans our downloads due to excessive bandwidth or whatever other reason).
The recommended approach is to download the JDK distribution using other installers, for example downloading it from a well known URL (preferably hosted on your own network) with ZIP Tool Installer, having it pre-installed in agent docker images, or executing a script to do the job.
But as far as I can tell, it only happens when attempting to download from the Azul content delivery network, not from a locally hosted web server. At least I’m unable to duplicate the issue when downloading from the nginx server that I run locally with JDK tar and zip files.