Does Jenkins wipe out plugin config in config.xml if unable to inspect plugin?

sslk · June 21, 2023, 1:22am

We have Jenkins instances running in a Kubernetes cluster. These instances have the Kubernetes plugin installed.
When upgrading the instance LTS version (by updating the tag in the Jenkins helm chart), we see an issue where the config.xml file wipes out the <clouds> configuration. In the logs, I see an exception related to the Plugin Manager trying to inspect the Kubernetes plugin:
2023-06-12 23:58:44.011+0000 [id=45] SEVERE jenkins.InitReactorRunner$1#onTaskFailed: Failed Inspecting plugin /var/jenkins/plugins/kubernetes.jpi Also: Also: java.nio.file.FileSystemException: /var/jenkins/plugins/kubernetes/WEB-INF/lib/.nfs00000000bad93ee400001f77: Device or resource busy at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:100) at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) at java.base/sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:249) at java.base/sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:110) at java.base/java.nio.file.Files.deleteIfExists(Files.java:1181) at jenkins.util.io.PathRemover.removeOrMakeRemovableThenRemove(PathRemover.java:234) Also: java.nio.file.FileSystemException: /var/jenkins/plugins/kubernetes/WEB-INF/lib/.nfs00000000bad93ee400001f77: Device or resource busy at java.base/sun.nio.fs.UnixException.translateToIOException(UnixException.java:100) at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:111) at java.base/sun.nio.fs.UnixException.rethrowAsIOException(UnixException.java:116) at java.base/sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:249) at java.base/sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:110) at java.base/java.nio.file.Files.deleteIfExists(Files.java:1181) at jenkins.util.io.PathRemover.removeOrMakeRemovableThenRemove(PathRemover.java:238) jenkins.util.io.CompositeIOException: Unable to remove file /var/jenkins/plugins/kubernetes/WEB-INF/lib/.nfs00000000bad93ee400001f77 at jenkins.util.io.PathRemover.removeOrMakeRemovableThenRemove(PathRemover.java:249) at jenkins.util.io.PathRemover.tryRemoveFile(PathRemover.java:202) at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:213) at jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:223) at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:212) at jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:223) at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:212) at jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:223) at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:212) at jenkins.util.io.PathRemover.forceRemoveRecursive(PathRemover.java:93) Also: Also: java.nio.file.DirectoryNotEmptyException: /var/jenkins/plugins/kubernetes/WEB-INF/lib at java.base/sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:247) at java.base/sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:110) at java.base/java.nio.file.Files.deleteIfExists(Files.java:1181) at jenkins.util.io.PathRemover.removeOrMakeRemovableThenRemove(PathRemover.java:234) Also: java.nio.file.DirectoryNotEmptyException: /var/jenkins/plugins/kubernetes/WEB-INF/lib at java.base/sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:247) at java.base/sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:110) at java.base/java.nio.file.Files.deleteIfExists(Files.java:1181) at jenkins.util.io.PathRemover.removeOrMakeRemovableThenRemove(PathRemover.java:238) jenkins.util.io.CompositeIOException: Unable to remove directory /var/jenkins/plugins/kubernetes/WEB-INF/lib with directory contents: [/var/jenkins/plugins/kubernetes/WEB-INF/lib/.nfs00000000bad93ee400001f77] at jenkins.util.io.PathRemover.removeOrMakeRemovableThenRemove(PathRemover.java:247) at jenkins.util.io.PathRemover.tryRemoveFile(PathRemover.java:202) at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:213) at jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:223) at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:212) at jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:223) at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:212) at jenkins.util.io.PathRemover.forceRemoveRecursive(PathRemover.java:93) Also: Also: java.nio.file.DirectoryNotEmptyException: /var/jenkins/plugins/kubernetes/WEB-INF at java.base/sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:247) at java.base/sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:110) at java.base/java.nio.file.Files.deleteIfExists(Files.java:1181) at jenkins.util.io.PathRemover.removeOrMakeRemovableThenRemove(PathRemover.java:234) Also: java.nio.file.DirectoryNotEmptyException: /var/jenkins/plugins/kubernetes/WEB-INF at java.base/sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:247) at java.base/sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:110) at java.base/java.nio.file.Files.deleteIfExists(Files.java:1181) at jenkins.util.io.PathRemover.removeOrMakeRemovableThenRemove(PathRemover.java:238) jenkins.util.io.CompositeIOException: Unable to remove directory /var/jenkins/plugins/kubernetes/WEB-INF with directory contents: [/var/jenkins/plugins/kubernetes/WEB-INF/lib] at jenkins.util.io.PathRemover.removeOrMakeRemovableThenRemove(PathRemover.java:247) at jenkins.util.io.PathRemover.tryRemoveFile(PathRemover.java:202) at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:213) at jenkins.util.io.PathRemover.tryRemoveDirectoryContents(PathRemover.java:223) at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:212) at jenkins.util.io.PathRemover.forceRemoveRecursive(PathRemover.java:93) Also: Also: java.nio.file.DirectoryNotEmptyException: /var/jenkins/plugins/kubernetes at java.base/sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:247) at java.base/sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:110) at java.base/java.nio.file.Files.deleteIfExists(Files.java:1181) at jenkins.util.io.PathRemover.removeOrMakeRemovableThenRemove(PathRemover.java:234) Also: java.nio.file.DirectoryNotEmptyException: /var/jenkins/plugins/kubernetes at java.base/sun.nio.fs.UnixFileSystemProvider.implDelete(UnixFileSystemProvider.java:247) at java.base/sun.nio.fs.AbstractFileSystemProvider.deleteIfExists(AbstractFileSystemProvider.java:110) at java.base/java.nio.file.Files.deleteIfExists(Files.java:1181) at jenkins.util.io.PathRemover.removeOrMakeRemovableThenRemove(PathRemover.java:238) jenkins.util.io.CompositeIOException: Unable to remove directory /var/jenkins/plugins/kubernetes with directory contents: [/var/jenkins/plugins/kubernetes/WEB-INF] at jenkins.util.io.PathRemover.removeOrMakeRemovableThenRemove(PathRemover.java:247) at jenkins.util.io.PathRemover.tryRemoveFile(PathRemover.java:202) at jenkins.util.io.PathRemover.tryRemoveRecursive(PathRemover.java:213) at jenkins.util.io.PathRemover.forceRemoveRecursive(PathRemover.java:93) jenkins.util.io.CompositeIOException: Unable to delete '/var/jenkins/plugins/kubernetes'. Tried 3 times (of a maximum of 3) waiting 0.1 sec between attempts. at jenkins.util.io.PathRemover.forceRemoveRecursive(PathRemover.java:96) at hudson.Util.deleteRecursive(Util.java:320) at hudson.Util.deleteRecursive(Util.java:309) at hudson.ClassicPluginStrategy.explode(ClassicPluginStrategy.java:493) at hudson.ClassicPluginStrategy.createPluginWrapper(ClassicPluginStrategy.java:170) at hudson.PluginManager$1$3$1.run(PluginManager.java:447) at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:177) at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:305) at jenkins.model.Jenkins$5.runTask(Jenkins.java:1164) at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:221) at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:120) at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68) at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128) at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628) at java.base/java.lang.Thread.run(Thread.java:829)

Is it possible that the PluginManager failing to inspect the Kubernetes plugin files is wiping out the <clouds> portion of the config.xml?

halkeye · June 21, 2023, 2:24am

no, but the helm chart populates it.

github.com

jenkinsci/helm-charts/blob/ffa95d9792912f4a69bb18a6e90634660d8f880f/charts/jenkins/templates/_helpers.tpl#L141


      
          {{- end }}
          {{- end }}
          markupFormatter:
            {{- if .Values.controller.enableRawHtmlMarkupFormatter }}
            rawHtml:
              disableSyntaxHighlighting: true
            {{- else }}
            {{- toYaml .Values.controller.markupFormatter | nindent 4 }}
            {{- end }}
          clouds:
          - kubernetes:
              containerCapStr: "{{ .Values.agent.containerCap }}"
              defaultsProviderTemplate: "{{ .Values.agent.defaultsProviderTemplate }}"
              connectTimeout: "{{ .Values.agent.kubernetesConnectTimeout }}"
              readTimeout: "{{ .Values.agent.kubernetesReadTimeout }}"
              {{- if .Values.agent.directConnection }}
              directConnection: true
              {{- else }}
              {{- if .Values.agent.jenkinsUrl }}
              jenkinsUrl: "{{ tpl .Values.agent.jenkinsUrl . }}"
              {{- else }}

I personally recommend not using the helm chart’s jcasc support, disable it, and manage your own config as code.

sslk · June 21, 2023, 4:23pm

Thanks @halkeye , so we are populating the config.xml <clouds> section ourselves, we are not using jcasc support. The problem is that when we upgrade the instance LTS and plugin versions by changing the helm chart, the clouds config gets wiped out.

The exception I posted happens after we upgrade, and then in the Jenkins instance UI, we can click a ‘Correct’ button and then the plugins are loaded successfully. My guess is that when we upgrade and get this exception, Jenkins removes this configuration in the config.xml because it’s unable to load the plugin. It’s only a guess, because I haven’t been able to confirm it through the Jenkins source code. So was looking for any insight on this.

halkeye · June 21, 2023, 4:42pm

if you are using the helm chart, you are using the jcasc section i linked.

like i said, you should probably disable the helm jcasc thing if you want to manage stuff yourself.

sslk · June 21, 2023, 4:47pm

Got it, we are actually using a pretty old helm chart from Jenkins, so it does not have this jcasc section. Right now we are managing things ourselves, like you mentioned, but running into this issue when upgrading the LTS version for our instances running in Kubernetes cluster.

Topic		Replies	Views
Cloud configs missing from config.xml after upgrade Ask a question question , plugin-site , kubernetes	2	1318	June 14, 2023
Nodes/config.xml not cleaned up after failed provisioning Community kubernetes	2	778	March 22, 2023
Jenkins plugins did not upgrade through helm chart Ask a question question , plugin-site , kubernetes	0	610	November 2, 2023
Kubernetes Plugin: The yaml merge strategy does not work for raw yaml Using Jenkins question , pipeline , kubernetes	0	30	November 7, 2024
Downgrading Jenkins LTS version and plugin versions not working Ask a question question , plugin-site , kubernetes	1	1589	July 26, 2023

Does Jenkins wipe out plugin config in config.xml if unable to inspect plugin?

Related topics