Unable to install jenkins helm chart 5.1.3

Using Jenkins Ask a question
1

Hi Team,

I am trying to upgrade jenkns helm chart 4.3.22 to 5.1.3 but running into issue only for one AWS US region. I have upgraded jenkins in other regions successfully.
After running into issue I have destroyed helm release and tried to recreate it but getting same error.
I have tried with different version of helm chart but getting different errors for different versions.

I am currently deploying jekkins helm version 5.1.3 on AWS EKS 1.25 version using helm chart and terraform.

Please find my app.yaml file configuration

  installPlugins:
    - kubernetes:4203.v1dd44f5b_1cf9
    - workflow-aggregator:596.v8c21c963d92d
    - git:5.2.1
    - configuration-as-code:1775.v810dc950b_514
  additionalPlugins:
    - matrix-auth:2.6.11
    - allure-jenkins-plugin:2.30.2
    - job-dsl:1.83
    - rebuild:1.33.1
    - pipeline-utility-steps:2.12.1
    - workflow-support:865.v43e78cc44e0d
    - ace-editor:1.1
    - pipeline-model-definition:2.2077.vc78ec45162f1
    - workflow-job:1385.vb_58b_86ea_fff1
    - parameterized-scheduler:1.2
    - file-parameters:205.vf6ce13b_e5dee
    - htmlpublisher:1.30
    - build-user-vars-plugin:1.8
    - kubernetes-client-api:6.10.0-240.v57880ce8b_0b_2
    - kubernetes-credentials:0.11
    - saml:2.333.vc81e525974a_c
    - snakeyaml-api:2.2-111.vc6598e30cc65
    - email-ext:2.99

Error logs:

Running from: /usr/share/jenkins/jenkins.war                                                                                                                                                                                                  
│ 2024-04-09 13:12:01.820+0000 [id=1]    INFO    winstone.Logger#logInternal: Beginning extraction from war file                                                                                                                                
│ 2024-04-09 13:12:03.223+0000 [id=1]    WARNING    o.e.j.s.handler.ContextHandler#setContextPath: Empty contextPath                                                                                                                            
│ 2024-04-09 13:12:03.338+0000 [id=1]    INFO    org.eclipse.jetty.server.Server#doStart: jetty-10.0.20; built: 2024-01-29T20:46:45.278Z; git: 3a745c71c23682146f262b99f4ddc4c1bc41630c; jvm 17.0.10+7                                          │
│ 2024-04-09 13:12:03.726+0000 [id=1]    INFO    o.e.j.w.StandardDescriptorProcessor#visitServlet: NO JSP Support for /, did not find org.eclipse.jetty.jsp.JettyJspServlet                                                                     │
│ 2024-04-09 13:12:03.826+0000 [id=1]    INFO    o.e.j.s.s.DefaultSessionIdManager#doStart: Session workerName=node0                                                                                                                            
│ 2024-04-09 13:12:04.698+0000 [id=1]    INFO    hudson.WebAppMain#contextInitialized: Jenkins home directory: /var/jenkins_home found at: EnvVars.masterEnvVars.get("JENKINS_HOME")                                                            │
│ 2024-04-09 13:12:04.853+0000 [id=1]    INFO    o.e.j.s.handler.ContextHandler#doStart: Started w.@524a2ffb{Jenkins v2.440.2,/,file:///var/jenkins_cache/war/,AVAILABLE}{/var/jenkins_cache/war}                                               │
│ 2024-04-09 13:12:04.885+0000 [id=1]    INFO    o.e.j.server.AbstractConnector#doStart: Started ServerConnector@68fa0ba8{HTTP/1.1, (http/1.1)}{0.0.0.0:8080}                                                                                   │
│ 2024-04-09 13:12:04.911+0000 [id=1]    INFO    org.eclipse.jetty.server.Server#doStart: Started Server@2a7686a7{STARTING}[10.0.20,sto=0] @4045ms                                                                                              
│ 2024-04-09 13:12:04.922+0000 [id=25]    INFO    winstone.Logger#logInternal: Winstone Servlet Engine running: controlPort=disabled                                                                                                            
│ 2024-04-09 13:12:05.296+0000 [id=31]    INFO    jenkins.InitReactorRunner$1#onAttained: Started initialization                                                                                                                                
│ 2024-04-09 13:12:07.454+0000 [id=31]    INFO    jenkins.InitReactorRunner$1#onAttained: Listed all plugins                                                                                                                                    
│ 2024-04-09 13:12:08.873+0000 [id=30]    WARNING    h.p.greenballs.GreenBallsPlugin#start: Unable to change BLUE ColorPalette                                                                                                                  
│ java.lang.reflect.InaccessibleObjectException: Unable to make field int java.awt.Color.value accessible: module java.desktop does not "opens java.awt" to unnamed module @6a6bc417                                                            │
│     at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(Unknown Source)                                                                                                                                                     
│     at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(Unknown Source)                                                                                                                                                     
│     at java.base/java.lang.reflect.Field.checkCanSetAccessible(Unknown Source)                                                                                                                                                                
│     at java.base/java.lang.reflect.Field.setAccessible(Unknown Source)                                                                                                                                                                        
│     at hudson.plugins.greenballs.GreenBallsPlugin.start(GreenBallsPlugin.java:29)                                                                                                                                                             
│     at hudson.ClassicPluginStrategy.startPlugin(ClassicPluginStrategy.java:393)                                                                                                                                                               
│     at hudson.ClassicPluginStrategy.load(ClassicPluginStrategy.java:382)                                                                                                                                                                      
│     at hudson.PluginManager$2$1$1.run(PluginManager.java:553)                                                                                                                                                                                 
│     at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:177)                                                                                                                                                      
│     at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:305)                                                                                                                                                                             
│     at jenkins.model.Jenkins$5.runTask(Jenkins.java:1170)                                                                                                                                                                                     
│     at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:221)                                                                                                                                                                               
│     at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:120)                                                                                                                                                                            
│     at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)                                                                                                                                              
│     at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)                                                                                                                                                            
│     at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)                                                                                                                                                           
│     at java.base/java.lang.Thread.run(Unknown Source)                                                                                                                                                                                         
│ 2024-04-09 13:12:09.050+0000 [id=31]    SEVERE    jenkins.InitReactorRunner$1#onTaskFailed: Failed Loading plugin Pipeline: Declarative Extension Points API v2.2188.v26e255fd2984 (pipeline-model-extensions)      
 - Update required: Pipeline: Job (workflow-job 1326.ve643e00e9220) to be updated to 1385.vb_58b_86ea_fff1 or higher                                                                                                                          
│     at hudson.PluginWrapper.resolvePluginDependencies(PluginWrapper.java:988)                                                                                                                                                                 
│     at hudson.PluginManager$2$1$1.run(PluginManager.java:552)                                                                                                                                                                                 
│     at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:177)                                                                                                                                                      
│     at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:305)                                                                                                                                                                             
│     at jenkins.model.Jenkins$5.runTask(Jenkins.java:1170)                                                                                                                                                                                     
│     at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:221)                                                                                                                                                                               
│     at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:120)                                                                                                                                                                            
│     at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)                                                                                                                                              
│     at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)                                                                                                                                                            
│     at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)                                                                                                                                                           
│     at java.base/java.lang.Thread.run(Unknown Source)                                                                                                                                                                                         
│ 2024-04-09 13:12:09.054+0000 [id=30]    SEVERE    jenkins.InitReactorRunner$1#onTaskFailed: Failed Loading plugin Kubernetes plugin v4203.v1dd44f5b_1cf9 (kubernetes)                                                                         │
│ java.io.IOException: Failed to load: Kubernetes plugin (kubernetes 4203.v1dd44f5b_1cf9)                                                                                                                                                       
│  - Update required: Kubernetes Client API Plugin (kubernetes-client-api 6.4.1-215.v2ed17097a_8e9) to be updated to 6.10.0-240.v57880ce8b_0b_2 or higher                                                                                       
│  - Update required: Kubernetes Credentials Plugin (kubernetes-credentials 0.10.0) to be updated to 0.11 or higher                                                                                                                             
│     at hudson.PluginWrapper.resolvePluginDependencies(PluginWrapper.java:988)                                                                                                                                                                 
│     at hudson.PluginManager$2$1$1.run(PluginManager.java:552)                                                                                                                                                                                 
│     at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:177)                                                                                                                                                      
│     at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:305)                                                                                                                                                                             
│     at jenkins.model.Jenkins$5.runTask(Jenkins.java:1170)                                                                                                                                                                                     
│     at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:221)                                                                                                                                                                               
│     at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:120)                                                                                                                                                                            
│     at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)                                                                                                                                              
│     at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)                                                                                                                                                            
│     at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)                                                                                                                                                           
│     at java.base/java.lang.Thread.run(Unknown Source)                                                                                                                                                                                         
│ 2024-04-09 13:12:09.081+0000 [id=33]    SEVERE    jenkins.InitReactorRunner$1#onTaskFailed: Failed Loading plugin Pipeline: Multibranch v773.vc4fe1378f1d5 (workflow-multibranch)      
java.io.IOException: Failed to load: Pipeline: Multibranch (workflow-multibranch 773.vc4fe1378f1d5)                                                                                                                                           
│  - Update required: Pipeline: Job (workflow-job 1326.ve643e00e9220) to be updated to 1385.vb_58b_86ea_fff1 or higher                                                                                                                          
│     at hudson.PluginWrapper.resolvePluginDependencies(PluginWrapper.java:988)                                                                                                                                                                 
│     at hudson.PluginManager$2$1$1.run(PluginManager.java:552)                                                                                                                                                                                 
│     at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:177)                                                                                                                                                      
│     at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:305)                                                                                                                                                                             
│     at jenkins.model.Jenkins$5.runTask(Jenkins.java:1170)                                                                                                                                                                                     
│     at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:221)                                                                                                                                                                               
│     at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:120)                                                                                                                                                                            
│     at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)                                                                                                                                              
│     at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)                                                                                                                                                            
│     at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)                                                                                                                                                           
│     at java.base/java.lang.Thread.run(Unknown Source)                                                                                                                                                                                         
│ 2024-04-09 13:12:09.085+0000 [id=33]    SEVERE    jenkins.InitReactorRunner$1#onTaskFailed: Failed Loading plugin Pipeline: Declarative v2.2118.v31fd5b_9944b_5 (pipeline-model-definition)                                                   │
│ java.io.IOException: Failed to load: Pipeline: Declarative (pipeline-model-definition 2.2118.v31fd5b_9944b_5)                                                                                                                                 
│  - Failed to load: Pipeline: Multibranch (workflow-multibranch 773.vc4fe1378f1d5)                                                                                                                                                             
│     at hudson.PluginWrapper.resolvePluginDependencies(PluginWrapper.java:988)                                                                                                                                                                 
│     at hudson.PluginManager$2$1$1.run(PluginManager.java:552)                                                                                                                                                                                 
│     at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:177)                                                                                                                                                      
│     at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:305)                                                                                                                                                                             
│     at jenkins.model.Jenkins$5.runTask(Jenkins.java:1170)                                                                                                                                                                                     
│     at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:221)                                                                                                                                                                               
│     at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:120)                                                                                                                                                                            
│     at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)                                                                                                                                              
│     at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)                                                                                                                                                            
│     at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)                                                                                                                                                           
│     at java.base/java.lang.Thread.run(Unknown Source)                                                                                                                                                                                         
│ 2024-04-09 13:12:09.134+0000 [id=33]    SEVERE    jenkins.InitReactorRunner$1#onTaskFailed: Failed Loading plugin Pipeline v596.v8c21c963d92d (workflow-aggregator)                                                                           │
│ java.io.IOException: Failed to load: Pipeline (workflow-aggregator 596.v8c21c963d92d)                                                                                                                      
java.io.IOException: Failed to load: Pipeline (workflow-aggregator 596.v8c21c963d92d)                                                                                                                                                         
│  - Failed to load: Pipeline: Multibranch (workflow-multibranch 773.vc4fe1378f1d5)                                                                                                                                                             
│     at hudson.PluginWrapper.resolvePluginDependencies(PluginWrapper.java:988)                                                                                                                                                                 
│     at hudson.PluginManager$2$1$1.run(PluginManager.java:552)                                                                                                                                                                                 
│     at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:177)                                                                                                                                                      
│     at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:305)                                                                                                                                                                             
│     at jenkins.model.Jenkins$5.runTask(Jenkins.java:1170)                                                                                                                                                                                     
│     at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:221)                                                                                                                                                                               
│     at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:120)                                                                                                                                                                            
│     at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)                                                                                                                                              
│     at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)                                                                                                                                                            
│     at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)     
  at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)                                                                                                                                             
│     at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)                                                                                                                                                  
│     at java.base/java.lang.Thread.run(Unknown Source)                                                                                                                                                                                         
│ 2024-04-09 13:13:10.504+0000 [id=16]    WARNING    o.e.j.s.h.ContextHandler$Context#log: Error while serving http://10.1.191.19:8080/oops                                                                                                     
│ java.lang.NullPointerException: Cannot invoke "javax.servlet.RequestDispatcher.forward(javax.servlet.ServletRequest, javax.servlet.ServletResponse)" because the return value of "org.kohsuke.stapler.StaplerRequest.getView(java.lang.Class, │
│     at io.jenkins.plugins.casc.ConfigurationAsCode.handleExceptionOnReloading(ConfigurationAsCode.java:207)                                                                                                                                   
│     at io.jenkins.plugins.casc.ConfigurationAsCodeBootFailure.doDynamic(ConfigurationAsCodeBootFailure.java:17)                                                                                                                               
│     at java.base/java.lang.invoke.MethodHandle.invokeWithArguments(Unknown Source)                                                                                                                                                            
│     at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:397)
2

You should remove the greenballs plugin (and also ace-editor which is deprecated)
Also it seems the list of plugins you mentioned is not the list of plugins that are actually installed. The log says that workflow-job 1326.ve643e00e9220 is installed but you list it as version 1385.vb_58b_86ea_fff1
You might want to review your yaml configuration if it is really correct.

3

Actually same yaml file worked in other two environments.

Please see my jenkins yaml file once. Where its picking up old plugin versions?
For different helm version its giving different errors. Its very strange…

controller:
  containerSecurityContext:
    runAsUser: 0
    runAsGroup: 0
    readOnlyRootFilesystem: false
    allowPrivilegeEscalation: true
  podSecurityContextOverride:
    runAsUser: 0
    runAsNonRoot: false
    supplementalGroups: [0]
  servicePort: ${master_service_port}
  nodeSelector:
    node-role: ${node_role}
  resources:
    requests:
      cpu: "500m"
      memory: "1536Mi"
    limits:
      cpu: "2000m"
      memory: "4096Mi"
  ingress:
    enabled: ${enable_ingress}
    hostName: ${fqdn}
    annotations:
      "kubernetes.io/ingress.class": "nginx"
  # javaOpts: '-Dhudson.model.DirectoryBrowserSupport.CSP="default-src ''self''; style-src ''self'' ''unsafe-inline''; font-src ''self'' data:;"'
  javaOpts: '-Dhudson.model.DirectoryBrowserSupport.CSP=""'
  installPlugins:
    - kubernetes:4203.v1dd44f5b_1cf9  
    - workflow-aggregator:596.v8c21c963d92d
    - git:5.2.1
    - configuration-as-code:1775.v810dc950b_514
  additionalPlugins:
    - matrix-auth:2.6.11
    - allure-jenkins-plugin:2.30.2
    - job-dsl:1.83
    - rebuild:1.33.1
    - pipeline-utility-steps:2.12.1
    - workflow-support:865.v43e78cc44e0d
    - ace-editor:1.1
    - pipeline-model-definition:2.2077.vc78ec45162f1
    - parameterized-scheduler:1.2
    - file-parameters:205.vf6ce13b_e5dee
    - htmlpublisher:1.30
    - build-user-vars-plugin:1.8
    - kubernetes-client-api:6.10.0-240.v57880ce8b_0b_2
    - kubernetes-credentials:0.11
    - saml:2.333.vc81e525974a_c
    - snakeyaml-api:2.2-111.vc6598e30cc65
    - email-ext:2.99
    - workflow-job:1400.v7fd111b_ec82f
  overwriteJobs: true
  jenkinsUrl: https://${fqdn}/
  JCasC:
    enabled: true
    defaultConfig: true
    configScripts:
      credentials: |
        credentials:
          system:
            domainCredentials:
            - credentials:
              - usernamePassword:
                  description: "GitHub token"
                  id: "xerox-github"
                  password: "${xerox-github-token}"
                  scope: GLOBAL
                  username: "alto-admin"
              - usernamePassword:
                  description: "onetimesecret.com - username and token"
                  id: "one-time-secret"
                  password: "${onetimesecret-token}"
                  scope: GLOBAL
                  username: "admin@alto-platform.ai"
              - usernamePassword:
                  description: "ses username and password"
                  id: "ses-creds"
                  password: "${ses-password}"
                  scope: GLOBAL
                  username: "${ses-username}"
      auth-strategy: |
        jenkins:
          authorizationStrategy:
            projectMatrix:
              permissions:
              - "Overall/Administer:authenticated"
      security: |
        security:
          globalJobDslSecurityConfiguration:
            useScriptSecurity: false
          scriptApproval:
            approvedSignatures:
            - "method hudson.FilePath readToString"
      unclassified: |
        unclassified:
          buildDiscarders:
            configuredBuildDiscarders:
            - "jobBuildDiscarder"
          buildUserVars:
            allBuilds: true
          fingerprints:
            fingerprintCleanupDisabled: false
            storage: "file"
          gitSCM:
            addGitTagAction: false
            allowSecondFetch: false
            createAccountBasedOnEmail: false
            disableGitToolChooser: false
            hideCredentials: false
            showEntireCommitSummaryInChanges: false
            useExistingAccountWithSameEmail: false
          globalLibraries:
            libraries:
            - defaultVersion: "master"
              name: "orbs"
              retriever:
                modernSCM:
                  scm:
                    git:
                      credentialsId: "xerox-github"
                      id: "bf00d5c6-ade7-4c45-9b5c-65d627c6ae25"
                      remote: "https://github.com/carear-io/alto-jenkins.git"
                      traits:
                      - "gitBranchDiscovery"
          mailer:
            charset: "UTF-8"
            useSsl: false
            useTls: false
          pollSCM:
            pollingThreadCount: 10
      tool: |
        tool:
          allure:
            installations:
            - name: "allure"
              properties:
              - installSource:
                  installers:
                  - allureCommandlineInstaller:
                      id: "2.13.9"
          git:
            installations:
            - home: "git"
              name: "Default"
      global-envs: |
        jenkins:
          globalNodeProperties:
          - envVars:
              env:
              - key: "AWS_REGION"
                value: "${region}"
              - key: "ENVIRONMENT"
                value: "${environment}"
              - key: "account_id"
                value: "${aws_account_id}"

      saml: |
        jenkins:
          securityRealm:
            saml:
              advancedConfiguration:
                forceAuthn: false
                spEntityId: "https://${fqdn}/"
              binding: "urn:oasis:names:tc:SAML:2.0:bindings:HTTP-POST"
              displayNameAttributeName: "http://schemas.xmlsoap.org/ws/2005/05/identity/claims/name"
              groupsAttributeName: "http://schemas.xmlsoap.org/claims/Group"
              idpMetadataConfiguration:
                period: 0
                url: "${metadata_url}"
              maximumAuthenticationLifetime: 86400
              usernameCaseConversion: "none"
      jobs-config: |
        jobs:
          - script: |
              job("Job_builder") {
                scm {
                  git{
                    branch("*/master")
                    remote {
                      credentials("xerox-github")
                      url("https://github.com/carear-io/alto-jenkins.git")
                    }
                  }
                }
                steps {
                  dsl {
                    external("./jobs/${cluster_environment}/dsl/*groovy")
                  }
                }
              }
              %{if region == "us-east-1" }
              pipelineJob("AWS_iam_key_rotation") {
                definition {
                  cpsScm {
                    scm {
                      git {
                        branch("*/master")
                        remote {
                          credentials("xerox-github")
                          url("https://github.com/carear-io/alto-jenkins.git")
                        }
                      }
                    }
                    scriptPath("./jobs/pipelines/aws-iam-key-rotation.jenkinsfile")
                  }
                }
              }
              %{ endif }
              %{if cluster_environment == "qa" }
              pipelineJob("Environment_start_stop") {
                displayName("Environments Start/Stop")
                properties {
                  pipelineTriggers {
                    triggers {
                      parameterizedTimerTrigger {
                        parameterizedSpecification('''
              H 17 * * * %TYPE=stop;
              ''')
                      }
                    }
                  }

                }
                definition {
                  cpsScm {
                    scm {
                      git {
                        branch("*/master")
                        remote {
                          credentials("xerox-github")
                          url("https://github.com/carear-io/alto-jenkins.git")
                        }
                      }
                    }
                    scriptPath("./jobs/pipelines/environment-start-stop.jenkinsfile")
                  }
                }
              }
              %{ endif }

agent:
  enabled: ${enable_agent}
  nodeSelector:
    node-role: ${node_role}

persistence:
  enabled: true
  existingClaim: ${pvc_name}

serviceAccountAgent:
  create: true

I have checked by ssh to POD plugins are available

<<K9s-Shell>> Pod: mgmt/jenkins-0 | Container: jenkins 
root@jenkins-0:/# cd $JENKINS_HOME/plugins
root@jenkins-0:/var/jenkins_home/plugins# ls
ace-editor                                                 file-parameters.jpi.pinned                     lockable-resources.jpi.pinned                        prism-api
ace-editor.jpi                                             file-parameters.jpi.version_from_image         lockable-resources.jpi.version_from_image            prism-api.jpi
ace-editor.jpi.pinned                                      font-awesome-api                               mailer                                               prism-api.jpi.pinned
ace-editor.jpi.version_from_image                          font-awesome-api.jpi                           mailer.jpi                                           prism-api.jpi.version_from_image
allure-jenkins-plugin                                      font-awesome-api.jpi.pinned                    mailer.jpi.pinned                                    rebuild
allure-jenkins-plugin.jpi                                  font-awesome-api.jpi.version_from_image        mailer.jpi.version_from_image                        rebuild.jpi
allure-jenkins-plugin.jpi.pinned                           git                                            matrix-auth                                          rebuild.jpi.pinned
allure-jenkins-plugin.jpi.version_from_image               git-client                                     matrix-auth.jpi                                      rebuild.jpi.version_from_image
antisamy-markup-formatter                                  git-client.jpi                                 matrix-auth.jpi.pinned                               saml
antisamy-markup-formatter.jpi                              git-client.jpi.pinned                          matrix-auth.jpi.version_from_image                   saml.jpi
antisamy-markup-formatter.jpi.pinned                       git-client.jpi.version_from_image              matrix-project                                       saml.jpi.pinned
antisamy-markup-formatter.jpi.version_from_image           git-server                                     matrix-project.jpi                                   saml.jpi.version_from_image
apache-httpcomponents-client-4-api                         git-server.jpi                                 matrix-project.jpi.pinned                            scm-api
apache-httpcomponents-client-4-api.jpi                     git-server.jpi.pinned                          matrix-project.jpi.version_from_image                scm-api.jpi
apache-httpcomponents-client-4-api.jpi.pinned              git-server.jpi.version_from_image              metrics                                              scm-api.jpi.pinned
apache-httpcomponents-client-4-api.jpi.version_from_image  git.jpi                                        metrics.jpi                                          scm-api.jpi.version_from_image
asm-api                                                    git.jpi.pinned                                 metrics.jpi.pinned                                   script-security
asm-api.jpi                                                git.jpi.version_from_image                     metrics.jpi.version_from_image                       script-security.jpi
asm-api.jpi.pinned                                         gson-api                                       mina-sshd-api-common                                 script-security.jpi.pinned
asm-api.jpi.version_from_image                             gson-api.jpi                                   mina-sshd-api-common.jpi                             script-security.jpi.version_from_image
authentication-tokens                                      gson-api.jpi.pinned                            mina-sshd-api-common.jpi.pinned                      snakeyaml-api
authentication-tokens.jpi                                  gson-api.jpi.version_from_image                mina-sshd-api-common.jpi.version_from_image          snakeyaml-api.jpi
authentication-tokens.jpi.pinned                           htmlpublisher                                  mina-sshd-api-core                                   snakeyaml-api.jpi.pinned
authentication-tokens.jpi.version_from_image               htmlpublisher.jpi                              mina-sshd-api-core.jpi                               snakeyaml-api.jpi.version_from_image
bootstrap5-api                                             htmlpublisher.jpi.pinned                       mina-sshd-api-core.jpi.pinned                        ssh-credentials
bootstrap5-api.jpi                                         htmlpublisher.jpi.version_from_image           mina-sshd-api-core.jpi.version_from_image            ssh-credentials.jpi
bootstrap5-api.jpi.pinned                                  instance-identity                              okhttp-api                                           ssh-credentials.jpi.pinned
bootstrap5-api.jpi.version_from_image                      instance-identity.jpi                          okhttp-api.jpi                                       ssh-credentials.jpi.version_from_image
bouncycastle-api                                           ionicons-api                                   okhttp-api.jpi.pinned                                sshd
bouncycastle-api.jpi                                       ionicons-api.jpi                               okhttp-api.jpi.version_from_image                    sshd.jpi
branch-api                                                 ionicons-api.jpi.pinned                        parameterized-scheduler                              sshd.jpi.pinned
branch-api.jpi                                             ionicons-api.jpi.version_from_image            parameterized-scheduler.jpi                          sshd.jpi.version_from_image
branch-api.jpi.pinned                                      jackson2-api                                   parameterized-scheduler.jpi.pinned                   structs
branch-api.jpi.version_from_image                          jackson2-api.jpi                               parameterized-scheduler.jpi.version_from_image       structs.jpi
build-user-vars-plugin                                     jackson2-api.jpi.pinned                        pipeline-build-step                                  structs.jpi.pinned
build-user-vars-plugin.jpi                                 jackson2-api.jpi.version_from_image            pipeline-build-step.jpi                              structs.jpi.version_from_image
build-user-vars-plugin.jpi.pinned                          jakarta-activation-api                         pipeline-build-step.jpi.pinned                       token-macro
build-user-vars-plugin.jpi.version_from_image              jakarta-activation-api.jpi                     pipeline-build-step.jpi.version_from_image           token-macro.jpi
caffeine-api                                               jakarta-activation-api.jpi.pinned              pipeline-graph-analysis                              token-macro.jpi.pinned
caffeine-api.jpi                                           jakarta-activation-api.jpi.version_from_image  pipeline-graph-analysis.jpi                          token-macro.jpi.version_from_image
caffeine-api.jpi.pinned                                    jakarta-mail-api                               pipeline-graph-analysis.jpi.pinned                   trilead-api
caffeine-api.jpi.version_from_image                        jakarta-mail-api.jpi                           pipeline-graph-analysis.jpi.version_from_image       trilead-api.jpi
checks-api                                                 jakarta-mail-api.jpi.pinned                    pipeline-groovy-lib                                  trilead-api.jpi.pinned
checks-api.jpi                                             jakarta-mail-api.jpi.version_from_image        pipeline-groovy-lib.jpi                              trilead-api.jpi.version_from_image
checks-api.jpi.pinned                                      javax-activation-api                           pipeline-groovy-lib.jpi.pinned                       variant
checks-api.jpi.version_from_image                          javax-activation-api.jpi                       pipeline-groovy-lib.jpi.version_from_image           variant.jpi
cloudbees-folder                                           javax-activation-api.jpi.pinned                pipeline-input-step                                  variant.jpi.pinned
cloudbees-folder.jpi                                       javax-activation-api.jpi.version_from_image    pipeline-input-step.jpi                              variant.jpi.version_from_image
cloudbees-folder.jpi.pinned                                javax-mail-api                                 pipeline-input-step.jpi.pinned                       workflow-aggregator
cloudbees-folder.jpi.version_from_image                    javax-mail-api.jpi                             pipeline-input-step.jpi.version_from_image           workflow-aggregator.jpi
command-launcher                                           jaxb                                           pipeline-milestone-step                              workflow-aggregator.jpi.pinned
command-launcher.jpi                                       jaxb.jpi                                       pipeline-milestone-step.jpi                          workflow-aggregator.jpi.version_from_image
commons-lang3-api                                          jaxb.jpi.pinned                                pipeline-milestone-step.jpi.pinned                   workflow-api
commons-lang3-api.jpi                                      jaxb.jpi.version_from_image                    pipeline-milestone-step.jpi.version_from_image       workflow-api.jpi
commons-lang3-api.jpi.pinned                               jdk-tool                                       pipeline-model-api                                   workflow-api.jpi.pinned
commons-lang3-api.jpi.version_from_image                   jdk-tool.jpi                                   pipeline-model-api.jpi                               workflow-api.jpi.version_from_image
commons-text-api                                           job-dsl                                        pipeline-model-api.jpi.pinned                        workflow-basic-steps
commons-text-api.jpi                                       job-dsl.jpi                                    pipeline-model-api.jpi.version_from_image            workflow-basic-steps.jpi
commons-text-api.jpi.pinned                                job-dsl.jpi.pinned                             pipeline-model-definition                            workflow-basic-steps.jpi.pinned
commons-text-api.jpi.version_from_image                    job-dsl.jpi.version_from_image                 pipeline-model-definition.jpi                        workflow-basic-steps.jpi.version_from_image
configuration-as-code                                      joda-time-api                                  pipeline-model-definition.jpi.pinned                 workflow-cps
configuration-as-code.jpi                                  joda-time-api.jpi                              pipeline-model-definition.jpi.version_from_image     workflow-cps-global-lib
configuration-as-code.jpi.pinned                           joda-time-api.jpi.pinned                       pipeline-model-extensions                            workflow-cps-global-lib.jpi
configuration-as-code.jpi.version_from_image               joda-time-api.jpi.version_from_image           pipeline-model-extensions.jpi                        workflow-cps-global-lib.jpi.pinned
credentials                                                jquery3-api                                    pipeline-model-extensions.jpi.pinned                 workflow-cps-global-lib.jpi.version_from_image
credentials-binding                                        jquery3-api.jpi                                pipeline-model-extensions.jpi.version_from_image     workflow-cps.jpi
credentials-binding.jpi                                    jquery3-api.jpi.pinned                         pipeline-rest-api                                    workflow-cps.jpi.pinned
credentials-binding.jpi.pinned                             jquery3-api.jpi.version_from_image             pipeline-rest-api.jpi                                workflow-cps.jpi.version_from_image
credentials-binding.jpi.version_from_image                 json-api                                       pipeline-rest-api.jpi.pinned                         workflow-durable-task-step
credentials.jpi                                            json-api.jpi                                   pipeline-rest-api.jpi.version_from_image             workflow-durable-task-step.jpi
credentials.jpi.pinned                                     json-api.jpi.pinned                            pipeline-stage-step                                  workflow-durable-task-step.jpi.pinned
credentials.jpi.version_from_image                         json-api.jpi.version_from_image                pipeline-stage-step.jpi                              workflow-durable-task-step.jpi.version_from_image
data-tables-api                                            json-path-api                                  pipeline-stage-step.jpi.pinned                       workflow-job
data-tables-api.jpi                                        json-path-api.jpi                              pipeline-stage-step.jpi.version_from_image           workflow-job.jpi
data-tables-api.jpi.pinned                                 json-path-api.jpi.pinned                       pipeline-stage-tags-metadata                         workflow-job.jpi.pinned
data-tables-api.jpi.version_from_image                     json-path-api.jpi.version_from_image           pipeline-stage-tags-metadata.jpi                     workflow-job.jpi.version_from_image
display-url-api                                            junit                                          pipeline-stage-tags-metadata.jpi.pinned              workflow-multibranch
display-url-api.jpi                                        junit.jpi                                      pipeline-stage-tags-metadata.jpi.version_from_image  workflow-multibranch.jpi
display-url-api.jpi.pinned                                 junit.jpi.pinned                               pipeline-stage-view                                  workflow-multibranch.jpi.pinned
display-url-api.jpi.version_from_image                     junit.jpi.version_from_image                   pipeline-stage-view.jpi                              workflow-multibranch.jpi.version_from_image
durable-task                                               kubernetes                                     pipeline-stage-view.jpi.pinned                       workflow-scm-step
durable-task.jpi                                           kubernetes-client-api                          pipeline-stage-view.jpi.version_from_image           workflow-scm-step.jpi
durable-task.jpi.pinned                                    kubernetes-client-api.jpi                      pipeline-utility-steps                               workflow-scm-step.jpi.pinned
durable-task.jpi.version_from_image                        kubernetes-client-api.jpi.pinned               pipeline-utility-steps.jpi                           workflow-scm-step.jpi.version_from_image
echarts-api                                                kubernetes-client-api.jpi.version_from_image   pipeline-utility-steps.jpi.pinned                    workflow-step-api
echarts-api.jpi                                            kubernetes-credentials                         pipeline-utility-steps.jpi.version_from_image        workflow-step-api.jpi
echarts-api.jpi.pinned                                     kubernetes-credentials.jpi                     plain-credentials                                    workflow-step-api.jpi.pinned
echarts-api.jpi.version_from_image                         kubernetes-credentials.jpi.pinned              plain-credentials.jpi                                workflow-step-api.jpi.version_from_image
email-ext                                                  kubernetes-credentials.jpi.version_from_image  plain-credentials.jpi.pinned                         workflow-support
email-ext.jpi                                              kubernetes.jpi                                 plain-credentials.jpi.version_from_image             workflow-support.jpi
email-ext.jpi.pinned                                       kubernetes.jpi.pinned                          plugin-util-api                                      workflow-support.jpi.pinned
email-ext.jpi.version_from_image                           kubernetes.jpi.version_from_image              plugin-util-api.jpi                                  workflow-support.jpi.version_from_image
file-parameters                                            lockable-resources                             plugin-util-api.jpi.pinned
file-parameters.jpi                                        lockable-resources.jpi                         plugin-util-api.jpi.version_from_image
root@jenkins-0:/var/jenkins_home/plugins# cd kubernetes
root@jenkins-0:/var/jenkins_home/plugins/kubernetes# cd ../kubernetes-client-api
root@jenkins-0:/var/jenkins_home/plugins/kubernetes-client-api# cd META-INF/
root@jenkins-0:/var/jenkins_home/plugins/kubernetes-client-api/META-INF# cat MANIFEST.MF 
Manifest-Version: 1.0
Created-By: Maven Archiver 3.6.0
Build-Jdk-Spec: 11
Specification-Title: Kubernetes Client API Plugin
Specification-Version: 6.10
Implementation-Title: Kubernetes Client API Plugin
Implementation-Version: 6.10.0-240.v57880ce8b_0b_2
Group-Id: org.jenkins-ci.plugins
Artifact-Id: kubernetes-client-api
Short-Name: kubernetes-client-api
Long-Name: Kubernetes Client API Plugin
Url: https://github.com/jenkinsci/kubernetes-client-api-plugin
Compatible-Since-Version: 6.8.1
Plugin-Version: 6.10.0-240.v57880ce8b_0b_2
Hudson-Version: 2.401.3
Jenkins-Version: 2.401.3
Plugin-Dependencies: okhttp-api:4.11.0-157.v6852a_a_fa_ec11,bouncycastle
 -api:2.30.1.77-225.v26ea_c9455fd9,jackson2-api:2.15.3-372.v309620682326
Plugin-Developers: 
Support-Dynamic-Loading: true
Plugin-License-Name: The Apache Software License, Version 2.0
Plugin-License-Url: http://www.apache.org/licenses/LICENSE-2.0.txt
Plugin-ScmConnection: scm:git:https://github.com/jenkinsci/kubernetes-cl
 ient-api-plugin
Plugin-ScmTag: 57880ce8b0b2e0faef776f26570572e6ed9240ba
Plugin-ScmUrl: https://github.com/jenkinsci/kubernetes-client-api-plugin
Plugin-GitHash: 57880ce8b0b2e0faef776f26570572e6ed9240ba
Implementation-Build: 57880ce8b0b2e0faef776f26570572e6ed9240ba
4

The log tells you that you have old plugins. Are you trying to upgrade an existing installation? Maybe you can clean the old plugins directory first

5

Yes I am trying to upgrade jenkins. I have tried to destroy jenkins helm release as well and tried to install newly but still having the issue. Same yaml file is working in different environment as well. I have tried to remove PLUGIN directory and install again but no luck.
Let me try one more time…

6

Hi @mawinter69

Its got resolved:

I have logged into jenkins pod and deleted plugins.txt file and its automatically came up and its wokring fine…I was installing different different helm chart versions from last two days but just removing plugins.txt solved my issue.

Thanks,