Jenkins Helm chart Upgrade Issue From Helm Version 4.3.22 to 5.1.5

Hi Team,

I was using jenkins helm chart version 4.3.22. I want to upgrade version to latest 5.1.5.

I have deployed jenkins helm chart on AWS EKS version 1.25 using terraform 1.1.2 version.

I am using below list of plugin. I have updated plugins version according to new helm chart version.

installPlugins:
    - kubernetes:4203.v1dd44f5b_1cf9 
    - workflow-aggregator:596.v8c21c963d92d
    - git:5.2.1
    - configuration-as-code:1775.v810dc950b_514
  additionalPlugins:
    - matrix-auth:2.6.11
    - allure-jenkins-plugin:2.30.2
    - job-dsl:1.83
    - rebuild:1.33.1
    - pipeline-utility-steps:2.12.1
    - workflow-support:865.v43e78cc44e0d
    - ace-editor:1.1
    - pipeline-model-definition:2.2077.vc78ec45162f1
    - workflow-job:1400.v7fd111b_ec82f
    - parameterized-scheduler:1.2
    - file-parameters:205.vf6ce13b_e5dee
    - htmlpublisher:1.30
    - build-user-vars-plugin:1.8
    - kubernetes-client-api:6.10.0-240.v57880ce8b_0b_2
    - kubernetes-credentials:0.11
    - saml:2.333.vc81e525974a_c
    - snakeyaml-api:2.2-111.vc6598e30cc65
    - email-ext:2.99

After terraform apply I getting error into jenkins container with below logs that I captured.

Note: Config init container is up and running but getting error in jenkins container.

Note: I have deployed same helm chart version in Europe region on similar environments. But getting issue in US region environments.

ERROR Logs : (I have updated the plugin version in values.yaml and applied)

2024-04-08 11:59:41.035+0000 [id=1]    INFO    org.eclipse.jetty.server.Server#doStart: Started Server@758a34ce{STARTING}[10.0.20,sto=0] @4128ms                                                                                              
│ 2024-04-08 11:59:41.048+0000 [id=26]    INFO    winstone.Logger#logInternal: Winstone Servlet Engine running: controlPort=disabled                                                                                                            
│ 2024-04-08 11:59:41.828+0000 [id=32]    INFO    jenkins.InitReactorRunner$1#onAttained: Started initialization                                                                                                                                
│ 2024-04-08 11:59:43.663+0000 [id=33]    INFO    jenkins.InitReactorRunner$1#onAttained: Listed all plugins                                                                                                                                    
│ 2024-04-08 11:59:44.785+0000 [id=31]    WARNING    h.p.greenballs.GreenBallsPlugin#start: Unable to change BLUE ColorPalette                                                                                                                  
│ java.lang.reflect.InaccessibleObjectException: Unable to make field int java.awt.Color.value accessible: module java.desktop does not "opens java.awt" to unnamed module @391a6f5a                                                            │
│     at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(Unknown Source)                                                                                                                                                     
│     at java.base/java.lang.reflect.AccessibleObject.checkCanSetAccessible(Unknown Source)                                                                                                                                                     
│     at java.base/java.lang.reflect.Field.checkCanSetAccessible(Unknown Source)                                                                                                                                                                
│     at java.base/java.lang.reflect.Field.setAccessible(Unknown Source)                                                                                                                                                                        
│     at hudson.plugins.greenballs.GreenBallsPlugin.start(GreenBallsPlugin.java:29)                                                                                                                                                             
│     at hudson.ClassicPluginStrategy.startPlugin(ClassicPluginStrategy.java:393)                                                                                                                                                               
│     at hudson.ClassicPluginStrategy.load(ClassicPluginStrategy.java:382)                                                                                                                                                                      
│     at hudson.PluginManager$2$1$1.run(PluginManager.java:553)                                                                                                                                                                                 
│     at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:177)                                                                                                                                                      
│     at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:305)                                                                                                                                                                             
│     at jenkins.model.Jenkins$5.runTask(Jenkins.java:1170)                                                                                                                                                                                     
│     at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:221)                                                                                                                                                                               
│     at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:120)                                                                                                                                                                            
│     at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)                                                                                                                                              
│     at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)                                                                                                                                                            
│     at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)                                                                                                                                                           
│     at java.base/java.lang.Thread.run(Unknown Source)                                                                                                                                                                                         
│ 2024-04-08 11:59:45.005+0000 [id=32]    SEVERE    jenkins.InitReactorRunner$1#onTaskFailed: Failed Loading plugin Pipeline: Declarative Extension Points API v2.2184.v0b_358b_953e69 (pipeline-model-extensions)                              │
│ java.io.IOException: Failed to load: Pipeline: Declarative Extension Points API (pipeline-model-extensions 2.2184.v0b_358b_953e69)                                                                                                            
│  - Update required: Pipeline: Job (workflow-job 1326.ve643e00e9220) to be updated to 1385.vb_58b_86ea_fff1 or higher                                                                                                                          
│     at hudson.PluginWrapper.resolvePluginDependencies(PluginWrapper.java:988)                                                                                                                                                                 
│     at hudson.PluginManager$2$1$1.run(PluginManager.java:552)                                                                                                                                                                                 
│     at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:177)                                                                                                                                                      
│     at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:305)                                                                                                                                                                             
│     at jenkins.model.Jenkins$5.runTask(Jenkins.java:1170)                                                                                                                                                                                     
│     at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:221)                                                                                                                                                            
  at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:305)                                                                                                                                                                             
│     at jenkins.model.Jenkins$5.runTask(Jenkins.java:1170)                                                                                                                                                                                     
│     at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:221)                                                                                                                                                                               
│     at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:120)                                                                                                                                                                            
│     at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)                                                                                                                                              
│     at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)                                                                                                                                                            
│     at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)                                                                                                                                                           
│     at java.base/java.lang.Thread.run(Unknown Source)                                                                                                                                                                                                                                                                                                                                             
│     at java.base/java.lang.Thread.run(Unknown Source)                                                                                                                                                                                        
│ 2024-04-08 11:59:45.025+0000 [id=33]    SEVERE    jenkins.InitReactorRunner$1#onTaskFailed: Failed Loading plugin Pipeline: Multibranch v773.vc4fe1378f1d5 (workflow-multibranch)                                                             │
│ java.io.IOException: Failed to load: Pipeline: Multibranch (workflow-multibranch 773.vc4fe1378f1d5)                                                                                                                                           
│  - Update required: Pipeline: Job (workflow-job 1326.ve643e00e9220) to be updated to 1385.vb_58b_86ea_fff1 or higher                                                                                                                          
│     at hudson.PluginWrapper.resolvePluginDependencies(PluginWrapper.java:988)                                                                                                                                                                 
│     at hudson.PluginManager$2$1$1.run(PluginManager.java:552)                                                                                                                                                                                 
│     at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:177)                                                                                                                                                      
│     at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:305)                                                                                                                                                                             
│     at jenkins.model.Jenkins$5.runTask(Jenkins.java:1170)                                                                                                                                        
at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:120)                                                                                                                                                                            
│     at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)                                                                                                                                              
│     at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)                                                                                                                                                            
│     at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)                                                                                                                                                           
│     at java.base/java.lang.Thread.run(Unknown Source)                                                                                                                                                                                         
│ 2024-04-08 11:59:45.029+0000 [id=33]    SEVERE    jenkins.InitReactorRunner$1#onTaskFailed: Failed Loading plugin Pipeline: Declarative v2.2118.v31fd5b_9944b_5 (pipeline-model-definition)                                                   │
│ java.io.IOException: Failed to load: Pipeline: Declarative (pipeline-model-definition 2.2118.v31fd5b_9944b_5)                                                                                                                                 
│  - Failed to load: Pipeline: Multibranch (workflow-multibranch 773.vc4fe1378f1d5)                                                                                                                                                             
│     at hudson.PluginWrapper.resolvePluginDependencies(PluginWrapper.java:988)                                                                                                                                                                 
│     at hudson.PluginManager$2$1$1.run(PluginManager.java:552)                                                                                                                                                                                 
│     at org.jvnet.hudson.reactor.TaskGraphBuilder$TaskImpl.run(TaskGraphBuilder.java:177)                                                                                                                                                      
│     at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:305)                                                                                                                                                                             
│     at jenkins.model.Jenkins$5.runTask(Jenkins.java:1170)                                                                                                                                                                                     
│     at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:221)                                                                                                                                                                               
│     at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:120)                                                                                                                                                                            
│     at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)                                                                                                                                              
│     at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)                                                                                                                                                            
│     at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)                                                                                                                                                           
│     at java.base/java.lang.Thread.run(Unknown Source)                                                                                                                                                                                         
│ 2024-04-08 11:59:45.067+0000 [id=32]    SEVERE    jenkins.InitReactorRunner$1#onTaskFailed: Failed Loading plugin Pipeline v596.v8c21c963d92d (workflow-aggregator)                                                                           │
│ java.io.IOException: Failed to load: Pipeline (workflow-aggregator 596.v8c21c963d92d)    

│ 2024-04-08 12:01:48.218+0000 [id=30]    INFO    jenkins.InitReactorRunner$1#onAttained: Prepared all plugins                                                                                                                                  
│ 2024-04-08 12:01:48.303+0000 [id=33]    INFO    jenkins.InitReactorRunner$1#onAttained: Started all plugins                                                                                                                                   
│ 2024-04-08 12:01:48.550+0000 [id=33]    INFO    jenkins.InitReactorRunner$1#onAttained: Augmented all extensions                                                                                                                              
│ 2024-04-08 12:01:52.570+0000 [id=32]    WARNING    o.j.p.m.AuthorizationContainer#add: Processing a permission assignment in the legacy format (without explicit TYPE prefix): hudson.model.Hudson.Administer:authenticated                   │
│ 2024-04-08 12:01:53.149+0000 [id=32]    INFO    jenkins.InitReactorRunner$1#onAttained: System config loaded                                                                                                                                  
│ 2024-04-08 12:01:54.590+0000 [id=31]    WARNING    o.j.p.m.AuthorizationContainer#add: Processing a permission assignment in the legacy format (without explicit TYPE prefix): Overall/Administer:authenticated                               │
│ 2024-04-08 12:01:54.602+0000 [id=31]    SEVERE    jenkins.InitReactorRunner$1#onTaskFailed: Failed ConfigurationAsCode.init                                                                                                                   
│ io.jenkins.plugins.casc.UnknownAttributesException: cloud: No hudson.slaves.Cloud implementation found for kubernetes                                                                                                                         
│     at io.jenkins.plugins.casc.impl.configurators.HeteroDescribableConfigurator.lambda$lookupDescriptor$12(HeteroDescribableConfigurator.java:236)                                                                                            
│     at io.vavr.control.Option.orElse(Option.java:321)                                                                                                                                                                                         
│     at io.jenkins.plugins.casc.impl.configurators.HeteroDescribableConfigurator.lookupDescriptor(HeteroDescribableConfigurator.java:227)                                                                                                      
│     at io.jenkins.plugins.casc.impl.configurators.HeteroDescribableConfigurator.lambda$configure$3(HeteroDescribableConfigurator.java:86)                                                                                                     
│     at io.vavr.Tuple2.apply(Tuple2.java:238)                                                                                                                                                                                                  
│     at io.jenkins.plugins.casc.impl.configurators.HeteroDescribableConfigurator.configure(HeteroDescribableConfigurator.java:86)                                                                                                              
│     at io.jenkins.plugins.casc.impl.configurators.HeteroDescribableConfigurator.check(HeteroDescribableConfigurator.java:94)              
2024-04-08 12:01:54.604+0000 [id=24]    SEVERE    hudson.util.BootFailure#publish: Failed to initialize Jenkins                                                                                                                               
│ io.jenkins.plugins.casc.UnknownAttributesException: cloud: No hudson.slaves.Cloud implementation found for kubernetes                                                                                                                         
│     at io.jenkins.plugins.casc.impl.configurators.HeteroDescribableConfigurator.lambda$lookupDescriptor$12(HeteroDescribableConfigurator.java:236)                                                                                            
│     at io.vavr.control.Option.orElse(Option.java:321)                                                                                                                                                                                         
│     at io.jenkins.plugins.casc.impl.configurators.HeteroDescribableConfigurator.lookupDescriptor(HeteroDescribableConfigurator.java:227) 
   at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:120)                                                                                                                                                                            
│     at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)                                                                                                                                              
│     at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)                                                                                                                                                            
│     at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)                                                                                                                                                           
│     at java.base/java.lang.Thread.run(Unknown Source)                                                                                                                                                                                         
│ 2024-04-08 12:01:54.664+0000 [id=24]    INFO    hudson.lifecycle.Lifecycle#onStatusUpdate: Stopping Jenkins                                                                                                                                   
│ 2024-04-08 12:01:54.721+0000 [id=24]    INFO    jenkins.model.Jenkins$16#onAttained: Started termination                                                                                                                                      
│ 2024-04-08 12:01:54.776+0000 [id=24]    INFO    jenkins.model.Jenkins$16#onAttained: Completed termination                                                                                                                                    
│ 2024-04-08 12:01:54.777+0000 [id=24]    INFO    jenkins.model.Jenkins#_cleanUpDisconnectComputers: Starting node disconnection                                                                                                                
│ 2024-04-08 12:01:54.782+0000 [id=24]    INFO    jenkins.model.Jenkins#_cleanUpShutdownPluginManager: Stopping plugin manager                                                                                                                  
│ 2024-04-08 12:01:55.128+0000 [id=24]    INFO    jenkins.model.Jenkins#_cleanUpPersistQueue: Persisting build queue                                                                                                                            
│ 2024-04-08 12:01:55.161+0000 [id=24]    INFO    jenkins.model.Jenkins#_cleanUpAwaitDisconnects: Waiting for node disconnection completion                                                                                                     
│ 2024-04-08 12:01:55.163+0000 [id=24]    INFO    hudson.lifecycle.Lifecycle#onStatusUpdate: Jenkins stopped                                                                                                                                    
│ 2024-04-08 12:02:00.776+0000 [id=16]    WARNING    o.e.j.s.h.ContextHandler$Context#log: Error while serving http://10.1.179.179:8080/login                                                                                                   
│ java.lang.NullPointerException: Cannot invoke "javax.servlet.RequestDispatcher.forward(javax.servlet.ServletRequest, javax.servlet.ServletResponse)" because the return value of "org.kohsuke.stapler.StaplerRequest.getView(java.lang.Class, │
│     at io.jenkins.plugins.casc.ConfigurationAsCode.handleExceptionOnReloading(ConfigurationAsCode.java:207)                                                                                                                                   
│     at io.jenkins.plugins.casc.ConfigurationAsCodeBootFailure.doDynamic(ConfigurationAsCodeBootFailure.java:17)                                                                                                                               
│     at java.base/java.lang.invoke.MethodHandle.invokeWithArguments(Unknown Source)                                                                                                                                                            
│     at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:397)                                                                                                                                                                                                                                                                                                                                        
│     at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:140)                                                                                                                                                          
│     at org.kohsuke.stapler.MetaClass$10.dispatch(MetaClass.java:501)                                  
  at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)                                                                                                                                                       
│     at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)                                                                                                                                             
│     at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)                                                                                                                                                  
│     at java.base/java.lang.Thread.run(Unknown Source)                                                                                                                                                                                         
│ 2024-04-08 12:02:00.823+0000 [id=16]    WARNING    o.e.j.s.h.ContextHandler$Context#log: Error while serving http://10.1.179.179:8080/oops                                                                                                    
│ java.lang.NullPointerException: Cannot invoke "javax.servlet.RequestDispatcher.forward(javax.servlet.ServletRequest, javax.servlet.ServletResponse)" because the return value of "org.kohsuke.stapler.StaplerRequest.getView(java.lang.Class, │
│     at io.jenkins.plugins.casc.ConfigurationAsCode.handleExceptionOnReloading(ConfigurationAsCode.java:207)                                                                                                                                   
│     at io.jenkins.plugins.casc.ConfigurationAsCodeBootFailure.doDynamic(ConfigurationAsCodeBootFailure.java:17)