Severe Slowness in Jenkins UI Builds Post-Upgrade (2.440.1) on AWS EC2 RHEL Instanc

Hello Everyone,

We are currently running our Jenkins server on an AWS EC2 RHEL Linux instance, with the Jenkins home directory mounted from an EFS volume. Recently, we upgraded Jenkins from version 2.361.4-1 to 2.440.1 LTS. Concurrently, we also upgraded our RHEL OS from version 7.9 to 8.9 and updated OpenJDK from version 11 to 17.

After conducting thorough testing of our CI/CD pipelines in a non-prod environment, we confirmed that everything was functioning as expected. However, upon implementing the same upgrades in our prod environment, we have encountered significant performance issues. Specifically, when initiating builds using the “Build Now” option in the Jenkins UI, we experience extreme slowness. This issue also persists with the “Rebuild” and “Replay” options, where none of the pipeline stages progress, and the pipeline runs for an extended period without completion. Interestingly, builds triggered through a Bitbucket webhook do not exhibit this problem; pipeline stages pass without delay when the build is initiated by a direct commit through Bitbucket.

Jenkins encountering an issue when ‘CpsFlowExecution’ class checking if there are any running steps that would block a Jenkins restart.

Below is an warning message from the Jenkins log that repeats when attempting to use the “Build Now/Rebuild/Replay” option from the Jenkins UI:

2024-07-03 15:09:38.562+0000 [id=20157] WARNING o.j.p.w.cps.CpsFlowExecution#blocksRestart: Not blocking restart due to problem checking running steps in CpsFlowExecution[Owner[ORCVS/sm-orc/a360/3:ORCVS/sm-orc/a360 #3]]
java.util.concurrent.TimeoutException: Waited 1 seconds (plus 66319 nanoseconds delay) for SettableFuture@e429c[status=PENDING]
        at com.google.common.util.concurrent.AbstractFuture.get(AbstractFuture.java:533)
        at com.google.common.util.concurrent.AbstractFuture$TrustedFuture.get(AbstractFuture.java:118)
        at org.jenkinsci.plugins.workflow.cps.CpsFlowExecution.blocksRestart(CpsFlowExecution.java:1011)
        at org.jenkinsci.plugins.workflow.job.WorkflowRun$2.blocksRestart(WorkflowRun.java:407)
        at org.jenkinsci.plugins.workflow.job.WorkflowRun$2.displayCell(WorkflowRun.java:410)
        at hudson.model.Executor.isDisplayCell(Executor.java:682)
        at hudson.model.Computer.getDisplayExecutors(Computer.java:1016)
        at jdk.internal.reflect.GeneratedMethodAccessor372.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
        at java.base/java.lang.reflect.Method.invoke(Method.java:568)
        at org.apache.commons.jexl.util.PropertyExecutor.execute(PropertyExecutor.java:125)
        at org.apache.commons.jexl.util.introspection.UberspectImpl$VelGetterImpl.invoke(UberspectImpl.java:314)
        at org.apache.commons.jexl.parser.ASTArrayAccess.evaluateExpr(ASTArrayAccess.java:185)
        at org.apache.commons.jexl.parser.ASTIdentifier.execute(ASTIdentifier.java:75)
        at org.apache.commons.jexl.parser.ASTReference.execute(ASTReference.java:83)
        at org.apache.commons.jexl.parser.ASTReference.value(ASTReference.java:57)
        at org.apache.commons.jexl.parser.ASTReferenceExpression.value(ASTReferenceExpression.java:51)
        at org.apache.commons.jexl.ExpressionImpl.evaluate(ExpressionImpl.java:80)
        at hudson.ExpressionFactory2$JexlExpression.evaluate(ExpressionFactory2.java:76)
        at org.apache.commons.jelly.tags.core.CoreTagLibrary$3.run(CoreTagLibrary.java:134)
        at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
        at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)
        at org.apache.commons.jelly.tags.core.ForEachTag.doTag(ForEachTag.java:150)
        at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:265)
        at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
        at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:101)
        at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
        at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:265)
        at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:101)
        at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)
        at org.apache.commons.jelly.tags.core.OtherwiseTag.doTag(OtherwiseTag.java:41)
        at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:265)
        at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
        at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:101)
        at org.apache.commons.jelly.TagSupport.invokeBody(TagSupport.java:161)
        at org.apache.commons.jelly.tags.core.ChooseTag.doTag(ChooseTag.java:38)
        at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:265)
        at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
        at org.kohsuke.stapler.jelly.ReallyStaticTagLibrary$1.run(ReallyStaticTagLibrary.java:101)
        at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
        at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
        at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:122)
        at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
        at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
        at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:122)
        at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
        at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:99)
        at org.kohsuke.stapler.jelly.IncludeTag.doTag(IncludeTag.java:172)
        at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:265)
        at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:101)
        at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
        at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:265)
        at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
        at org.kohsuke.stapler.jelly.CallTagLibScript$1.run(CallTagLibScript.java:101)
        at org.apache.commons.jelly.tags.define.InvokeBodyTag.doTag(InvokeBodyTag.java:91)
        at org.apache.commons.jelly.impl.TagScript.run(TagScript.java:265)
        at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
        at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
        at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:122)
        at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
        at org.apache.commons.jelly.tags.core.CoreTagLibrary$1.run(CoreTagLibrary.java:98)
        at org.apache.commons.jelly.impl.ScriptBlock.run(ScriptBlock.java:95)
        at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
        at org.kohsuke.stapler.jelly.CallTagLibScript.run(CallTagLibScript.java:122)
        at org.apache.commons.jelly.tags.core.CoreTagLibrary$2.run(CoreTagLibrary.java:105)
        at org.kohsuke.stapler.jelly.JellyViewScript.run(JellyViewScript.java:99)
        at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:66)
        at org.kohsuke.stapler.jelly.DefaultScriptInvoker.invokeScript(DefaultScriptInvoker.java:55)
        at org.kohsuke.stapler.jelly.ScriptInvoker.execute(ScriptInvoker.java:56)
        at org.kohsuke.stapler.jelly.ScriptInvoker.execute(ScriptInvoker.java:43)
        at org.kohsuke.stapler.Facet$1.dispatch(Facet.java:240)
        at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:770)
        at org.kohsuke.stapler.Stapler.invoke(Stapler.java:900)
        at org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:289)
        at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:59)
        at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:770)
        at org.kohsuke.stapler.Stapler.invoke(Stapler.java:900)
        at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:836)
        at org.kohsuke.stapler.Stapler.invoke(Stapler.java:900)
        at org.kohsuke.stapler.Stapler.invoke(Stapler.java:698)
        at org.kohsuke.stapler.Stapler.service(Stapler.java:248)
        at javax.servlet.http.HttpServlet.service(HttpServlet.java:590)
        at org.eclipse.jetty.servlet.ServletHolder.handle(ServletHolder.java:764)
        at org.eclipse.jetty.servlet.ServletHandler$ChainEnd.doFilter(ServletHandler.java:1665)
        at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:163)
        at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:239)
        at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:215)
        at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:88)
        at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:121)
        at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:160)
        at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125)
        at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:160)
        at javax.servlet.FilterChain$doFilter$0.call(Unknown Source)
        at com.ceilfors.jenkins.plugins.jiratrigger.ExceptionLoggingFilter.doFilter(ExceptionLoggingFilter.groovy:29)
        at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:160)
        at jenkins.util.HttpServletFilter$1.doFilter(HttpServletFilter.java:76)
        at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:160)
        at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:166)
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
        at jenkins.ErrorAttributeFilter.doFilter(ErrorAttributeFilter.java:29)
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
        at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:154)
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:94)
        at jenkins.security.AcegiSecurityExceptionFilter.doFilter(AcegiSecurityExceptionFilter.java:52)
        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
        at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:54)
        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
        at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:126)
        at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:120)
        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
        at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:100)
        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
        at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:110)
        at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:101)
        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
        at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:227)
        at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:221)
        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
        at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:97)
        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
        at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:117)
        at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:87)
        at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:63)
        at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
        at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:111)
        at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:172)
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
        at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:53)
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
        at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:86)
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
        at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
        at jenkins.security.SuspiciousRequestFilter.doFilter(SuspiciousRequestFilter.java:38)
        at org.eclipse.jetty.servlet.FilterHolder.doFilter(FilterHolder.java:202)
        at org.eclipse.jetty.servlet.ServletHandler$Chain.doFilter(ServletHandler.java:1635)
        at org.eclipse.jetty.servlet.ServletHandler.doHandle(ServletHandler.java:527)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:131)
        at org.eclipse.jetty.security.SecurityHandler.handle(SecurityHandler.java:569)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:223)
        at org.eclipse.jetty.server.session.SessionHandler.doHandle(SessionHandler.java:1570)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextHandle(ScopedHandler.java:221)
        at org.eclipse.jetty.server.handler.ContextHandler.doHandle(ContextHandler.java:1384)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:176)
        at org.eclipse.jetty.servlet.ServletHandler.doScope(ServletHandler.java:484)
        at org.eclipse.jetty.server.session.SessionHandler.doScope(SessionHandler.java:1543)
        at org.eclipse.jetty.server.handler.ScopedHandler.nextScope(ScopedHandler.java:174)
        at org.eclipse.jetty.server.handler.ContextHandler.doScope(ContextHandler.java:1306)
        at org.eclipse.jetty.server.handler.ScopedHandler.handle(ScopedHandler.java:129)
        at org.eclipse.jetty.server.handler.HandlerWrapper.handle(HandlerWrapper.java:122)
        at org.eclipse.jetty.server.Server.handle(Server.java:563)
        at org.eclipse.jetty.server.HttpChannel$RequestDispatchable.dispatch(HttpChannel.java:1598)
        at org.eclipse.jetty.server.HttpChannel.dispatch(HttpChannel.java:753)
        at org.eclipse.jetty.server.HttpChannel.handle(HttpChannel.java:501)
        at org.eclipse.jetty.server.HttpConnection.onFillable(HttpConnection.java:287)
        at org.eclipse.jetty.io.AbstractConnection$ReadCallback.succeeded(AbstractConnection.java:314)
        at org.eclipse.jetty.io.FillInterest.fillable(FillInterest.java:100)
        at org.eclipse.jetty.io.SelectableChannelEndPoint$1.run(SelectableChannelEndPoint.java:53)
        at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.runTask(AdaptiveExecutionStrategy.java:421)
        at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.consumeTask(AdaptiveExecutionStrategy.java:390)
        at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.tryProduce(AdaptiveExecutionStrategy.java:277)
        at org.eclipse.jetty.util.thread.strategy.AdaptiveExecutionStrategy.run(AdaptiveExecutionStrategy.java:199)
        at org.eclipse.jetty.util.thread.ReservedThreadExecutor$ReservedThread.run(ReservedThreadExecutor.java:411)
        at org.eclipse.jetty.util.thread.QueuedThreadPool.runJob(QueuedThreadPool.java:969)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.doRunJob(QueuedThreadPool.java:1194)
        at org.eclipse.jetty.util.thread.QueuedThreadPool$Runner.run(QueuedThreadPool.java:1149)
        at java.base/java.lang.Thread.run(Thread.java:840)

Our Jenkins setup, OS, Java and Plugin information below -

Jenkins: 2.440.1
OS: Linux - 4.18.0-553.el8_10.x86_64
Java: 17.0.11 - Red Hat, Inc. (OpenJDK 64-Bit Server VM)
---
Office-365-Connector:4.21.1
amazon-ecr:1.136.v914ea_5948634
ansible-tower:0.16.0
ansicolor:1.0.4
ant:497.v94e7d9fffa_b_9
antisamy-markup-formatter:162.v0e6ec0fcfcf6
apache-httpcomponents-client-4-api:4.5.14-208.v438351942757
apache-httpcomponents-client-5-api:5.3.1-1.0
artifactory:4.0.6
asm-api:9.7-33.v4d23ef79fcc8
authentication-tokens:1.113.v81215a_241826
authorize-project:1.7.2
aws-credentials:231.v08a_59f17d742
aws-java-sdk:1.12.730-457.v3403b_37d2170
aws-java-sdk-api-gateway:1.12.730-457.v3403b_37d2170
aws-java-sdk-autoscaling:1.12.730-457.v3403b_37d2170
aws-java-sdk-cloudformation:1.12.730-457.v3403b_37d2170
aws-java-sdk-cloudfront:1.12.730-457.v3403b_37d2170
aws-java-sdk-codebuild:1.12.730-457.v3403b_37d2170
aws-java-sdk-codedeploy:1.12.730-457.v3403b_37d2170
aws-java-sdk-ec2:1.12.730-457.v3403b_37d2170
aws-java-sdk-ecr:1.12.730-457.v3403b_37d2170
aws-java-sdk-ecs:1.12.730-457.v3403b_37d2170
aws-java-sdk-efs:1.12.730-457.v3403b_37d2170
aws-java-sdk-elasticbeanstalk:1.12.730-457.v3403b_37d2170
aws-java-sdk-elasticloadbalancingv2:1.12.730-457.v3403b_37d2170
aws-java-sdk-iam:1.12.730-457.v3403b_37d2170
aws-java-sdk-kinesis:1.12.730-457.v3403b_37d2170
aws-java-sdk-lambda:1.12.730-457.v3403b_37d2170
aws-java-sdk-logs:1.12.730-457.v3403b_37d2170
aws-java-sdk-minimal:1.12.730-457.v3403b_37d2170
aws-java-sdk-organizations:1.12.730-457.v3403b_37d2170
aws-java-sdk-secretsmanager:1.12.730-457.v3403b_37d2170
aws-java-sdk-sns:1.12.730-457.v3403b_37d2170
aws-java-sdk-sqs:1.12.730-457.v3403b_37d2170
aws-java-sdk-ssm:1.12.730-457.v3403b_37d2170
azure-cli:0.9
azure-credentials:312.v0f3973cd1e59
azure-sdk:174.va_89c1df897d2
basic-branch-build-strategies:81.v05e333931c7d
bootstrap5-api:5.3.3-1
bouncycastle-api:2.30.1.78.1-233.vfdcdeb_0a_08a_a_
branch-api:2.1169.va_f810c56e895
build-name-setter:2.4.2
build-timeout:1.32
caffeine-api:3.1.8-133.v17b_1ff2e0599
checks-api:2.2.0
cloud-stats:336.v788e4055508b_
cloudbees-folder:6.858.v898218f3609d
command-launcher:107.v773860566e2e
commons-compress-api:1.26.1-2
commons-lang3-api:3.14.0-76.vda_5591261cfe
commons-text-api:1.12.0-119.v73ef73f2345d
config-file-provider:973.vb_a_80ecb_9a_4d0
copyartifact:746.vd2a_674fb_4f6f
credentials:1344.v5a_3f65a_1e173
credentials-binding:677.vdc9d38cb_254d
custom-tools-plugin:0.8
dashboard-view:2.508.va_74654f026d1
data-tables-api:2.0.8-1
dependency-check-jenkins-plugin:5.5.0
display-url-api:2.204.vf6fddd8a_8b_e9
docker-build-publish:1.4.0
docker-commons:439.va_3cb_0a_6a_fb_29
docker-java-api:3.3.6-90.ve7c5c7535ddd
docker-plugin:1.6.2
docker-slaves:1.0.7
docker-workflow:580.vc0c340686b_54
durable-task:555.v6802fe0f0b_82
echarts-api:5.5.0-1
eddsa-api:0.3.0-4.v84c6f0f4969e
email-ext:1814.v404722f34263
extended-choice-parameter:382.v5697b_32134e8
extended-read-permission:53.v6499940139e5
external-monitor-job:215.v2e88e894db_f8
file-leak-detector:1.13
font-awesome-api:6.5.2-1
generic-webhook-trigger:2.2.1
git:5.2.2
git-client:4.7.0
git-server:126.v0d945d8d2b_39
golang:1.4
google-oauth-plugin:1.330.vf5e86021cb_ec
gradle:2.12
groovy:457.v99900cb_85593
gson-api:2.11.0-41.v019fcf6125dc
htmlpublisher:1.35
instance-identity:185.v303dc7c645f9
ionicons-api:74.v93d5eb_813d5f
ivy:2.6
jackson2-api:2.17.0-379.v02de8ec9f64c
jakarta-activation-api:2.1.3-1
jakarta-mail-api:2.1.3-1
javadoc:243.vb_b_503b_b_45537
javax-activation-api:1.2.0-7
javax-mail-api:1.6.2-10
jaxb:2.3.9-1
jdk-tool:73.vddf737284550
jenkins-plugin:3.7.0
jersey2-api:2.42-147.va_28a_44603b_d5
jfrog:1.5.0
jira:3.13
jira-trigger:1.0.3
jnr-posix-api:3.1.19-2
jobConfigHistory:1229.v3039470161a_d
joda-time-api:2.12.7-29.v5a_b_e3a_82269a_
jquery:1.12.4-1
jquery-detached:1.2.1
jquery3-api:3.7.1-2
jsch:0.2.16-86.v42e010d9484b_
json-api:20240303-41.v94e11e6de726
json-path-api:2.9.0-58.v62e3e85b_a_655
junit:1265.v65b_14fa_f12f0
kubernetes:4250.v93f47a_8e67b_f
kubernetes-client-api:6.10.0-240.v57880ce8b_0b_2
kubernetes-credentials:174.va_36e093562d9
ldap:725.v3cb_b_711b_1a_ef
lockable-resources:1255.vf48745da_35d0
mailer:472.vf7c289a_4b_420
mapdb-api:1.0.9-40.v58107308b_7a_7
matrix-auth:3.2.2
matrix-project:822.824.v14451b_c0fd42
maven-plugin:3.23
mercurial:1260.vdfb_723cdcc81
metrics:4.2.21-451.vd51df8df52ec
mina-sshd-api-common:2.12.1-113.v4d3ea_5eb_7f72
mina-sshd-api-core:2.12.1-113.v4d3ea_5eb_7f72
monitoring:1.99.0
nodejs:1.6.1
notification:1.17
oauth-credentials:0.653.v14cf2088e950
okhttp-api:4.11.0-172.vda_da_1feeb_c6e
openshift-client:1.1.0.424.v829cb_ccf8798
pam-auth:1.11
perfecto:1.21
pipeline-aws:1.45
pipeline-build-step:540.vb_e8849e1a_b_d8
pipeline-graph-analysis:216.vfd8b_ece330ca_
pipeline-groovy-lib:727.ve832a_9244dfa_
pipeline-input-step:495.ve9c153f6067b_
pipeline-milestone-step:119.vdfdc43fc3b_9a_
pipeline-model-api:2.2198.v41dd8ef6dd56
pipeline-model-definition:2.2198.v41dd8ef6dd56
pipeline-model-extensions:2.2198.v41dd8ef6dd56
pipeline-rest-api:2.34
pipeline-stage-step:312.v8cd10304c27a_
pipeline-stage-tags-metadata:2.2198.v41dd8ef6dd56
pipeline-stage-view:2.34
pipeline-utility-steps:2.17.0
plain-credentials:183.va_de8f1dd5a_2b_
plugin-usage-plugin:4.4
plugin-util-api:4.1.0
popper2-api:2.11.6-4
project-stats-plugin:23.v47fee1f77b_84
publish-over:0.22
publish-over-ssh:1.25
pyenv-pipeline:2.1.2
python:1.3
qualys-cs:1.7.0.3
rebuild:330.v645b_7df10e2a_
resource-disposer:0.23
role-strategy:717.v6a_69a_fe98974
run-condition:1.7
s3:466.vf5b_3db_8e3eb_2
scm-api:690.vfc8b_54395023
script-security:1341.va_2819b_414686
shiningpanda:0.24
snakeyaml-api:2.2-111.vc6598e30cc65
sonar:2.17.2
ssh-agent:367.vf9076cd4ee21
ssh-credentials:337.v395d2403ccd4
ssh-slaves:2.973.v0fa_8c0dea_f9f
sshd:3.330.vc866a_8389b_58
structs:338.v848422169819
subversion:1256.vee91953217b_6
throttle-concurrents:2.14
timestamper:1.27
token-macro:400.v35420b_922dcb_
trilead-api:2.147.vb_73cc728a_32e
update-sites-manager:2.0.0
variant:60.v7290fc0eb_b_cd
windows-slaves:1.8.1
workflow-aggregator:596.v8c21c963d92d
workflow-api:1316.v33eb_726c50b_a_
workflow-basic-steps:1058.vcb_fc1e3a_21a_9
workflow-cps:3903.v48a_8836749e9
workflow-durable-task-step:1353.v1891a_b_01da_18
workflow-job:1400.v7fd111b_ec82f
workflow-multibranch:773.vc4fe1378f1d5
workflow-scm-step:427.v4ca_6512e7df1
workflow-step-api:657.v03b_e8115821b_
workflow-support:907.v6713a_ed8a_573
ws-cleanup:0.46

We are seeking insights or solutions to address this severe slowness issue in the Jenkins UI. Any advice or recommendations would be greatly appreciated. Thank you.

Hello Everyone,

I’m reaching out to kindly ask if anyone might have had a chance to look into it or share any insights or suggestions on how to address this problem. Your advice / suggestions would be greatly appreciated as we’re keen to resolve this issue.

Thank you for your time and assistance!

Hello and welcome to this community, @Rabi. :wave:

The error message you’re encountering seems to indicate you have a TimeoutException within Jenkins, specifically during a process that checks running steps in a CpsFlowExecution. This exception suggests that Jenkins waited for a certain operation to complete but it did not finish within the expected timeframe, leading to a timeout. :person_shrugging:

This issue could be caused by several factors, including but not limited to:

  • Jenkins or the underlying system might be under heavy load, causing operations to take longer than expected. This could be due to high CPU usage, memory constraints, or disk I/O limitations.
  • A plugin involved in the process might be behaving inefficiently or encountering an error, leading to prolonged execution times.
  • If the operation involves network communication (e.g., with a remote git repository or a database), network latency or connectivity issues could cause delays.
  • Misconfigurations in Jenkins or related tools could lead to inefficient execution paths or deadlocks.

What are the main differences between the server that was behaving correctly and this one?

The main differences between the non-prod and prod servers are as follows: On the infrastructure side, in non-prod, we are using a 2 CPU and 4 GB RAM EC2 instance, whereas in prod, we are using an 8 CPU and 32 GB RAM EC2 instance. Another difference is that we have around 200+ GB of data in the jobs folder in prod and 40+ GB of data in the jobs folder in non-prod. Additionally, the Jenkins plugins update server firewall is enabled for the non-prod server and disabled for the prod server. However, we use the exact same replica of plugins in both non-prod and prod.