Hi,
We have been using the Gerrit Trigger plugin for a long time, but recently added the Events-log plugin to Gerrit to allow bring down Jenkins and having missed jobs be triggered when it comes back up.
To test, I shutdown Jenkins, published a change to Gerrit then brought Jenkins back up but the job is not triggered. I see in the log that it sees the Change
2025-07-15 21:03:28.039+0000 [id=72] INFO c.s.h.p.g.t.p.GerritMissedEventsPlaybackManager#connectionEstablished: (gerrit-bvg-scm.devops.broadcom.net:8081) Triggering: PatchsetCreated: Change-Id for #202259: I5bd1dba2de1a8da701636de27b554cbec3a760cb PatchSet: 1
2025-07-15 21:03:28.039+0000 [id=72] INFO c.s.h.p.g.t.p.GerritMissedEventsPlaybackManager#connectionEstablished: (gerrit-bvg-scm.devops.broadcom.net:8081) Triggering: RefReplicated [project=Chip/BCM1000, ref=refs/changes/59/202259/1, status=succ
eeded, targetNode=mirror-acc]
2025-07-15 21:03:28.039+0000 [id=72] INFO c.s.h.p.g.t.p.GerritMissedEventsPlaybackManager#connectionEstablished: (gerrit-bvg-scm.devops.broadcom.net:8081) Triggering: RefReplicationDone [project=Chip/BCM1000, ref=refs/changes/59/202259/1, nodesC
ount=1]
2025-07-15 21:03:28.039+0000 [id=71] SEVERE c.s.t.g.g.GerritHandler#notifyListeners: When notifying listener: com.sonyericsson.hudson.plugins.gerrit.trigger.playback.GerritMissedEventsPlaybackManager@354b9925 about event: RefReplicationDone [proje
ct=Chip/mimas, ref=refs/changes/56/202256/1, nodesCount=1]
2025-07-15 21:03:28.039+0000 [id=72] INFO c.s.h.p.g.t.p.GerritMissedEventsPlaybackManager#connectionEstablished: (gerrit-bvg-scm.devops.broadcom.net:8081) Triggering: com.sonymobile.tools.gerrit.gerritevents.dto.events.RefUpdated@3ce555e5
2025-07-15 21:03:28.039+0000 [id=71] SEVERE c.s.t.g.g.GerritHandler#notifyListeners: Notify-error:
java.util.ConcurrentModificationException
at java.base/java.util.ArrayList$Itr.checkForComodification(ArrayList.java:1013)
at java.base/java.util.ArrayList$Itr.next(ArrayList.java:967)
at PluginClassLoader for gerrit-trigger//com.sonyericsson.hudson.plugins.gerrit.trigger.playback.GerritMissedEventsPlaybackManager.gerritEvent(GerritMissedEventsPlaybackManager.java:352)
at PluginClassLoader for gerrit-trigger//com.sonymobile.tools.gerrit.gerritevents.GerritHandler.notifyListener(GerritHandler.java:592)
at PluginClassLoader for gerrit-trigger//com.sonymobile.tools.gerrit.gerritevents.GerritHandler.notifyListeners(GerritHandler.java:559)
at PluginClassLoader for gerrit-trigger//com.sonyericsson.hudson.plugins.gerrit.trigger.JenkinsAwareGerritHandler.notifyListeners(JenkinsAwareGerritHandler.java:80)
at PluginClassLoader for gerrit-trigger//com.sonymobile.tools.gerrit.gerritevents.workers.AbstractGerritEventWork.perform(AbstractGerritEventWork.java:46)
at PluginClassLoader for gerrit-trigger//com.sonymobile.tools.gerrit.gerritevents.workers.GerritEventWork.perform(GerritEventWork.java:55)
at PluginClassLoader for gerrit-trigger//com.sonymobile.tools.gerrit.gerritevents.GerritHandler$EventWorker.run(GerritHandler.java:324)
at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:539)
at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1136)
at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:635)
at java.base/java.lang.Thread.run(Thread.java:842)
2025-07-15 21:03:28.039+0000 [id=72] WARNING c.s.t.g.g.GerritHandler#checkQueueSize: The Gerrit incoming events queue contains 40 items! Something might be stuck, or your system can't process the commands fast enough. Try to increase the number of
receiving worker threads. Current thread-pool size: 1
2025-07-15 21:03:28.039+0000 [id=72] WARNING c.s.t.g.g.GerritHandler#checkQueueSize: The Gerrit incoming events queue contains 40 items! Something might be stuck, or your system can't process the commands fast enough. Try to increase the number of
receiving worker threads. Current thread-pool size: 1
2025-07-15 21:03:28.040+0000 [id=72] INFO c.s.h.p.g.t.p.GerritMissedEventsPlaybackManager#connectionEstablished: (gerrit-bvg-scm.devops.broadcom.net:8081) Triggering: com.sonymobile.tools.gerrit.gerritevents.dto.events.CommentAdded@264c52a8
2025-07-15 21:03:28.040+0000 [id=72] INFO c.s.h.p.g.t.p.GerritMissedEventsPlaybackManager#connectionEstablished: Processing completed for server: gerrit-bvg-scm.devops.broadcom.net:8081
How can we know the number of threads needed? Our production system could have thousands of things to process if Jenkins has been down for a while to upgrade the system? I increased it to 8 and no errors:
2025-07-15 21:24:38.187+0000 [id=72] INFO c.s.h.p.g.t.p.GerritMissedEventsPlaybackManager#connectionEstablished: (37) missed events to process for server: gerrit-bvg-scm.devops.broadcom.net:8081 ...
2025-07-15 21:24:38.187+0000 [id=72] INFO c.s.h.p.g.t.p.GerritMissedEventsPlaybackManager#connectionEstablished: (gerrit-bvg-scm.devops.broadcom.net:8081) Triggering: RefReplicated [project=Core/DDRPHY/Q2, ref=refs/changes/65/202265/meta, status=succeeded, targetNode=mirror-acc]
2025-07-15 21:24:38.189+0000 [id=72] INFO c.s.h.p.g.t.p.GerritMissedEventsPlaybackManager#connectionEstablished: (gerrit-bvg-scm.devops.broadcom.net:8081) Triggering: RefReplicationDone [project=Core/DDRPHY/Q2, ref=refs/changes/65/202265/meta, nodesCount=1]
2025-07-15 21:24:38.190+0000 [id=72] INFO c.s.h.p.g.t.p.GerritMissedEventsPlaybackManager#connectionEstablished: (gerrit-bvg-scm.devops.broadcom.net:8081) Triggering: com.sonymobile.tools.gerrit.gerritevents.dto.events.RefUpdated@ad736825
2025-07-15 21:24:38.190+0000 [id=72] INFO c.s.h.p.g.t.p.GerritMissedEventsPlaybackManager#connectionEstablished: (gerrit-bvg-scm.devops.broadcom.net:8081) Triggering: com.sonymobile.tools.gerrit.gerritevents.dto.events.ChangeAbandoned@5a0ac0a3
2025-07-15 21:24:38.190+0000 [id=72] INFO c.s.h.p.g.t.p.GerritMissedEventsPlaybackManager#connectionEstablished: (gerrit-bvg-scm.devops.broadcom.net:8081) Triggering: com.sonymobile.tools.gerrit.gerritevents.dto.events.RefUpdated@e6927370
2025-07-15 21:24:38.190+0000 [id=72] INFO c.s.h.p.g.t.p.GerritMissedEventsPlaybackManager#connectionEstablished: (gerrit-bvg-scm.devops.broadcom.net:8081) Triggering: com.sonymobile.tools.gerrit.gerritevents.dto.events.CommentAdded@993b34d3
2025-07-15 21:24:38.190+0000 [id=72] INFO c.s.h.p.g.t.p.GerritMissedEventsPlaybackManager#connectionEstablished: (gerrit-bvg-scm.devops.broadcom.net:8081) Triggering: com.sonymobile.tools.gerrit.gerritevents.dto.events.RefUpdated@f1e050f9
2025-07-15 21:24:38.190+0000 [id=72] INFO c.s.h.p.g.t.p.GerritMissedEventsPlaybackManager#connectionEstablished: (gerrit-bvg-scm.devops.broadcom.net:8081) Triggering: com.sonymobile.tools.gerrit.gerritevents.dto.events.CommentAdded@8dd1aa2b
2025-07-15 21:24:38.190+0000 [id=72] INFO c.s.h.p.g.t.p.GerritMissedEventsPlaybackManager#connectionEstablished: (gerrit-bvg-scm.devops.broadcom.net:8081) Triggering: com.sonymobile.tools.gerrit.gerritevents.dto.events.RefUpdated@9172ff63
2025-07-15 21:24:38.191+0000 [id=72] INFO c.s.h.p.g.t.p.GerritMissedEventsPlaybackManager#connectionEstablished: (gerrit-bvg-scm.devops.broadcom.net:8081) Triggering: com.sonymobile.tools.gerrit.gerritevents.dto.events.RefUpdated@a2adcd92
2025-07-15 21:24:38.191+0000 [id=72] INFO c.s.h.p.g.t.p.GerritMissedEventsPlaybackManager#connectionEstablished: (gerrit-bvg-scm.devops.broadcom.net:8081) Triggering: com.sonymobile.tools.gerrit.gerritevents.dto.events.RefUpdated@e73d6f7a
2025-07-15 21:24:38.191+0000 [id=72] INFO c.s.h.p.g.t.p.GerritMissedEventsPlaybackManager#connectionEstablished: (gerrit-bvg-scm.devops.broadcom.net:8081) Triggering: com.sonymobile.tools.gerrit.gerritevents.dto.events.ChangeMerged@5cdb5be2
2025-07-15 21:24:38.191+0000 [id=72] INFO c.s.h.p.g.t.p.GerritMissedEventsPlaybackManager#connectionEstablished: (gerrit-bvg-scm.devops.broadcom.net:8081) Triggering: RefReplicated [project=Chip/BCM1000, ref=refs/changes/40/202240/meta, status=succeeded, targetNode=mirror-acc]
2025-07-15 21:24:38.191+0000 [id=72] INFO c.s.h.p.g.t.p.GerritMissedEventsPlaybackManager#connectionEstablished: (gerrit-bvg-scm.devops.broadcom.net:8081) Triggering: RefReplicationDone [project=Chip/BCM1000, ref=refs/changes/40/202240/meta, nodesCount=1]
2025-07-15 21:24:38.191+0000 [id=72] INFO c.s.h.p.g.t.p.GerritMissedEventsPlaybackManager#connectionEstablished: (gerrit-bvg-scm.devops.broadcom.net:8081) Triggering: RefReplicated [project=Core/DDRPHY/Q2, ref=refs/changes/65/202265/2, status=succeeded, targetNode=mirror-acc]
2025-07-15 21:24:38.191+0000 [id=72] INFO c.s.h.p.g.t.p.GerritMissedEventsPlaybackManager#connectionEstablished: (gerrit-bvg-scm.devops.broadcom.net:8081) Triggering: RefReplicationDone [project=Core/DDRPHY/Q2, ref=refs/changes/65/202265/2, nodesCount=1]
2025-07-15 21:24:38.191+0000 [id=72] INFO c.s.h.p.g.t.p.GerritMissedEventsPlaybackManager#connectionEstablished: (gerrit-bvg-scm.devops.broadcom.net:8081) Triggering: RefReplicated [project=Core/DDRPHY/Q2, ref=refs/heads/main, status=succeeded, targetNode=mirror-acc]
2025-07-15 21:24:38.191+0000 [id=72] INFO c.s.h.p.g.t.p.GerritMissedEventsPlaybackManager#connectionEstablished: (gerrit-bvg-scm.devops.broadcom.net:8081) Triggering: RefReplicationDone [project=Core/DDRPHY/Q2, ref=refs/heads/main, nodesCount=1]
2025-07-15 21:24:38.191+0000 [id=72] INFO c.s.h.p.g.t.p.GerritMissedEventsPlaybackManager#connectionEstablished: (gerrit-bvg-scm.devops.broadcom.net:8081) Triggering: com.sonymobile.tools.gerrit.gerritevents.dto.events.RefUpdated@b0f957d2
2025-07-15 21:24:38.191+0000 [id=72] INFO c.s.h.p.g.t.p.GerritMissedEventsPlaybackManager#connectionEstablished: (gerrit-bvg-scm.devops.broadcom.net:8081) Triggering: com.sonymobile.tools.gerrit.gerritevents.dto.events.RefUpdated@3c1e8fb2
2025-07-15 21:24:38.191+0000 [id=72] INFO c.s.h.p.g.t.p.GerritMissedEventsPlaybackManager#connectionEstablished: (gerrit-bvg-scm.devops.broadcom.net:8081) Triggering: PatchsetCreated: Change-Id for #202259: I5bd1dba2de1a8da701636de27b554cbec3a760cb PatchSet: 2
2025-07-15 21:24:38.191+0000 [id=72] INFO c.s.h.p.g.t.p.GerritMissedEventsPlaybackManager#connectionEstablished: (gerrit-bvg-scm.devops.broadcom.net:8081) Triggering: com.sonymobile.tools.gerrit.gerritevents.dto.events.RefUpdated@262844e
2025-07-15 21:24:38.191+0000 [id=72] INFO c.s.h.p.g.t.p.GerritMissedEventsPlaybackManager#connectionEstablished: (gerrit-bvg-scm.devops.broadcom.net:8081) Triggering: com.sonymobile.tools.gerrit.gerritevents.dto.events.CommentAdded@610669a3
2025-07-15 21:24:38.192+0000 [id=72] INFO c.s.h.p.g.t.p.GerritMissedEventsPlaybackManager#connectionEstablished: (gerrit-bvg-scm.devops.broadcom.net:8081) Triggering: RefReplicated [project=Chip/BCM1000, ref=refs/changes/59/202259/meta, status=succeeded, targetNode=mirror-acc]
2025-07-15 21:24:38.192+0000 [id=72] INFO c.s.h.p.g.t.p.GerritMissedEventsPlaybackManager#connectionEstablished: (gerrit-bvg-scm.devops.broadcom.net:8081) Triggering: RefReplicationDone [project=Chip/BCM1000, ref=refs/changes/59/202259/meta, nodesCount=1]
2025-07-15 21:24:38.192+0000 [id=72] INFO c.s.h.p.g.t.p.GerritMissedEventsPlaybackManager#connectionEstablished: (gerrit-bvg-scm.devops.broadcom.net:8081) Triggering: RefReplicated [project=Chip/BCM1000, ref=refs/changes/59/202259/2, status=succeeded, targetNode=mirror-acc]
2025-07-15 21:24:38.192+0000 [id=72] INFO c.s.h.p.g.t.p.GerritMissedEventsPlaybackManager#connectionEstablished: (gerrit-bvg-scm.devops.broadcom.net:8081) Triggering: RefReplicationDone [project=Chip/BCM1000, ref=refs/changes/59/202259/2, nodesCount=1]
2025-07-15 21:24:38.192+0000 [id=72] INFO c.s.h.p.g.t.p.GerritMissedEventsPlaybackManager#connectionEstablished: Processing completed for server: gerrit-bvg-scm.devops.broadcom.net:8081
2025-07-15 21:24:38.192+0000 [id=72] INFO c.s.t.g.g.GerritConnection#run: Ready to receive data from Gerrit: gerrit-bvg-scm.devops.broadcom.net:8081
But while those messages fly by in the log, Jenkins still says “Jenkins is getting ready to work” and the job is never triggered.
To confirm everything is configured correctly, I then amended the commit and Jenkins started the job
2025-07-15 21:32:49.801+0000 [id=446] INFO c.s.h.p.g.t.g.ToGerritRunListener#onTriggered: Project [BCM1000-Z9-publish_regress_full] triggered by Gerrit: [PatchsetCreated: Change-Id for #202259: I5bd1dba2de1a8da701636de27b554cbec3a760cb PatchSet: 3]
2025-07-15 21:32:49.804+0000 [id=446] INFO c.s.h.p.g.t.h.EventListener#schedule: Project BCM1000-Z9-publish_regress_full Build Scheduled: true By event: 202259/3
2025-07-15 21:33:00.251+0000 [id=832] INFO c.s.h.p.g.t.g.ToGerritRunListener#onStarted: Gerrit build [Chip/BCM1000/Z9/BCM1000-Z9-publish_regress_full #432] Started for cause: [GerritCause: PatchsetCreated: Change-Id for #202259: I5bd1dba2de1a8da70
→
Jenkins setup:
Jenkins: 2.479.2
OS: Linux - 4.18.0-553.5.1.el8_10.x86_64
Java: 17.0.10 - Oracle Corporation (Java HotSpot(TM) 64-Bit Server VM)
---
all-changes:1.5
ansicolor:1.0.6
ant:513.vde9e7b_a_0da_0f
antisamy-markup-formatter:173.v680e3a_b_69ff3
anything-goes-formatter:19.v3e2b_1b_3e0ee5
apache-httpcomponents-client-4-api:4.5.14-269.vfa_2321039a_83
asm-api:9.8-135.vb_2239d08ee90
authentication-tokens:1.131.v7199556c3004
bootstrap5-api:5.3.3-1
bouncycastle-api:2.30.1.80-256.vf98926042a_9b_
branch-api:2.1217.v43d8b_b_d8b_2c7
build-blocker-plugin:166.vc82fc20b_a_ed6
build-monitor-plugin:1.14-973.ve458d62e7c4f
build-timeout:1.37
build-timestamp:1.1.0
build-with-parameters:76.v9382db_f78962
caffeine-api:3.2.0-166.v72a_6d74b_870f
changes-since-last-success:0.6
checks-api:2.2.3
cloudbees-folder:6.1012.v79a_86a_1ea_c1f
command-launcher:123.v37cfdc92ef67
commons-compress-api:1.26.1-2
commons-lang3-api:3.17.0-84.vb_b_938040b_078
commons-text-api:1.13.0-153.v91dcd89e2a_22
conditional-buildstep:1.5.0
console-column-plugin:261.vd7e35335049b_
credentials:1415.v831096eb_5534
credentials-binding:687.v619cb_15e923f
dashboard-view:2.528.v3470c02b_d7c9
data-tables-api:2.1.8-1
delivery-pipeline-plugin:1.4.2
description-setter:264.v1957f215dcd5
disable-job-button:1.v9db_352414f90
display-url-api:2.209.v582ed814ff2f
docker-commons:451.vd12c371eeeb_3
docker-workflow:596.v3e6972b_46b_e2
dropdown-viewstabbar-plugin:1.7
durable-task:587.v84b_877235b_45
dynamic-search-view:0.4.0
echarts-api:5.5.1-5
eddsa-api:0.3.0.1-19.vc432d923e5ee
email-ext:1876.v28d8d38315b_d
envinject:2.926.v69c9b_3896a_96
envinject-api:1.235.va_14c74f8f487
external-monitor-job:223.vb_fddcf42c9b_3
file-operations:331.v9d88a_0206e2e
folder-auth:217.vd5b_18537403e
font-awesome-api:6.6.0-2
gerrit-code-review:0.4.9
gerrit-trigger:2.42.0
gerrit-verify-status-reporter:0.0.3
git:5.7.0
git-client:6.1.3
git-server:137.ve0060b_432302
github:1.42.0
github-api:1.321-488.v9b_c0da_9533f8
github-branch-source:1815.v9152b_2ff7a_1b_
google-chat-notification:166.ve6b_de280f2e8
gradle:2.15
groovy:497.v7b_061a_a_de65d
gson-api:2.13.1-139.v4569c2ef303f
htmlpublisher:425
instance-identity:201.vd2a_b_5a_468a_a_6
ionicons-api:88.va_4187cb_eddf1
jackson2-api:2.18.3-402.v74c4eb_f122b_2
jakarta-activation-api:2.1.3-2
jakarta-mail-api:2.1.3-2
javadoc:310.v032f3f16b_0f8
javax-activation-api:1.2.0-8
javax-mail-api:1.6.2-11
jaxb:2.3.9-133.vb_ec76a_73f706
jdk-tool:83.v417146707a_3d
jersey2-api:2.45-154.v4ded3dc34f81
jira:3.13
jjwt-api:0.11.5-120.v0268cf544b_89
jnr-posix-api:3.1.20-138.vdb_9db_a_39182f
job-dsl:1.90
job-import-plugin:122.v35289550f1e6
jobConfigHistory:1305.vf20a_356586b_8
jobrevision:0.6
joda-time-api:2.14.0-127.v7d9da_295a_d51
join:1.21
jquery:1.12.4-3
jquery3-api:3.7.1-2
jsch:0.2.16-95.v3eecb_55fa_b_78
json-api:20250107-125.v28b_a_ffa_eb_f01
json-path-api:2.9.0-148.v22a_7ffe323ce
junit:1322.v1556dc1c59a_f
last-changes:445.v6e78a_a_05a_645
ldap:780.vcb_33c9a_e4332
lockable-resources:1349.v8b_ccb_c5487f7
log-parser:2.4.2
mailer:489.vd4b_25144138f
mapdb-api:1.0.9-44.va_1e1310c9118
matrix-auth:3.2.6
matrix-project:849.v0cd64ed7e531
maven-plugin:3.25
metrics:4.2.21-464.vc9fa_a_0d6265d
mina-sshd-api-common:2.14.0-143.v2b_362fc39576
mina-sshd-api-core:2.14.0-143.v2b_362fc39576
monitoring:2.5.0
next-build-number:66.v4b_4762172d53
okhttp-api:4.11.0-189.v976fa_d3379d6
oss-symbols-api:296.v4981240eeb_1a_
pam-auth:1.12
parameterized-trigger:840.v3c7d4a_a_5e6c7
pipeline-build-step:567.vea_ce550ece97
pipeline-github-lib:65.v203688e7727e
pipeline-graph-analysis:237.v2b_75640ca_888
pipeline-graph-view:471.vd78d487a_a_1a_0
pipeline-groovy-lib:752.vdddedf804e72
pipeline-input-step:517.vf8e782ee645c
pipeline-milestone-step:134.vdf60d179845f
pipeline-model-api:2.2221.vc657003fb_d93
pipeline-model-definition:2.2221.vc657003fb_d93
pipeline-model-extensions:2.2221.vc657003fb_d93
pipeline-rest-api:2.38
pipeline-stage-step:322.vecffa_99f371c
pipeline-stage-tags-metadata:2.2221.vc657003fb_d93
pipeline-stage-view:2.38
pipeline-utility-steps:2.18.0
plain-credentials:183.va_de8f1dd5a_2b_
plugin-util-api:5.1.0
postbuildscript:3.4.1-695.vf6b_0b_8053979
preSCMbuildstep:115.vf9808414429d
prometheus:819.v50953a_c560dd
promoted-builds:978.v4536a_f065dd8
remote-file:1.24
resource-disposer:0.25
rich-text-publisher-plugin:1.5
role-strategy:756.v978cb_392eb_d3
run-condition:243.v3c3f94e46a_8b_
scm-api:704.v3ce5c542825a_
script-security:1373.vb_b_4a_a_c26fa_00
sidebar-link:2.4.2
snakeyaml-api:2.3-123.v13484c65210a_
ssh-credentials:355.v9b_e5b_cde5003
ssh-slaves:3.1031.v72c6b_883b_869
sshd:3.353.v2b_d33c46e970
structs:343.vdcf37b_a_c81d5
subversion:1287.vd2d507146906
text-finder:1.31
text-finder-run-condition:6.vdf94e6f8d2c3
timestamper:1.28
token-macro:444.v52de7e9c573d
tracking-svn:1.1
trilead-api:2.209.v0e69b_c43c245
variant:70.va_d9f17f859e0
workflow-aggregator:600.vb_57cdd26fdd7
workflow-api:1373.v7b_813f10efa_b_
workflow-basic-steps:1079.vce64b_a_929c5a_
workflow-cps:4046.v90b_1b_9edec67
workflow-durable-task-step:1405.v1fcd4a_d00096
workflow-job:1520.v56d65e3b_4566
workflow-multibranch:806.vb_b_688f609ee9
workflow-scm-step:437.v05a_f66b_e5ef8
workflow-step-api:700.v6e45cb_a_5a_a_21
workflow-support:968.v8f17397e87b_8
ws-cleanup:0.48