Hi,
I’m using Jenkins LTS with version 2.401.3 on AWS EKS version 1.28.
It was running fine and works properly until Jenkins release 2.440.3 (2024-04-17).
I was restarting Jenkins every couple of days, as part of some tests we do, so last time I restarted Jenkins 2.401.3, was on April 16th with no issues, and on April 19th Jenkins couldn’t came up and no informative logs were found except the following from jcasc-reload container.
I changed lots of configurations including startupProbe but nothing fixed the issues until upgrading the from 2.401.3 to 2.440.3.
Can you please assist here and provide guidance or root cause of why the issue happened?
below is the full logs on the jcasc container, no errors were found on init, Jenkins or any other containers.
kubectl logs -f jenkins-0 -n jenkins-qa-stg -c config-reload
[2024-04-25 11:03:35] Starting collector
[2024-04-25 11:03:35] No folder annotation was provided, defaulting to k8s-sidecar-target-directory
[2024-04-25 11:03:35] Selected resource type: ('configmap',)
[2024-04-25 11:03:35] Loading incluster config ...
[2024-04-25 11:03:35] Config for cluster api at 'https://172.20.0.1:443' loaded...
[2024-04-25 11:03:35] Unique filenames will not be enforced.
[2024-04-25 11:03:35] 5xx response content will not be enabled.
[2024-04-25 11:03:40] Working on ADDED configmap jenkins-qa-stg/jenkins-jenkins-config-freestyle-jobs
[2024-04-25 11:04:19] Received unknown exception: HTTPConnectionPool(host='localhost', port=8080): Max retries exceeded with url: /reload-configuration-as-code/?casc-reload-token=jenkins-0 (Caused by ResponseError('too many 503 error responses'))
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
resp = conn.urlopen(
File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 846, in urlopen
return self.urlopen(
File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 846, in urlopen
return self.urlopen(
File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 846, in urlopen
return self.urlopen(
[Previous line repeated 2 more times]
File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 836, in urlopen
retries = retries.increment(method, url, response=response, _pool=self)
File "/usr/local/lib/python3.8/site-packages/urllib3/util/retry.py", line 574, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=8080): Max retries exceeded with url: /reload-configuration-as-code/?casc-reload-token=jenkins-0 (Caused by ResponseError('too many 503 error responses'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/app/resources.py", line 239, in _watch_resource_loop
_watch_resource_iterator(*args)
File "/app/resources.py", line 227, in _watch_resource_iterator
request(request_url, request_method, enable_5xx, request_payload)
File "/app/helpers.py", line 123, in request
res = r.post("%s" % url, auth=auth, json=payload, timeout=REQ_TIMEOUT)
File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 590, in post
return self.request('POST', url, data=data, json=json, **kwargs)
File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 542, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 655, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 507, in send
raise RetryError(e, request=request)
requests.exceptions.RetryError: HTTPConnectionPool(host='localhost', port=8080): Max retries exceeded with url: /reload-configuration-as-code/?casc-reload-token=jenkins-0 (Caused by ResponseError('too many 503 error responses'))
[2024-04-25 11:04:24] Working on ADDED configmap jenkins-qa-stg/jenkins-jenkins-config-freestyle-jobs
[2024-04-25 11:04:24] Contents of freestyle-jobs.yaml haven't changed. Not overwriting existing file
[2024-04-25 11:04:24] Working on ADDED configmap jenkins-qa-stg/jenkins-jenkins-config-jenkins-casc-configs
[2024-04-25 11:04:57] Received unknown exception: HTTPConnectionPool(host='localhost', port=8080): Max retries exceeded with url: /reload-configuration-as-code/?casc-reload-token=jenkins-0 (Caused by ResponseError('too many 503 error responses'))
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
resp = conn.urlopen(
File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 846, in urlopen
return self.urlopen(
File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 846, in urlopen
return self.urlopen(
File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 846, in urlopen
return self.urlopen(
[Previous line repeated 2 more times]
File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 836, in urlopen
retries = retries.increment(method, url, response=response, _pool=self)
File "/usr/local/lib/python3.8/site-packages/urllib3/util/retry.py", line 574, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=8080): Max retries exceeded with url: /reload-configuration-as-code/?casc-reload-token=jenkins-0 (Caused by ResponseError('too many 503 error responses'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/app/resources.py", line 239, in _watch_resource_loop
_watch_resource_iterator(*args)
File "/app/resources.py", line 227, in _watch_resource_iterator
request(request_url, request_method, enable_5xx, request_payload)
File "/app/helpers.py", line 123, in request
res = r.post("%s" % url, auth=auth, json=payload, timeout=REQ_TIMEOUT)
File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 590, in post
return self.request('POST', url, data=data, json=json, **kwargs)
File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 542, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 655, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 507, in send
raise RetryError(e, request=request)
requests.exceptions.RetryError: HTTPConnectionPool(host='localhost', port=8080): Max retries exceeded with url: /reload-configuration-as-code/?casc-reload-token=jenkins-0 (Caused by ResponseError('too many 503 error responses'))
[2024-04-25 11:05:02] Working on ADDED configmap jenkins-qa-stg/jenkins-jenkins-config-freestyle-jobs
[2024-04-25 11:05:02] Contents of freestyle-jobs.yaml haven't changed. Not overwriting existing file
[2024-04-25 11:05:02] Working on ADDED configmap jenkins-qa-stg/jenkins-jenkins-config-jenkins-casc-configs
[2024-04-25 11:05:02] Contents of jenkins-casc-configs.yaml haven't changed. Not overwriting existing file
[2024-04-25 11:05:02] Working on ADDED configmap jenkins-qa-stg/jenkins-jenkins-jcasc-config
[2024-04-25 11:05:36] Received unknown exception: HTTPConnectionPool(host='localhost', port=8080): Max retries exceeded with url: /reload-configuration-as-code/?casc-reload-token=jenkins-0 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f8263cbc1f0>: Failed to establish a new connection: [Errno 111] Connection refused'))
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 174, in _new_conn
conn = connection.create_connection(
File "/usr/local/lib/python3.8/site-packages/urllib3/util/connection.py", line 96, in create_connection
raise err
File "/usr/local/lib/python3.8/site-packages/urllib3/util/connection.py", line 86, in create_connection
sock.connect(sa)
ConnectionRefusedError: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 699, in urlopen
httplib_response = self._make_request(
File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 394, in _make_request
conn.request(method, url, **httplib_request_kw)
File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 239, in request
super(HTTPConnection, self).request(method, url, body=body, headers=headers)
File "/usr/local/lib/python3.8/http/client.py", line 1256, in request
self._send_request(method, url, body, headers, encode_chunked)
File "/usr/local/lib/python3.8/http/client.py", line 1302, in _send_request
self.endheaders(body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.8/http/client.py", line 1251, in endheaders
self._send_output(message_body, encode_chunked=encode_chunked)
File "/usr/local/lib/python3.8/http/client.py", line 1011, in _send_output
self.send(msg)
File "/usr/local/lib/python3.8/http/client.py", line 951, in send
self.connect()
File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 205, in connect
conn = self._new_conn()
File "/usr/local/lib/python3.8/site-packages/urllib3/connection.py", line 186, in _new_conn
raise NewConnectionError(
urllib3.exceptions.NewConnectionError: <urllib3.connection.HTTPConnection object at 0x7f8263cbc1f0>: Failed to establish a new connection: [Errno 111] Connection refused
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 439, in send
resp = conn.urlopen(
File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 846, in urlopen
return self.urlopen(
File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 846, in urlopen
return self.urlopen(
File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 846, in urlopen
return self.urlopen(
[Previous line repeated 2 more times]
File "/usr/local/lib/python3.8/site-packages/urllib3/connectionpool.py", line 755, in urlopen
retries = retries.increment(
File "/usr/local/lib/python3.8/site-packages/urllib3/util/retry.py", line 574, in increment
raise MaxRetryError(_pool, url, error or ResponseError(cause))
urllib3.exceptions.MaxRetryError: HTTPConnectionPool(host='localhost', port=8080): Max retries exceeded with url: /reload-configuration-as-code/?casc-reload-token=jenkins-0 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f8263cbc1f0>: Failed to establish a new connection: [Errno 111] Connection refused'))
During handling of the above exception, another exception occurred:
Traceback (most recent call last):
File "/app/resources.py", line 239, in _watch_resource_loop
_watch_resource_iterator(*args)
File "/app/resources.py", line 227, in _watch_resource_iterator
request(request_url, request_method, enable_5xx, request_payload)
File "/app/helpers.py", line 123, in request
res = r.post("%s" % url, auth=auth, json=payload, timeout=REQ_TIMEOUT)
File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 590, in post
return self.request('POST', url, data=data, json=json, **kwargs)
File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 542, in request
resp = self.send(prep, **send_kwargs)
File "/usr/local/lib/python3.8/site-packages/requests/sessions.py", line 655, in send
r = adapter.send(request, **kwargs)
File "/usr/local/lib/python3.8/site-packages/requests/adapters.py", line 516, in send
raise ConnectionError(e, request=request)
requests.exceptions.ConnectionError: HTTPConnectionPool(host='localhost', port=8080): Max retries exceeded with url: /reload-configuration-as-code/?casc-reload-token=jenkins-0 (Caused by NewConnectionError('<urllib3.connection.HTTPConnection object at 0x7f8263cbc1f0>: Failed to establish a new connection: [Errno 111] Connection refused'))
[2024-04-25 11:05:41] Working on ADDED configmap jenkins-qa-stg/jenkins-jenkins-config-freestyle-jobs
[2024-04-25 11:05:41] Contents of freestyle-jobs.yaml haven't changed. Not overwriting existing file
[2024-04-25 11:05:41] Working on ADDED configmap jenkins-qa-stg/jenkins-jenkins-config-jenkins-casc-configs
[2024-04-25 11:05:41] Contents of jenkins-casc-configs.yaml haven't changed. Not overwriting existing file
[2024-04-25 11:05:41] Working on ADDED configmap jenkins-qa-stg/jenkins-jenkins-jcasc-config
[2024-04-25 11:05:41] Contents of jcasc-default-config.yaml haven't changed. Not overwriting existing file
[2024-04-25 11:06:46] Working on ADDED configmap jenkins-qa-stg/jenkins-jenkins-config-freestyle-jobs
[2024-04-25 11:06:46] Contents of freestyle-jobs.yaml haven't changed. Not overwriting existing file
[2024-04-25 11:06:46] Working on ADDED configmap jenkins-qa-stg/jenkins-jenkins-config-jenkins-casc-configs
[2024-04-25 11:06:46] Contents of jenkins-casc-configs.yaml haven't changed. Not overwriting existing file
[2024-04-25 11:06:46] Working on ADDED configmap jenkins-qa-stg/jenkins-jenkins-jcasc-config
[2024-04-25 11:06:46] Contents of jcasc-default-config.yaml haven't changed. Not overwriting existing file
[2024-04-25 11:07:51] Working on ADDED configmap jenkins-qa-stg/jenkins-jenkins-config-freestyle-jobs
[2024-04-25 11:07:51] Contents of freestyle-jobs.yaml haven't changed. Not overwriting existing file
[2024-04-25 11:07:51] Working on ADDED configmap jenkins-qa-stg/jenkins-jenkins-config-jenkins-casc-configs
[2024-04-25 11:07:51] Contents of jenkins-casc-configs.yaml haven't changed. Not overwriting existing file
[2024-04-25 11:07:51] Working on ADDED configmap jenkins-qa-stg/jenkins-jenkins-jcasc-config
[2024-04-25 11:07:51] Contents of jcasc-default-config.yaml haven't changed. Not overwriting existing file
[2024-04-25 11:08:56] Working on ADDED configmap jenkins-qa-stg/jenkins-jenkins-config-freestyle-jobs
[2024-04-25 11:08:56] Contents of freestyle-jobs.yaml haven't changed. Not overwriting existing file
[2024-04-25 11:08:56] Working on ADDED configmap jenkins-qa-stg/jenkins-jenkins-config-jenkins-casc-configs
[2024-04-25 11:08:56] Contents of jenkins-casc-configs.yaml haven't changed. Not overwriting existing file
[2024-04-25 11:08:56] Working on ADDED configmap jenkins-qa-stg/jenkins-jenkins-jcasc-config
[2024-04-25 11:08:56] Contents of jcasc-default-config.yaml haven't changed. Not overwriting existing file
[2024-04-25 11:10:01] Working on ADDED configmap jenkins-qa-stg/jenkins-jenkins-config-freestyle-jobs
[2024-04-25 11:10:01] Contents of freestyle-jobs.yaml haven't changed. Not overwriting existing file
[2024-04-25 11:10:01] Working on ADDED configmap jenkins-qa-stg/jenkins-jenkins-config-jenkins-casc-configs
[2024-04-25 11:10:01] Contents of jenkins-casc-configs.yaml haven't changed. Not overwriting existing file
[2024-04-25 11:10:01] Working on ADDED configmap jenkins-qa-stg/jenkins-jenkins-jcasc-config
[2024-04-25 11:10:01] Contents of jcasc-default-config.yaml haven't changed. Not overwriting existing file