Failed to initialise k8s secret provider

Using Jenkins Ask a question
1

I’m running the jenkins-operator in a Kubernetes cluster that sits behind a proxy.
I have a very vanilla CR to start new Jenkins instances. The only tweak my CR has is the JAVA_OPTS required for Jenkins to use the proxy:

http.proxyHost=my.proxy.com -Dhttp.proxyPort=443 -Dhttps.proxyHost=my.proxy.com -Dhttps.proxyPort=443 -Dhttps.nonProxyHosts=kubernetes.default.svc

My Jenkins instance starts successfully, however, the Kubernetes secret provider failes with this error:

SEVERE  c.c.j.p.k.KubernetesCredentialProvider#startWatchingForSecrets: Failed to initialise k8s secret provider, secrets from Kubernetes will not be available

java.net.SocketException: Socket closed
        at java.base/sun.nio.ch.NioSocketImpl.endRead(Unknown Source)
        at java.base/sun.nio.ch.NioSocketImpl.implRead(Unknown Source)
        at java.base/sun.nio.ch.NioSocketImpl.read(Unknown Source)
        at java.base/sun.nio.ch.NioSocketImpl$1.read(Unknown Source)
        at java.base/java.net.Socket$SocketInputStream.read(Unknown Source)
        at PluginClassLoader for okhttp-api//okio.InputStreamSource.read(JvmOkio.kt:93)
        at PluginClassLoader for okhttp-api//okio.AsyncTimeout$source$1.read(AsyncTimeout.kt:128)
        at PluginClassLoader for okhttp-api//okio.RealBufferedSource.indexOf(RealBufferedSource.kt:430)
        at PluginClassLoader for okhttp-api//okio.RealBufferedSource.readUtf8LineStrict(RealBufferedSource.kt:323)
        at PluginClassLoader for okhttp-api//okhttp3.internal.http1.HeadersReader.readLine(HeadersReader.kt:29)
        at PluginClassLoader for okhttp-api//okhttp3.internal.http1.Http1ExchangeCodec.readResponseHeaders(Http1ExchangeCodec.kt:180)
        at PluginClassLoader for okhttp-api//okhttp3.internal.connection.RealConnection.createTunnel(RealConnection.kt:457)
        at PluginClassLoader for okhttp-api//okhttp3.internal.connection.RealConnection.connectTunnel(RealConnection.kt:262)
        at PluginClassLoader for okhttp-api//okhttp3.internal.connection.RealConnection.connect(RealConnection.kt:201)
        at PluginClassLoader for okhttp-api//okhttp3.internal.connection.ExchangeFinder.findConnection(ExchangeFinder.kt:226)
        at PluginClassLoader for okhttp-api//okhttp3.internal.connection.ExchangeFinder.findHealthyConnection(ExchangeFinder.kt:106)
        at PluginClassLoader for okhttp-api//okhttp3.internal.connection.ExchangeFinder.find(ExchangeFinder.kt:74)
        at PluginClassLoader for okhttp-api//okhttp3.internal.connection.RealCall.initExchange$okhttp(RealCall.kt:255)
        at PluginClassLoader for okhttp-api//okhttp3.internal.connection.ConnectInterceptor.intercept(ConnectInterceptor.kt:32)
        at PluginClassLoader for okhttp-api//okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
        at PluginClassLoader for okhttp-api//okhttp3.internal.cache.CacheInterceptor.intercept(CacheInterceptor.kt:95)
        at PluginClassLoader for okhttp-api//okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
        at PluginClassLoader for okhttp-api//okhttp3.internal.http.BridgeInterceptor.intercept(BridgeInterceptor.kt:83)
        at PluginClassLoader for okhttp-api//okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
        at PluginClassLoader for okhttp-api//okhttp3.internal.http.RetryAndFollowUpInterceptor.intercept(RetryAndFollowUpInterceptor.kt:76)
        at PluginClassLoader for okhttp-api//okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
        at PluginClassLoader for okhttp-api//okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201)
 Caused: java.io.InterruptedIOException: timeout
        at PluginClassLoader for okhttp-api//okhttp3.internal.connection.RealCall.timeoutExit(RealCall.kt:398)
        at PluginClassLoader for okhttp-api//okhttp3.internal.connection.RealCall.callDone(RealCall.kt:360)
        at PluginClassLoader for okhttp-api//okhttp3.internal.connection.RealCall.noMoreExchanges$okhttp(RealCall.kt:325)
        at PluginClassLoader for okhttp-api//okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:209)
        at PluginClassLoader for okhttp-api//okhttp3.internal.connection.RealCall$AsyncCall.run(RealCall.kt:517)
 Caused: java.io.IOException: timeout
        at PluginClassLoader for kubernetes-client-api//io.fabric8.kubernetes.client.dsl.internal.OperationSupport.waitForResult(OperationSupport.java:504)
        at PluginClassLoader for kubernetes-client-api//io.fabric8.kubernetes.client.dsl.internal.BaseOperation.list(BaseOperation.java:450)
 Caused: io.fabric8.kubernetes.client.KubernetesClientException: Operation: [list]  for kind: [Secret]  with name: [null]  in namespace: [jenkins]  failed.
        at PluginClassLoader for kubernetes-client-api//io.fabric8.kubernetes.client.KubernetesClientException.launderThrowable(KubernetesClientException.java:159)
        at PluginClassLoader for kubernetes-client-api//io.fabric8.kubernetes.client.dsl.internal.BaseOperation.list(BaseOperation.java:452)
        at PluginClassLoader for kubernetes-client-api//io.fabric8.kubernetes.client.dsl.internal.BaseOperation.list(BaseOperation.java:418)
        at PluginClassLoader for kubernetes-client-api//io.fabric8.kubernetes.client.dsl.internal.BaseOperation.list(BaseOperation.java:97)
        at PluginClassLoader for kubernetes-credentials-provider//com.cloudbees.jenkins.plugins.kubernetes_credentials_provider.KubernetesCredentialProvider.startWatchingForSecrets(KubernetesCredentialProvider.java:123)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
        at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(Unknown Source)
        at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(Unknown Source)
        at java.base/java.lang.reflect.Method.invoke(Unknown Source)
        at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:109)
        at hudson.init.TaskMethodFinder$TaskImpl.run(TaskMethodFinder.java:185)
        at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:304)
        at jenkins.model.Jenkins$5.runTask(Jenkins.java:1144)
        at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:221)
        at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:120)
        at jenkins.security.ImpersonatingExecutorService$1.run(ImpersonatingExecutorService.java:68)
        at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
        at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
        at java.base/java.lang.Thread.run(Unknown Source)
 2025-04-03 15:39:27.467+0000 [id=33]   INFO    c.c.j.p.k.KubernetesCredentialProvider#reconnectLater: Attempting to reconnect Kubernetes client in 5 mins

Any help is appreciated.

I was thinking that this might be an issue with the plugin not able to connect to Kubernetes.default.svc, that’s why I added the nonProxy in JAVA_OPTS but that didn’t solve the issue.

I’m running jenkins:lts 2.492.2

2

Have you configured the proxy settings in Jenkins itself?

3

Do you mean once I access the GUI?
I tried adding the proxy in Manage Jenkins > System and I validated the configuration as well. The issue remains the same. “Socked closed” and timeout… the Kubernetes CredentialProvider attempts to reconnect every 5 minutes…

4

I have confirmed the issue I related to proxy. I deployed another operator in another Kubernetes instance that is not behind a proxy and I don’t see the issue. same K8s version, same values file for the operator, same CR definition minus the proxy settings in JAVA_OPTS

5

Ok, I solve my own problem, but this is something that other people with a similar infrastructure might run into.

IF you have a kubernetes cluster and that cluster needs to use a PROXY for external access such as downloading plugins when using the jenkins-operator to instantiate jenkins, you will possibly run into this situation. The Kubernetes Credentials Plugin is extremely finicky when it comes to proxy settings.

The proxy settings for Jenkins can be configured via ENV variables or via JAVA_OPTS. The Kubernetes Credentials Provider couldn’t care less about JAVA_OPTS, in fact, it couldn’t care less about canonical names such as kubernetes.default.svc either. Nope, it will only use the Kubernetes API Service IP. Therefore, the only way to get the Kubernetes service provider to work is to set HTTPS_PROXY, HTTP_PROXY, and NO_PROXY (all 3 must be present) and the NO_PROXY must contain the IP address of your Kubernetes API.

1 Like