Hello Community!
I am seeking help if anyone might have an idea on how to resolve it or where can i look for answers. I have been tirelessly looking over the net for a week now. and every article stack overflow brings me to either nowhere or rabbits hole
We have a Jenkins Setup running on an Azure VM on an Intranet Env. All working fine except for Azure AD/SSO integration. components setup and config as follows
- Azure E16VM
-Azure Application GW
-Internal FQDN
-Digicert SSL Certificate
-RHEL 7.9
-Tomcat 7.0.76
-OpenJDK 1.8.0_322
-Jenkins Ver 2.332.3
-Azure App Registration MSGraph Grants (all the permissions i can possibly grant actually)
DirectoryReadAll (app/delegated)
GroupReadAll (app/delegated)
PeopleReadAll (app/delegated)
UserReadAll (app/delegated)
Problem:
After configuring the Client creds on Azure AD and testing to pull user I’m getting this below errors
FROM Jenkins UI (verify service principal)
java.nio.channels.ClosedChannelException
at io.netty.handler.ssl.SslHandler.channelInactive(SslHandler.java:1064)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
at io.netty.channel.AbstractChannelHandlerContext.fireChannelInactive(AbstractChannelHandlerContext.java:241)
at io.netty.channel.DefaultChannelPipeline$HeadContext.channelInactive(DefaultChannelPipeline.java:1405)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:262)
at io.netty.channel.AbstractChannelHandlerContext.invokeChannelInactive(AbstractChannelHandlerContext.java:248)
at io.netty.channel.DefaultChannelPipeline.fireChannelInactive(DefaultChannelPipeline.java:901)
at io.netty.channel.AbstractChannel$AbstractUnsafe$7.run(AbstractChannel.java:813)
at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469)
at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:391)
at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
at java.lang.Thread.run(Thread.java:750)
Also: java.lang.Exception: #block terminated with an error
at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:99)
Caused: reactor.core.Exceptions$ReactiveException
at reactor.core.Exceptions.propagate(Exceptions.java:392)
at reactor.core.publisher.BlockingSingleSubscriber.blockingGet(BlockingSingleSubscriber.java:97)
at reactor.core.publisher.Mono.block(Mono.java:1707)
at com.azure.identity.implementation.HttpPipelineAdapter.send(HttpPipelineAdapter.java:85)
at com.microsoft.aad.msal4j.HttpHelper.executeHttpRequestWithRetries(HttpHelper.java:96)
at com.microsoft.aad.msal4j.HttpHelper.executeHttpRequest(HttpHelper.java:49)
Caused: com.microsoft.aad.msal4j.MsalClientException
at com.microsoft.aad.msal4j.HttpHelper.executeHttpRequest(HttpHelper.java:53)
at com.microsoft.aad.msal4j.AadInstanceDiscoveryProvider.executeRequest(AadInstanceDiscoveryProvider.java:232)
at com.microsoft.aad.msal4j.AadInstanceDiscoveryProvider.sendInstanceDiscoveryRequest(AadInstanceDiscoveryProvider.java:187)
at com.microsoft.aad.msal4j.AadInstanceDiscoveryProvider.doInstanceDiscoveryAndCache(AadInstanceDiscoveryProvider.java:285)
at com.microsoft.aad.msal4j.AadInstanceDiscoveryProvider.getMetadataEntry(AadInstanceDiscoveryProvider.java:57)
at com.microsoft.aad.msal4j.AuthenticationResultSupplier.getAuthorityWithPrefNetworkHost(AuthenticationResultSupplier.java:41)
at com.microsoft.aad.msal4j.AcquireTokenByAuthorizationGrantSupplier.execute(AcquireTokenByAuthorizationGrantSupplier.java:59)
at com.microsoft.aad.msal4j.AcquireTokenByClientCredentialSupplier.acquireTokenByClientCredential(AcquireTokenByClientCredentialSupplier.java:63)
at com.microsoft.aad.msal4j.AcquireTokenByClientCredentialSupplier.execute(AcquireTokenByClientCredentialSupplier.java:49)
at com.microsoft.aad.msal4j.AuthenticationResultSupplier.get(AuthenticationResultSupplier.java:71)
at com.microsoft.aad.msal4j.AuthenticationResultSupplier.get(AuthenticationResultSupplier.java:20)
at java.util.concurrent.CompletableFuture$AsyncSupply.run(CompletableFuture.java:1604)
at java.util.concurrent.CompletableFuture$AsyncSupply.exec(CompletableFuture.java:1596)
at java.util.concurrent.ForkJoinTask.doExec(ForkJoinTask.java:289)
at java.util.concurrent.ForkJoinPool$WorkQueue.runTask(ForkJoinPool.java:1056)
at java.util.concurrent.ForkJoinPool.runWorker(ForkJoinPool.java:1692)
at java.util.concurrent.ForkJoinWorkerThread.run(ForkJoinWorkerThread.java:175)
Caused: java.util.concurrent.ExecutionException
at java.util.concurrent.CompletableFuture.reportGet(CompletableFuture.java:357)
at java.util.concurrent.CompletableFuture.get(CompletableFuture.java:1908)
at com.microsoft.graph.httpcore.AuthenticationHandler.intercept(AuthenticationHandler.java:55)
Caused: java.io.IOException
at com.microsoft.graph.httpcore.AuthenticationHandler.intercept(AuthenticationHandler.java:67)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
at com.microsoft.graph.httpcore.TelemetryHandler.intercept(TelemetryHandler.java:69)
at okhttp3.internal.http.RealInterceptorChain.proceed(RealInterceptorChain.kt:109)
at okhttp3.internal.connection.RealCall.getResponseWithInterceptorChain$okhttp(RealCall.kt:201)
at okhttp3.internal.connection.RealCall.execute(RealCall.kt:154)
at com.microsoft.graph.http.CoreHttpProvider.sendRequestInternal(CoreHttpProvider.java:407)
Caused: com.microsoft.graph.core.ClientException: Error executing the request
at com.microsoft.graph.http.CoreHttpProvider.sendRequestInternal(CoreHttpProvider.java:410)
at com.microsoft.graph.http.CoreHttpProvider.send(CoreHttpProvider.java:225)
at com.microsoft.graph.http.CoreHttpProvider.send(CoreHttpProvider.java:202)
at com.microsoft.graph.http.BaseRequest.send(BaseRequest.java:335)
at com.microsoft.graph.requests.UserRequest.get(UserRequest.java:138)
at com.microsoft.jenkins.azuread.AzureSecurityRealm$DescriptorImpl.doVerifyConfiguration(AzureSecurityRealm.java:784)
at java.lang.invoke.MethodHandle.invokeWithArguments(MethodHandle.java:627)
at org.kohsuke.stapler.Function$MethodFunction.invoke(Function.java:398)
at org.kohsuke.stapler.Function$InstanceFunction.invoke(Function.java:410)
at org.kohsuke.stapler.Function.bindAndInvoke(Function.java:208)
at org.kohsuke.stapler.Function.bindAndInvokeAndServeResponse(Function.java:141)
at org.kohsuke.stapler.MetaClass$11.doDispatch(MetaClass.java:558)
at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:59)
at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:898)
at org.kohsuke.stapler.MetaClass$4.doDispatch(MetaClass.java:289)
at org.kohsuke.stapler.NameBasedDispatcher.dispatch(NameBasedDispatcher.java:59)
at org.kohsuke.stapler.Stapler.tryInvoke(Stapler.java:766)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:898)
at org.kohsuke.stapler.Stapler.invoke(Stapler.java:694)
at org.kohsuke.stapler.Stapler.service(Stapler.java:240)
at javax.servlet.http.HttpServlet.service(HttpServlet.java:731)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:303)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.tomcat.websocket.server.WsFilter.doFilter(WsFilter.java:52)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:157)
at org.jenkinsci.plugins.ssegateway.Endpoint$SSEListenChannelFilter.doFilter(Endpoint.java:248)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
at com.atlassian.bitbucket.jenkins.internal.applink.oauth.serviceprovider.auth.OAuth1aRequestFilter.doFilter(OAuth1aRequestFilter.java:91)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
at hudson.plugins.audit_trail.AuditTrailFilter.doFilter(AuditTrailFilter.java:112)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
at jenkins.telemetry.impl.UserLanguages$AcceptLanguageFilter.doFilter(UserLanguages.java:129)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
at jenkins.security.ResourceDomainFilter.doFilter(ResourceDomainFilter.java:81)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
at io.jenkins.blueocean.ResourceCacheControl.doFilter(ResourceCacheControl.java:134)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
at io.jenkins.blueocean.auth.jwt.impl.JwtAuthenticationFilter.doFilter(JwtAuthenticationFilter.java:60)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:239)
at net.bull.javamelody.MonitoringFilter.doFilter(MonitoringFilter.java:215)
at net.bull.javamelody.PluginMonitoringFilter.doFilter(PluginMonitoringFilter.java:88)
at org.jvnet.hudson.plugins.monitoring.HudsonMonitoringFilter.doFilter(HudsonMonitoringFilter.java:121)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
at jenkins.metrics.impl.MetricsFilter.doFilter(MetricsFilter.java:125)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
at javax.servlet.FilterChain$doFilter.call(Unknown Source)
at com.ceilfors.jenkins.plugins.jiratrigger.ExceptionLoggingFilter.doFilter(ExceptionLoggingFilter.groovy:29)
at hudson.util.PluginServletFilter$1.doFilter(PluginServletFilter.java:154)
at hudson.util.PluginServletFilter.doFilter(PluginServletFilter.java:160)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at hudson.security.csrf.CrumbFilter.doFilter(CrumbFilter.java:154)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:94)
at jenkins.security.AcegiSecurityExceptionFilter.doFilter(AcegiSecurityExceptionFilter.java:52)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
at hudson.security.UnwrapSecurityExceptionFilter.doFilter(UnwrapSecurityExceptionFilter.java:54)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:122)
at org.springframework.security.web.access.ExceptionTranslationFilter.doFilter(ExceptionTranslationFilter.java:116)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
at org.springframework.security.web.authentication.AnonymousAuthenticationFilter.doFilter(AnonymousAuthenticationFilter.java:109)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:136)
at org.springframework.security.web.authentication.rememberme.RememberMeAuthenticationFilter.doFilter(RememberMeAuthenticationFilter.java:93)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:219)
at org.springframework.security.web.authentication.AbstractAuthenticationProcessingFilter.doFilter(AbstractAuthenticationProcessingFilter.java:213)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
at jenkins.security.BasicHeaderProcessor.doFilter(BasicHeaderProcessor.java:97)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:110)
at org.springframework.security.web.context.SecurityContextPersistenceFilter.doFilter(SecurityContextPersistenceFilter.java:80)
at hudson.security.HttpSessionContextIntegrationFilter2.doFilter(HttpSessionContextIntegrationFilter2.java:63)
at hudson.security.ChainedServletFilter$1.doFilter(ChainedServletFilter.java:99)
at hudson.security.ChainedServletFilter.doFilter(ChainedServletFilter.java:111)
at hudson.security.HudsonFilter.doFilter(HudsonFilter.java:172)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.kohsuke.stapler.compression.CompressionFilter.doFilter(CompressionFilter.java:53)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at hudson.util.CharacterEncodingFilter.doFilter(CharacterEncodingFilter.java:86)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.kohsuke.stapler.DiagnosticThreadNameFilter.doFilter(DiagnosticThreadNameFilter.java:30)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at jenkins.security.SuspiciousRequestFilter.doFilter(SuspiciousRequestFilter.java:38)
at org.apache.catalina.core.ApplicationFilterChain.internalDoFilter(ApplicationFilterChain.java:241)
at org.apache.catalina.core.ApplicationFilterChain.doFilter(ApplicationFilterChain.java:208)
at org.apache.catalina.core.StandardWrapperValve.invoke(StandardWrapperValve.java:218)
at org.apache.catalina.core.StandardContextValve.invoke(StandardContextValve.java:110)
at org.apache.catalina.authenticator.AuthenticatorBase.invoke(AuthenticatorBase.java:607)
at org.apache.catalina.core.StandardHostValve.invoke(StandardHostValve.java:169)
at org.apache.catalina.valves.ErrorReportValve.invoke(ErrorReportValve.java:103)
at org.apache.catalina.valves.AccessLogValve.invoke(AccessLogValve.java:962)
at org.apache.catalina.core.StandardEngineValve.invoke(StandardEngineValve.java:116)
at org.apache.catalina.connector.CoyoteAdapter.service(CoyoteAdapter.java:445)
at org.apache.coyote.http11.AbstractHttp11Processor.process(AbstractHttp11Processor.java:1091)
at org.apache.coyote.AbstractProtocol$AbstractConnectionHandler.process(AbstractProtocol.java:637)
at org.apache.tomcat.util.net.JIoEndpoint$SocketProcessor.run(JIoEndpoint.java:318)
at java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1149)
at java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:624)
at org.apache.tomcat.util.threads.TaskThread$WrappingRunnable.run(TaskThread.java:61)
at java.lang.Thread.run(Thread.java:750)
FROM tomcat service logs
Group: /system.slice/tomcat.service
└─32750 /usr/lib/jvm/jre/bin/java -classpath /usr/share/tomcat/bin/bootstrap.jar:/usr/share/tomcat/bin/tomcat-juli.jar:/usr/share/java/commons-daemon.jar -Dcatalina.base=/usr/share/tomcat -Dcatalina.home=/usr/share/tomcat -Djava.endorsed.dirs= -Djava.io.tmpdir=/var/cache/tomcat/temp -Djava.util.logging.config.file=/usr/share/tomcat/conf/logging.properties -Djava.util.logging.manager=org.apache.juli.ClassLoaderLogManager org.apache.catalina.startup.Bootstrap start
Aug 25 14:14:09 sgrtssj6zq0gm1i server[32750]: at io.netty.channel.AbstractChannel$AbstractUnsafe$7.run(AbstractChannel.java:813)
Aug 25 14:14:09 sgrtssj6zq0gm1i server[32750]: at io.netty.util.concurrent.AbstractEventExecutor.safeExecute(AbstractEventExecutor.java:164)
Aug 25 14:14:09 sgrtssj6zq0gm1i server[32750]: at io.netty.util.concurrent.SingleThreadEventExecutor.runAllTasks(SingleThreadEventExecutor.java:469)
Aug 25 14:14:09 sgrtssj6zq0gm1i server[32750]: at io.netty.channel.epoll.EpollEventLoop.run(EpollEventLoop.java:391)
Aug 25 14:14:09 sgrtssj6zq0gm1i server[32750]: at io.netty.util.concurrent.SingleThreadEventExecutor$4.run(SingleThreadEventExecutor.java:986)
Aug 25 14:14:09 sgrtssj6zq0gm1i server[32750]: at io.netty.util.internal.ThreadExecutorMap$2.run(ThreadExecutorMap.java:74)
Aug 25 14:14:09 sgrtssj6zq0gm1i server[32750]: at io.netty.util.concurrent.FastThreadLocalRunnable.run(FastThreadLocalRunnable.java:30)
Aug 25 14:14:09 sgrtssj6zq0gm1i server[32750]: at java.lang.Thread.run(Thread.java:750)
Aug 25 14:14:09 sgrtssj6zq0gm1i server[32750]: Aug 25, 2022 2:14:09 PM com.azure.core.util.logging.ClientLogger performDeferredLogging
Aug 25 14:14:09 sgrtssj6zq0gm1i server[32750]: SEVERE: Azure Identity => ERROR in getToken() call for scopes [https://graph.microsoft.com/.default]: reactor.core.Exceptions$ReactiveException: java.nio.channels.ClosedChannelException
Troubleshoot Preformed:
-changed the JDK version to JAVA v11 - no difference
-add almost all MSGraph Permission - no difference
-whitelist MSGraph’s FQDN from FW - no difference
Appreciate any Help. Thanks a lot