Upgrade to 2.332.x Jenkins no longer works with JSM

grendel · April 21, 2022, 1:08am

I have Jenkins in an environment that requires JSM (Java Security Manager) to be enabled. Running it as a war in Tomcat. Worked fine through 2.319.3. Fails when upgrading to 2.332.x. Access denied messages to class files under /opt/tomcat/webapps/jenkins (more to the path) even though my catalina.policy gives fill permissions to this whole directory. Therefore there’s something else introduced in 2.332.x where I need to give access but it shows nothing except read access problems to class files in catalina.out.

Have verified it’s JSM because I change JSM so it has complete access to root (/) and things work. So there’s something. I cannot leave it to have complete access to the whole filesystem.

saper · April 21, 2022, 12:41pm

This is hard to tell without having the actual policy and the real output. At least the stack trace of the exception might reveal something.

You are probably aware of the discussion about the removal of JSM:

grendel · April 21, 2022, 3:17pm

Main part of the policy. Note the bottom section which is the only way to make it work so far.

grant codeBase "file:${catalina.base}/webapps/jenkins/-" {
        permission java.security.AllPermission;
};

grant codeBase "file:/jenkins/-" {
        permission java.security.AllPermission;
};

grant {
        permission java.security.AllPermission;
};

saper · April 21, 2022, 3:18pm

… and the exception you get? (stack trace)

grendel · April 21, 2022, 4:18pm

First one few things:

20-Apr-2022 15:31:10.692 SEVERE [main] org.apache.catalina.session.StandardManager.startInternal Exception loading sessions from persistent storage
        java.io.InvalidClassException: The class [org.springframework.security.web.savedrequest.DefaultSavedRequest] did not match the regular expression [java\.lang\.(?:Boolean|Integer|Long|Number|String)|org\.apache\.catalina\.realm\.GenericPrincipal\$SerializablePrincipal|\[Ljava.lang.String;] for classes allowed to be deserialized
                at org.apache.catalina.util.CustomObjectInputStream.resolveClass(CustomObjectInputStream.java:154)
                at java.base/java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1997)
                at java.base/java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1864)

Then the rest are like this with access denied to different classes:

20-Apr-2022 15:31:18.011 INFO [pool-6-thread-1] jenkins.InitReactorRunner$1.onAttained Prepared all plugins
java.util.logging.ErrorManager: 5
com.google.common.util.concurrent.UncheckedExecutionException: java.security.AccessControlException: access denied ("java.io.FilePermission" "/opt/tomcat/webapps/jenkins/WEB-INF/classes/jenkins/security/CustomClassFilter$Static.class" "read")
        at com.google.common.cache.LocalCache$Segment.get(LocalCache.java:2055)
        at com.google.common.cache.LocalCache.get(LocalCache.java:3966)
        at com.google.common.cache.LocalCache.getOrLoad(LocalCache.java:3989)
        at com.google.common.cache.LocalCache$LocalLoadingCache.get(LocalCache.java:4950)
        at com.google.common.cache.LocalCache$LocalLoadingCache.getUnchecked(LocalCache.java:4956)
        at com.google.inject.internal.util.StackTraceElements.forMember(StackTraceElements.java:67)
        at com.google.inject.internal.SourceFormatter.formatMember(SourceFormatter.java:91)
        at com.google.inject.internal.SourceFormatter.formatInjectionPoint(SourceFormatter.java:97)
        at com.google.inject.internal.SourceFormatter.format(SourceFormatter.java:49)
        at com.google.inject.internal.GenericErrorDetail.formatDetail(GenericErrorDetail.java:26)
        at com.google.inject.spi.ErrorDetail.format(ErrorDetail.java:63)
        at com.google.inject.internal.Messages.formatMessages(Messages.java:90)

grendel · April 21, 2022, 4:34pm

The first one may be unrelated from my research. The main ones are the access denied to the classes.

saper · April 21, 2022, 4:35pm

How do you set Jenkins home directory? (Looking at Remove JNDI support by basil · Pull Request #6111 · jenkinsci/jenkins · GitHub)

grendel · April 21, 2022, 4:36pm

I was setting like you are hinting at via JNDI. Still have that file but I am now also setting via setenv.sh that is in the /opt/tomcat/bin directory which is chmodded to be able to execute, owned by tomcat user.

saper · April 21, 2022, 4:38pm

As usual, please do not tell the story, just show the files…

saper · April 21, 2022, 4:46pm

Did you upgrade Tomcat by chance at the same time?
Can you try setting -Djava.security.debug=access in your CATALINA_OPTS and restart a whole thing? Expect huge logs.

grendel · April 21, 2022, 4:58pm

Hi there. Yes previously was at 9.0.56 and upgraded to the latest, 9.0.62 of Tomcat. But did that on the prior version of Jenkins 319.1 and then verified I could upgrade to 319.3 succesfully.

I will try that.

Here’s is my setenv.sh

LD_LIBRARY_PATH=$LD_LIBRARY_PATH:/usr/local/apr/lib
export LD_LIBRARY_PATH
export JENKINS_HOME=/jenkins

saper · April 21, 2022, 5:34pm

Looking at:

grendel:

20-Apr-2022 15:31:10.692 SEVERE [main] org.apache.catalina.session.StandardManager.startInternal Exception loading sessions from persistent storage
        java.io.InvalidClassException: The class [org.springframework.security.web.savedrequest.DefaultSavedRequest] did not match the regular expression [java\.lang\.(?:Boolean|Integer|Long|Number|String)|org\.apache\.catalina\.realm\.GenericPrincipal\$SerializablePrincipal|\[Ljava.lang.String;] for classes allowed to be deserialized
                at org.apache.catalina.util.CustomObjectInputStream.resolveClass(CustomObjectInputStream.java:154)
                at java.base/java.io.ObjectInputStream.readNonProxyDesc(ObjectInputStream.java:1997)
                at java.base/java.io.ObjectInputStream.readClassDesc(ObjectInputStream.java:1864)

To fix this one, one needs to define sessionAttributeValueClassNameFilter property of the session manager (one of these)

https://tomcat.apache.org/tomcat-9.0-doc/config/manager.html
https://tomcat.apache.org/tomcat-9.0-doc/config/cluster-manager.html

Adding org.springframework.security.web.savedrequest.DefaultSavedRequest and possibly some other names to the pattern may resolve the issue. (To reproduce, you need to restart it in a way to have open web sessions).

And yes, this is related to the security manager, because Tomcat checks this only when running under security manager. I am not sure Jenkins can do anything about it.

grendel · April 22, 2022, 12:41pm

We did the following to resolve at least the complaints about class files.

grant {
         permission java.io.FilePermission
         "${catalina.base}${file.separator}webapps${file.separator}jenkins${file.separator}WEB-INF${file.separator}-", "read";
};

grendel · April 22, 2022, 12:43pm

Debugging really didn’t show but it did indeed have more logs. But then these errors also were there. At this point we decided to leave in the line allowing JSM full access to get past this for right now. We may visit this again.

access: access denied ("java.util.PropertyPermission" "org.jenkinsci.remoting.protocol.IOHub.selectorWakeupTimeout" "read")

21-Apr-2022 20:55:37.869 WARNING [Jenkins initialization thread] hudson.ExtensionFinder$GuiceFinder$FaultTolerantScope$1.error Failed to instantiate Key[type=jenkins.slaves.IOHubProvider, annotation=[none]]; skipping this component
        com.google.inject.ProvisionException: Unable to provision, see the following errors:

1) [Guice/ErrorInjectingConstructor]: ExceptionInInitializerError
  at IOHubProvider.<init>(IOHubProvider.java:51)

Learn more:
  https://github.com/google/guice/wiki/ERROR_INJECTING_CONSTRUCTOR

1 error

21-Apr-2022 20:55:37.871 SEVERE [Jenkins initialization thread] jenkins.model.Jenkins$16.onTaskFailed Failed IOHubProvider.cleanUp
        java.lang.NullPointerException
                at java.base/java.lang.reflect.Method.invoke(Method.java:559)
                at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:109)
                at hudson.init.TaskMethodFinder$TaskImpl.run(TaskMethodFinder.java:185)
                at org.jvnet.hudson.reactor.Reactor.runTask(Reactor.java:305)
                at org.jvnet.hudson.reactor.Reactor$2.run(Reactor.java:222)
                at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:121)
                at org.jvnet.hudson.reactor.Reactor$Node.runIfPossible(Reactor.java:143)
                at org.jvnet.hudson.reactor.Reactor$Node.run(Reactor.java:132)
                at org.jvnet.hudson.reactor.Reactor$Node.runIfPossible(Reactor.java:143)
                at org.jvnet.hudson.reactor.Reactor.execute(Reactor.java:285)
                at jenkins.model.Jenkins._cleanUpRunTerminators(Jenkins.java:3649)
                at jenkins.model.Jenkins.cleanUp(Jenkins.java:3574)
                at hudson.WebAppMain$3.run(WebAppMain.java:268)

21-Apr-2022 20:55:37.887 INFO [Jenkins initialization thread] jenkins.model.Jenkins._cleanUpShutdownPluginManager Stopping plugin manager
access: access denied ("java.lang.RuntimePermission" "setContextClassLoader")

saper · April 22, 2022, 1:42pm

Yes, you should add those two permissions:

java.util.PropertyPermission org.jenkinsci.remoting.protocol.IOHub.selectorWakeupTimeout read
java.lang.RuntimePermission setContextClassLoader

Regarding your file problem: sure "${catalina.base}${file.separator}webapps${file.separator}jenkins${file.separator}WEB-INF${file.separator}-", "read"; may help, but I think the issue is somewhere else - somehow the code is not running from your codeBase.

Can you try to figure out what was the codeBase used when trying to access files under WEB-INF?

Do you have additional libraries installed in Tomcat that are shared between apps?

There were two changes that might have affected you: Guava update (more likely) and the change of the classloader.

grendel · April 26, 2022, 1:57pm

I am trying to add those 2 permissions as follows.

grant codeBase “file:${catalina.base}/webapps/jenkins/-” {
permission java.util.PropertyPermission “org.jenkinsci.remoting.protocol.IOHub.selectorWakeupTimeout”, “read”;
permission java.lang.RuntimePermission “setContextClassLoader”;

I can try though not sure how to figure out the codeBase. Only app running on Tomcat is Jenkins and this is brand new install nothing even in it yet.

No additional libraries except the ones to support the APR connector which is needed. See the LD_LIBRARY_PATH in my setenv.sh above

grendel · April 26, 2022, 5:22pm

26-Apr-2022 13:19:35.277 WARNING [Contributed.load] hudson.ExtensionFinder$GuiceFinder$FaultTolerantScope$1.error Failed to instantiate Key[type=jenkins.security.CustomClassFilter$Static, annotation=[none]]; skipping this component
        com.google.inject.ProvisionException: Unable to provision, see the following errors:

1) [Guice/ErrorInjectingConstructor]: AccessControlException: access denied ("java.util.PropertyPermission" "hudson.remoting.ClassFilter" "read")
  at CustomClassFilter$Static.<init>(CustomClassFilter.java:96)

Learn more:
  https://github.com/google/guice/wiki/ERROR_INJECTING_CONSTRUCTOR

1 error

======================
Full classname legend:
======================
AccessControlException:   "java.security.AccessControlException"
CustomClassFilter$Static: "jenkins.security.CustomClassFilter$Static"
========================
End of classname legend:

grendel · April 26, 2022, 5:22pm

26-Apr-2022 13:19:35.666 WARNING [RekeySecretAdminMonitor.scanOnReboot] hudson.ExtensionFinder$GuiceFinder$FaultTolerantScope$1.error Failed to instantiate Key[type=jenkins.security.RekeySecretAdminMonitor, annotation=[none]]; skipping this component
com.google.inject.ProvisionException: Unable to provision, see the following errors:

[Guice/ErrorInjectingConstructor]: AccessControlException: access denied (“java.io.FilePermission” “/jenkins/secret.key.not-so-secret” “read”)
at RekeySecretAdminMonitor.(RekeySecretAdminMonitor.java:56)

Learn more:
ERROR_INJECTING_CONSTRUCTOR · google/guice Wiki · GitHub

1 error

grendel · April 26, 2022, 5:23pm

26-Apr-2022 13:19:35.719 SEVERE [pool-6-thread-1] jenkins.InitReactorRunner$1.onTaskFailed Failed RekeySecretAdminMonitor.scanOnReboot
java.lang.NullPointerException
at java.base/java.lang.reflect.Method.invoke(Method.java:559)
at hudson.init.TaskMethodFinder.invoke(TaskMethodFinder.java:109)

grendel · April 26, 2022, 5:25pm

======================
Full classname legend:
======================
AccessControlException:             "java.security.AccessControlException"
ArchitectureMonitor$DescriptorImpl: "hudson.node_monitors.ArchitectureMonitor$DescriptorImpl"
========================
End of classname legend:
========================

                at com.google.inject.internal.InternalProvisionException.toProvisionException(InternalProvisionException.java:251)

Topic		Replies	Views
Jenkins 2.401.3 after upgrade issue Ask a question	2	649	December 14, 2023
Jenkins Stopped after upgrading from Java 11 to JAVA 17 Community	2	101	August 2, 2024
Adom group issue after jenkins upgrade [JENKINS-71092] Ask a question	1	570	April 19, 2023
Jenkins upgrade to 2.289.3 failed & can't login with LDAP Using Jenkins	5	5848	October 19, 2021
Jenkins 2.346.3 Tomcat 9 APIs Ask a question	7	943	November 8, 2022

Upgrade to 2.332.x Jenkins no longer works with JSM

Related topics