I recently upgraded to the latest jenkinsci/blueocean container. Now at startup I am getting an exception related to the Jackson 2 API Plugin failing to load.
I have tried the workaround of adding the JAXB plugin to my plugins. I have even verified in my local container build that the correct plugins are installed both the:
jackson2-api:2.13.3-285.vc03c0256d517
jaxb:2.3.6-1
These are the correct versions. The exceptions seems to occur after Hudson tries to update Jenkins
INFO hudson.PluginManager#loadDetachedPlugins: Upgraded Jenkins from version 2.332.1 to version 2.332.3. Loaded detached plugins (and dependencies):
INFO jenkins.InitReactorRunner$1#onAttained: Listed all plugins
SEVERE jenkins.InitReactorRunner$1#onTaskFailed: Failed Loading plugin Jackson 2 API Plugin v2.13.3-285.vc03c0256d517 (jackson2-api)
java.io.IOException: Failed to load: Jackson 2 API Plugin (jackson2-api 2.13.3-285.vc03c0256d517)
Update required: JAXB plugin (jaxb 2.3.0) to be updated to 2.3.6-1 or higher
at hudson.PluginWrapper.resolvePluginDependencies(PluginWrapper.java:1018)
Is my assumption correct that somehow when Hudson upgrades my Jenkins version its overwriting my plugin set in my container, and if so why?
Yes I did a docker pull on the latest. Note I have tried to roll back to several tags prior to the latest, and all of them have that same problem.
Further complicating my problem is that I am using the latest version of all of the plugins, and I haven’t been delineating specific plugin versions within my instance, so if I rollback too far in the past, my plugins stop working (note I am aware I need to correct this it just hasn’t been my highest priority until now).
Please keep in mind I inherited this configuration and have been trying to adjust it over time.
One final note, I believe if I had my plugins in a better state I could probably roll back to Jenkins 2.332.1, but as I stated several of them see to require the newer 2.332.3 version.
Plugins are stored in $JENKINS_HOME/plugins
Reference plugins (onces stored in the image) are stored in /usr/share/jenkins/ref/plugins and copied without replacing onto $JENKINS_HOME/plugins
If you want the versions baked into the docker image to override your plugins you’ll have to clear out $JENKINS_HOME/plugins first, or copy with overwrites from /usr/share/jenkins/ref/plugins to $JENKINS_HOME/plugins
Note that we did not have to upgrade to 2.332.3 to experience the issue, we also saw / see it on 2.332.1 which were running before.
I’m trying to get some more context here. @halkeye does your comment imply that upstream plugin versions do override plugin versions installed with jenkins-plugin-cli, because they are copied “without replacing”?
We will be sure to pin our plugin versions in future, but we are also wondering what’s the status here? Is the latest Blue Ocean broken for everyone using 2.332.1, 2.332.3, and more?
It’s a little difficult to get to our plugins folder as our Jenkins is on ECS Fargate and JENKINS_HOME on Elastic File System. I guess we can spin up an EC2 instance and connect to the EFS, this is a bit of a hassle but doable if we know it will help.
Unfortunately according to [JENKINS-58085] BlueOcean UI stuck in "Waiting for run to start" - Jenkins Jira, it looks like just pinning blueocean to an old version won’t help. I’d be in favour of wiping the plugins directory completely even if we have to spin up an EC2, but it still doesn’t answer the core point which is which versions of what suddenly broke? (We don’t want to install them again…)
Cloudbees abandoned blueocean a few years ago, they put in a few fixes here and there for big customers, or security fixes, but otherwise its dead. Its such a huge undertaking with nobody at the helm, its pretty stuck as it is. I recommend you try and switch to the more maintainable and upgraded classic ui, and there are some standalone plugins that take care of visualization.
Plugin manager cli I believe will install to the ref directory if jenkins home isn’t populated, if jenkins home has content at time of cli running, then it should install it there. OP was talking about the docker image, the docker image has plugins installed to ref/plugins directory, and will not overwrite existing plugins. My recommendation is to roll your own docker image with the plugins you want baked into it (see docker-jenkins/Dockerfile at 5263eb50691b59b56e16981467ff1be20b7748d4 · halkeye-docker/docker-jenkins · GitHub as an example) with a startup script that empties your jenkins_home/plugins directory. That way you always get a clean startup with the versions you expect.
Its been years since I’ve had anything to do with blueocean so I can’t help with those specifics, hopefully someone else can.
Very useful, thank you! We never would have known that Blue Ocean was deprecated / barely moving forward! It even looks like a first class citizen at Blue Ocean. Even the Jenkins Docker installation instructions include Blue Ocean in the example!
I am also talking about our Docker image, we currently have a very simple Dockerfile:
We build the images on local machines as part of a Terraform module and upload them to AWS ECR for deployment on ECS Fargate.
As far as the “startup script that empties your jenkins_home/plugins directory” goes, how can we achieve this? A web search reveals the existence of $JENKINS_HOME/init.groovy.d/, but it looks like that is run after the plugin installation, and even worse, $JENKINS_HOME is stored on AWS Elastic File System (EFS), so it is inaccessible outside the cloud. At this point, I don’t see another solution other than spinning up an EC2 instance to connect to the EFS or connecting to the EFS from my workstation which involves some network changes, and wipe the plugin directory. Have I missed anything?
So the helm chart actually has explicit support for doing this, and I don’t use AWS,so I haven’t looked into it for a while, but what I would do is:
Find out what the entrypoint is for the docker image
(looking at Docker Hub gives me ENTRYPOINT ["/usr/bin/tini" "--" "/usr/local/bin/jenkins.sh"]) and change it to call your own script before calling the existing one.
For “can’t really help”, I’ll take it, thanks again!
In the end I had an EC2 instance in the relevant subnet so I was able to clean the directory out manually, and all is now well without blueocean (and dependencies).
But for the record, my Docker-fu is still at a beginning stage. I actually did see the official image Dockerfile that you refer to, and I didn’t understand (until now and searching the web) that ENTRYPOINT can be overridden. So helpful, much appreciated.
@halkeye, thanks for the update on BlueOcean. I totally agree with @antgel from DockerHub it completely looks like it is moving forward not barely being maintained. I’ll try a couple of things in this vein that you have described like switching to the standard Jenkins container. I am going to attempt to rollback to 2.332.1, and specifically document all the plugin versions. I will also spend some more time cleaning up those plugin versions. Thanks again for the input @halkeye!!
And UI/UX Special Interest Group has put a lot of work into modernizing the css and javascript so we can start bringing in some javascript functionality (one of the reasons blueocean exists as a complete rewrite is jenkins used protoype.js which breaks most modern javascript). They are always looking for more help if you know anyone interested.
Hi guys, is there any specific solution for this issue? im using helm chart for jenkins on aks. Met the same issue last week. Tried roll back to earlier jenkins docker tag or add jaxb in the plugin property of value.yaml file, Neither worked.
