Jenkins Upgrade Error - no valid crumb

Hey guys,
we are currently on 2.107.3 and I was trying to upgrade either to the latest or to 2.277 version but as soon as the upgrade is done I am getting this error message on login: HTTP Erroor 403 No valid crub was included in the request and it brings me to this URL: http://server1:8080/j_spring_security_check.
I tried to disable CSRF Protection in Configure Security section and I also set excludeClientIPFromCrumb to true in the config file but I am still getting that error.
Any help would be appreciated.
Thank you,
T.

Are you using active directory plugin? Most people with this issue are and need to update it.

I recommend disabling security, updating plugins then re-enabling. Or using the jenkins plugin cli to do it on the disk.

Then once that’s done, try again, and check the logs in the $JENKINS_HOME/logs or the logs under /manage to see whats erroring.

cc @MarkEWaite this might be a good FAQ question if we ever get one on jenkins.io

Gavin,
are you suggesting updating plugins before we upgrade jenkins? If yes, I don’t think I can do that as I get this warning in my current version:

Warning: This plugin is built for Jenkins 2.235.5 or newer. Jenkins will refuse to load this plugin if installed.
Warning: This plugin requires dependent plugins that require Jenkins 2.235.5 or newer. Jenkins will refuse to load the dependent plugins requiring a newer version of Jenkins, and in turn loading this plugin will fail.

Is there a way to do it after even if I can’t log in?

Thank you,
T.

As I said, you can disable security and do the upgrade, or you can upgrade via the cli tool, both won’t require you to login.

I would recommend one of:

  1. upgrading to the latest available release for the version you have, then upgrade jenkins, then upgrade again.
  2. Disabling security, upgrading everything
  3. Upgrade everything via cli (my personal preference)

I’ve preferred a slightly different approach for “long jump upgrades”. My steps were:

  1. Install a fresh copy of the target Jenkins version on a Docker image, confirm it works
  2. Install the latest version of matching plugins on the Docker image with plugins.txt, confirm it works
  3. Copy system configuration files from existing Jenkins to Docker image, confirm it works
  4. Copy job configuration files from existing Jenkins to Docker image, confirm it works
  5. If job history must be preserved, then copy job history to the Docker image, confirm it works

Once those steps are complete and confirmed they are working, backup the existing system and replace the files on it with the files created in the Docker image.

An example of that type of thing is available at GitHub - MarkEWaite/docker-lfs at lts-with-plugins

@MarkEWaite,
I like this approach but we don’t use Docker images here but I can build a brand new VM and install all from scratch.
However, is there a wiki or something that I can follow to know which files to copy (from your #3,4,5 above)?
Once I confirm all is working I can just decomission old server.

T.

I recommend the Docker approach even if you ultimately use virtual machines because it is faster to iterate on definitions of Docker images than to recycle virtual machines. The technique can be used with virtual machines but it takes longer for each iteration than with a container image.

Try Explaining $JENKINS_HOME and Where does Jenkins store configuration files for the jobs it runs? - Stack Overflow