Jenkins Controller Restart Resets Location URL to localhost:8080, Disconnecting Agent Nodes

Jenkins Controller Version: 2.401.2 LTS

We have a Jenkins controller installed and configured using the official Jenkins Helm chart on an EKS cluster, with all resource provisioning managed by Terraform.

Currently, it is functioning well and is accessible at https://jenkins-utility.example.com/ using our LDAP credentials. All agent nodes connect to same jenkins endpoint.

However, when the Jenkins controller is restarted either via the URL using safe restart or due to the pod being killed/restarted, I can still access Jenkins using the same URL. Despite this, all agent nodes get disconnected because the Jenkins location URL resets to localhost:8080 instead of https://jenkins-utility.example.com/. Please see the attached image for reference.

Below is the Jenkins Configuration as Code (CasC) configuration we are using:

jenkins:
  authorizationStrategy:
    projectMatrix:
      entries:
        - group:
            name: "authenticated"
            permissions:
              - "Agent/Configure"
              - "Agent/Connect"
              - "Agent/Create"
              - "Agent/Delete"
              - "Agent/Disconnect"
              - "Credentials/View"
              - "Job/Build"
              - "Job/Cancel"
              - "Job/Configure"
              - "Job/Create"
              - "Job/Delete"
              - "Job/Discover"
              - "Job/Move"
              - "Job/Read"
              - "Job/Workspace"
              - "Overall/Read"
              - "Overall/SystemRead"
              - "Run/Replay"
              - "View/Configure"
              - "View/Create"
              - "View/Delete"
              - "View/Read"
        - group:
            name: "jenkins-admins"
            permissions:
              - "Overall/Administer"
        - user:
            name: "anonymous"
            permissions:
              - "Job/Discover"
              - "Job/ViewStatus"
              - "Overall/Read"
        - user:
            name: "altif@example.com"
            permissions:
              - "Overall/Administer"
  securityRealm:
    activeDirectory:
      bindPassword: "xxxxxxxx"
      cache:
        size: 1000
        ttl: 3600
      customDomain: true
      domains:
        - bindPassword: "xxxxxxxx"
          name: "cor.example.com"
          servers: "ldap.example.com:3268"
          tlsConfiguration: TRUST_ALL_CERTIFICATES
      groupLookupStrategy: TOKENGROUPS
      removeIrrelevantGroups: false
      requireTLS: false
      startTls: true
  disableRememberMe: false
  mode: NORMAL
  numExecutors: 2
  labelString: "controller"
  projectNamingStrategy: "standard"
  markupFormatter:
    plainText
  clouds:
  - kubernetes:
      containerCapStr: "10"
      defaultsProviderTemplate: ""
      connectTimeout: "5"
      readTimeout: "15"
      jenkinsUrl: "http://jenkins-utility-prod.jenkins-utility-prod.svc.cluster.local:8080"
      jenkinsTunnel: "jenkins-utility-prod-agent.jenkins-utility-prod.svc.cluster.local:50000"
      maxRequestsPerHostStr: "32"
      name: "kubernetes"
      namespace: "jenkins-utility-prod"
      serverUrl: "https://kubernetes.default"
  crumbIssuer:
    standard:
      excludeClientIPFromCrumb: false
security:
  apiToken:
    creationOfLegacyTokenEnabled: false
    tokenGenerationOnCreationEnabled: false
    usageStatisticsEnabled: true
unclassified:
  location:
    adminAddress: devops@example.com
    url: https://jenkins-utility.example.com/

I have spent many hours troubleshooting this issue but have not found a solution yet. Any assistance or references to resolve this issue would be greatly appreciated.

Thanks.

Hello and welcome to this community, @altif. :wave:

Is your jcasc file located in /usr/share/jenkins/ref/jenkins.yaml?

Thanks @poddingue for the response, appreciate it.

No, it is placed under /var/jenkins_home/casc_configs path.

My only experience with Jcasc is through Docker, and I’ve been bitten several times when I forgot to put the Jcasc file in /usr/share/jenkins/ref, but I don’t know if that applies to “standard” Linux installations. :thinking:

Thanks again :slight_smile:

Adding bit more details if this can help.
As shared in previous thread, our Jenkins controller CasC configuration is located at /var/jenkins_home/casc_configs, with the configuration file named jcasc-default-config.yaml, which I have shared in the issue description. When inspecting the pod, the environment variable is correctly set to the specified path:

CASC_JENKINS_CONFIG=/var/jenkins_home/casc_configs

The /var/jenkins_home/ directory is mounted on a persistent volume (EBS gp3).

jenkins@jenkins-utility-prod-0:~/casc_configs$ env | grep -i casc_jenkins
CASC_JENKINS_CONFIG=/var/jenkins_home/casc_configs

jenkins@jenkins-utility-prod-0:~/casc_configs$ pwd
/var/jenkins_home/casc_configs

jenkins@jenkins-utility-prod-0:~/casc_configs$ ls -ltrh
total 4.0K
-rw-rw-r-- 1 jenkins jenkins 2.7K Jun  6 14:24 jcasc-default-config.yaml

jenkins@jenkins-utility-prod-0:~/casc_configs$ df -hT /var/jenkins_home/
jenkins@jenkins-utility-prod-0:~/casc_configs$ df -hT /var/jenkins_home/
Filesystem     Type  Size  Used Avail Use% Mounted on
/dev/nvme2n1   ext4  492G   30G  463G   7% /var/jenkins_home

The issue I am facing is that whenever the Jenkins controller restarts, the Jenkins Location URL isn’t updated according to the CasC file. Surprisingly, if I navigate to https://jenkins-utility.example.com/manage/configuration-as-code/ and click on the Reload existing configuration option under Actions, the CasC configuration reloads correctly and reflects properly. However, upon restarting Jenkins, the URL resets to http://localhost:8080.

I’m not sure how jenkins controller constructing/reading this URL from and resetting it to localhost:8080.

Is there a way to force Jenkins to reload the CasC configuration whenever it restarts? Is there any cache or something that needs to be cleaned up :thinking:

Any pointers or references would be highly appreciated!

Does the helm chart also define the location in some way?

yes, we have customized jenkins-values.yaml where I’m specifiying the jenkins url. BASE_URL is jenkins endpoint passed in terraform deployment.

## `controller.jenkinsUrl` to change the url definition.
  jenkinsUrl: "${BASE_URL}"

For reference:

I guess it should be qualified as tinkering instead of proper method, but whenever I change anything in the Jcasc file, and after restarting Jenkins, I trigger a Jcasc reload thanks to the REST API.
There must be a clean way of doing that. :blush:

Thank you @mawinter69 @poddingue for your prompt responses and willingness to assist, much appreciated!

Although, the initial suggestions didn’t resolve my issue, I continued my investigation. After exploring and digging JENKINS_HOME i.e. /var/jenkins_home, I noticed the init.groovy.d directory, I discovered groovy script named base.groovy that was resetting the Jenkins Location URL to localhost:8080. By modifying this script, I was able to set the correct base URL, and now everything works as expected.

Thank you all for your support!

1 Like

Thank you so much for your feedback, @altif. :+1:
I’m delighted to hear that you found a solution.
Do you have any insight into why this script was written or generated in this particular manner?

@poddingue We have 5 to 6 Jenkins controllers currently running as Docker containers on EC2 instances. These init Groovy scripts are used for the initial setup of our Jenkins controllers, including configuring the admin email, base URL, LDAP setup, user access, and other settings. These scripts are an integral part of our Jenkins controller setup, allowing us to avoid manual changes and updates when provisioning a new Jenkins controller.

Recently, we began migrating all our container-based Jenkins controllers to EKS to leverage Kubernetes capabilities. During the data migration, these scripts were copied over, which caused issues that we didn’t realize until yesterday.

It was a valuable learning experience. :blush:

1 Like

I get it now, thanks for taking the time to explain it. :+1: