Jenkins Controller Restart Resets Location URL to localhost:8080, Disconnecting Agent Nodes

altif · June 5, 2024, 5:53pm

Jenkins Controller Version: 2.401.2 LTS

We have a Jenkins controller installed and configured using the official Jenkins Helm chart on an EKS cluster, with all resource provisioning managed by Terraform.

Currently, it is functioning well and is accessible at https://jenkins-utility.example.com/ using our LDAP credentials. All agent nodes connect to same jenkins endpoint.

However, when the Jenkins controller is restarted either via the URL using safe restart or due to the pod being killed/restarted, I can still access Jenkins using the same URL. Despite this, all agent nodes get disconnected because the Jenkins location URL resets to localhost:8080 instead of https://jenkins-utility.example.com/. Please see the attached image for reference.

Below is the Jenkins Configuration as Code (CasC) configuration we are using:

jenkins:
  authorizationStrategy:
    projectMatrix:
      entries:
        - group:
            name: "authenticated"
            permissions:
              - "Agent/Configure"
              - "Agent/Connect"
              - "Agent/Create"
              - "Agent/Delete"
              - "Agent/Disconnect"
              - "Credentials/View"
              - "Job/Build"
              - "Job/Cancel"
              - "Job/Configure"
              - "Job/Create"
              - "Job/Delete"
              - "Job/Discover"
              - "Job/Move"
              - "Job/Read"
              - "Job/Workspace"
              - "Overall/Read"
              - "Overall/SystemRead"
              - "Run/Replay"
              - "View/Configure"
              - "View/Create"
              - "View/Delete"
              - "View/Read"
        - group:
            name: "jenkins-admins"
            permissions:
              - "Overall/Administer"
        - user:
            name: "anonymous"
            permissions:
              - "Job/Discover"
              - "Job/ViewStatus"
              - "Overall/Read"
        - user:
            name: "altif@example.com"
            permissions:
              - "Overall/Administer"
  securityRealm:
    activeDirectory:
      bindPassword: "xxxxxxxx"
      cache:
        size: 1000
        ttl: 3600
      customDomain: true
      domains:
        - bindPassword: "xxxxxxxx"
          name: "cor.example.com"
          servers: "ldap.example.com:3268"
          tlsConfiguration: TRUST_ALL_CERTIFICATES
      groupLookupStrategy: TOKENGROUPS
      removeIrrelevantGroups: false
      requireTLS: false
      startTls: true
  disableRememberMe: false
  mode: NORMAL
  numExecutors: 2
  labelString: "controller"
  projectNamingStrategy: "standard"
  markupFormatter:
    plainText
  clouds:
  - kubernetes:
      containerCapStr: "10"
      defaultsProviderTemplate: ""
      connectTimeout: "5"
      readTimeout: "15"
      jenkinsUrl: "http://jenkins-utility-prod.jenkins-utility-prod.svc.cluster.local:8080"
      jenkinsTunnel: "jenkins-utility-prod-agent.jenkins-utility-prod.svc.cluster.local:50000"
      maxRequestsPerHostStr: "32"
      name: "kubernetes"
      namespace: "jenkins-utility-prod"
      serverUrl: "https://kubernetes.default"
  crumbIssuer:
    standard:
      excludeClientIPFromCrumb: false
security:
  apiToken:
    creationOfLegacyTokenEnabled: false
    tokenGenerationOnCreationEnabled: false
    usageStatisticsEnabled: true
unclassified:
  location:
    adminAddress: devops@example.com
    url: https://jenkins-utility.example.com/

I have spent many hours troubleshooting this issue but have not found a solution yet. Any assistance or references to resolve this issue would be greatly appreciated.

Thanks.

poddingue · June 6, 2024, 12:49pm

Hello and welcome to this community, @altif.

Is your jcasc file located in /usr/share/jenkins/ref/jenkins.yaml?

altif · June 6, 2024, 1:00pm

Thanks @poddingue for the response, appreciate it.

No, it is placed under /var/jenkins_home/casc_configs path.

poddingue · June 6, 2024, 1:24pm

My only experience with Jcasc is through Docker, and I’ve been bitten several times when I forgot to put the Jcasc file in /usr/share/jenkins/ref, but I don’t know if that applies to “standard” Linux installations.

altif · June 6, 2024, 5:58pm

Thanks again

Adding bit more details if this can help.
As shared in previous thread, our Jenkins controller CasC configuration is located at /var/jenkins_home/casc_configs, with the configuration file named jcasc-default-config.yaml, which I have shared in the issue description. When inspecting the pod, the environment variable is correctly set to the specified path:

CASC_JENKINS_CONFIG=/var/jenkins_home/casc_configs

The /var/jenkins_home/ directory is mounted on a persistent volume (EBS gp3).

jenkins@jenkins-utility-prod-0:~/casc_configs$ env | grep -i casc_jenkins
CASC_JENKINS_CONFIG=/var/jenkins_home/casc_configs

jenkins@jenkins-utility-prod-0:~/casc_configs$ pwd
/var/jenkins_home/casc_configs

jenkins@jenkins-utility-prod-0:~/casc_configs$ ls -ltrh
total 4.0K
-rw-rw-r-- 1 jenkins jenkins 2.7K Jun  6 14:24 jcasc-default-config.yaml

jenkins@jenkins-utility-prod-0:~/casc_configs$ df -hT /var/jenkins_home/
jenkins@jenkins-utility-prod-0:~/casc_configs$ df -hT /var/jenkins_home/
Filesystem     Type  Size  Used Avail Use% Mounted on
/dev/nvme2n1   ext4  492G   30G  463G   7% /var/jenkins_home

The issue I am facing is that whenever the Jenkins controller restarts, the Jenkins Location URL isn’t updated according to the CasC file. Surprisingly, if I navigate to https://jenkins-utility.example.com/manage/configuration-as-code/ and click on the Reload existing configuration option under Actions, the CasC configuration reloads correctly and reflects properly. However, upon restarting Jenkins, the URL resets to http://localhost:8080.

I’m not sure how jenkins controller constructing/reading this URL from and resetting it to localhost:8080.

Is there a way to force Jenkins to reload the CasC configuration whenever it restarts? Is there any cache or something that needs to be cleaned up

Any pointers or references would be highly appreciated!

–

mawinter69 · June 6, 2024, 6:23pm

Does the helm chart also define the location in some way?

altif · June 6, 2024, 7:15pm

yes, we have customized jenkins-values.yaml where I’m specifiying the jenkins url. BASE_URL is jenkins endpoint passed in terraform deployment.

## `controller.jenkinsUrl` to change the url definition.
  jenkinsUrl: "${BASE_URL}"

For reference:

github.com

jenkinsci/helm-charts/blob/fb443e5eefbc4cdce044d3a4fc51af97a042249a/charts/jenkins/values.yaml#L168


      
          jenkinsOpts:
          
          # If you are using the ingress definitions provided by this chart via the `controller.ingress` block,
          # the configured hostname will be the ingress hostname starting with `https://`
          # or `http://` depending on the `tls` configuration.
          # The Protocol can be overwritten by specifying `controller.jenkinsUrlProtocol`.
          # -- Set protocol for Jenkins URL; `https` if `controller.ingress.tls`, `http` otherwise
          jenkinsUrlProtocol:
          
          # -- Set Jenkins URL if you are not using the ingress definitions provided by the chart
          jenkinsUrl:
          
          # If you set this prefix and use ingress controller, then you might want to set the ingress path below
          # I.e., "/jenkins"
          # -- Root URI Jenkins will be served on
          jenkinsUriPrefix:
          
          # -- Enable pod security context (must be `true` if podSecurityContextOverride, runAsUser or fsGroup are set)
          usePodSecurityContext: true
          
          # Note that `runAsUser`, `fsGroup`, and `securityContextCapabilities` are

poddingue · June 6, 2024, 7:43pm

I guess it should be qualified as tinkering instead of proper method, but whenever I change anything in the Jcasc file, and after restarting Jenkins, I trigger a Jcasc reload thanks to the REST API.
There must be a clean way of doing that.

altif · June 10, 2024, 3:56am

Thank you @mawinter69 @poddingue for your prompt responses and willingness to assist, much appreciated!

Although, the initial suggestions didn’t resolve my issue, I continued my investigation. After exploring and digging JENKINS_HOME i.e. /var/jenkins_home, I noticed the init.groovy.d directory, I discovered groovy script named base.groovy that was resetting the Jenkins Location URL to localhost:8080. By modifying this script, I was able to set the correct base URL, and now everything works as expected.

Thank you all for your support!

poddingue · June 10, 2024, 6:35am

Thank you so much for your feedback, @altif.
I’m delighted to hear that you found a solution.
Do you have any insight into why this script was written or generated in this particular manner?

altif · June 10, 2024, 4:00pm

@poddingue We have 5 to 6 Jenkins controllers currently running as Docker containers on EC2 instances. These init Groovy scripts are used for the initial setup of our Jenkins controllers, including configuring the admin email, base URL, LDAP setup, user access, and other settings. These scripts are an integral part of our Jenkins controller setup, allowing us to avoid manual changes and updates when provisioning a new Jenkins controller.

Recently, we began migrating all our container-based Jenkins controllers to EKS to leverage Kubernetes capabilities. During the data migration, these scripts were copied over, which caused issues that we didn’t realize until yesterday.

It was a valuable learning experience.

poddingue · June 10, 2024, 4:12pm

I get it now, thanks for taking the time to explain it.

Topic		Replies	Views
Agents connecting to controller are cleared after a Jenkins restart Ask a question question	7	527	June 4, 2024
Jenkins SSO Issue after restart in GKE Using Jenkins	2	20	November 6, 2024
GitHub Authentication reset after jenkins restart Ask a question	2	915	March 15, 2023
Cannot recover job when jenkins agent lose network connectivity Ask a question	1	2259	September 7, 2023
Remote Agent Can't Connect to Jenkins Controller (HELM) Using Jenkins	2	3634	December 28, 2023

Jenkins Controller Restart Resets Location URL to localhost:8080, Disconnecting Agent Nodes

Related topics