Unable to open Jenkins from browser after fresh deployment on Kubernetes

Using Jenkins Ask a question
1

I have deployed Jenkins on Kubernetes successfully without any error messages in Pod logs. However, the application is not opening from GUI using “node port”.
Please suggest how can i approach on troubleshooting it. It seems some issue Network on my Kubernetes cluster(since same setup running fine on another k8s cluster), but not getting any hint from logs.

Note :

  1. There’s no issue with Node port as other applications can be accessed if using same Node Port.
  2. There was “Readiness probe error with code http 503” initially but it was gone after pod restart. Currently in Ready state.
  3. https://updates.jenkins.io/update-center.json is used in UpdateCenter.xml is accessible from Pod.
  4. Error message in GUI " This page isn’t working right now
    didn’t send any data."

Logs
Logs1528×805 180 KB

Jenkins setup:
Image used - jenkins/jenkins:lts

2

Could you share the definition files of the pod and svc?

3

Thanks for responding on this issue. The manifest files are taken from official site(PFB). I’m able to see the endpoints for Jenkins service.

I had tried installing Jenkins on Docker as well, on same worker node but the issue was same from GUI.
Command : docker run --rm --name jenkins -p 8080:8080 -p 50000:50000 jenkins/jenkins:2.303.1-jdk8

From Kubernetes Nodes :

kubectl get ep
NAME              ENDPOINTS              AGE
jenkins-service   ****:8080   35h

**deployment.yaml**
apiVersion: apps/v1
kind: Deployment
metadata:
  name: jenkins
  namespace: jenkins
spec:
  replicas: 1
  selector:
    matchLabels:
      app: jenkins-server
  template:
    metadata:
      labels:
        app: jenkins-server
    spec:
      securityContext:
            fsGroup: 1000
            runAsUser: 1000
      serviceAccountName: jenkins-admin
      containers:
        - name: jenkins
          image: jenkins/jenkins:lts
          resources:
            limits:
              memory: "2Gi"
              cpu: "1000m"
            requests:
              memory: "500Mi"
              cpu: "500m"
          ports:
            - name: httpport
              containerPort: 8080
            - name: jnlpport
              containerPort: 50000
          livenessProbe:
            httpGet:
              path: "/login"
              port: 8080
            initialDelaySeconds: 90
            periodSeconds: 10
            timeoutSeconds: 5
            failureThreshold: 5
          readinessProbe:
            httpGet:
              path: "/login"
              port: 8080
            initialDelaySeconds: 60
            periodSeconds: 10
            timeoutSeconds: 5
            failureThreshold: 3
          volumeMounts:
            - name: jenkins-data
              mountPath: /var/jenkins_home
      volumes:
        - name: jenkins-data
          persistentVolumeClaim:
              claimName: jenkins-pv-claim


**service.yaml**
apiVersion: v1
kind: Service
metadata:
  name: jenkins-service
  namespace: jenkins
  annotations:
      prometheus.io/scrape: 'true'
      prometheus.io/path:   /
      prometheus.io/port:   '8080'
spec:
  selector:
    app: jenkins-server
  type: NodePort
  ports:
    - port: 8080
      targetPort: 8080
      nodePort: 32123

Tried unlocking via CLI, but getting below error

java -jar /usr/share/jenkins/jenkins-cli.jar -s http://localhost:8080/ -auth admin:b41f9565e6f142ae8092c636045f542b who-am-i
Authenticated as: admin
Authorities:
  authenticated

root@jenkins-bf6b8d5fb-ph2qt:/var/jenkins_home# java -jar /usr/share/jenkins/jenkins-cli.jar -s http://localhost:8080/ groovy = < unlock.groovy

ERROR: anonymous is missing the Overall/Read permission

root@jenkins-bf6b8d5fb-ph2qt:/var/jenkins_home# cat /var/jenkins_home/secrets/initialAdminPassword
b41f9565e6f142ae8092c636045f542b

root@jenkins-bf6b8d5fb-ph2qt:/var/jenkins_home# cat unlock.groovy
import jenkins.model.Jenkins
import hudson.security.*
import jenkins.security.s2m.AdminWhitelistRule

def instance = Jenkins.getInstance()
def hudsonRealm = new HudsonPrivateSecurityRealm(false)
hudsonRealm.createAccount("admin", "b41f9565e6f142ae8092c636045f542b")
instance.setSecurityRealm(hudsonRealm)
def strategy = new FullControlOnceLoggedInAuthorizationStrategy()
instance.setAuthorizationStrategy(strategy)
instance.save()
4

Hi, I have noticed that you have docker installed on the same k8s worker node. Are you using docker as the container runtime? Expect that I suggest that you could do something as follows:

  1. Check if it is there any restart or recreate actions of your Jenkins pod cause there are probe checks in your manifest. Run kubectl describe po <your-jenkins-pod> if there is no restart or recreate.

  2. Login to the worker node and see if you can access jenkins from these commands:

[root@k8s-2 jenkins-install]# k get svc jenkins
NAME      TYPE       CLUSTER-IP      EXTERNAL-IP   PORT(S)                          AGE
jenkins   NodePort   10.101.35.236   <none>        8080:30001/TCP,50000:30779/TCP   170d
[root@k8s-2 jenkins-install]# curl http://10.101.35.236:8080/login





    <!DOCTYPE html><html lang="en"><head resURL="/static/754f2694" data-rooturl="" data-resurl="/static/754f2694" data-imagesurl="/static/754f2694/images"><title>Sign in [Jenkins]</title><meta name="ROBOTS" content="NOFOLLOW"><meta name="viewport" content="width=device-width, initial-scale=1"><link rel="icon" href="/static/754f2694/favicon.svg" type="image/svg+xml"><link sizes="any" rel="alternate icon" href="/static/754f2694/favicon.ico"><link rel="stylesheet" href="/static/754f2694/jsbundles/simple-page.css" type="text/css"></head><body class="app-sign-in-register"><section class="app-sign-in-register__branding"><div class="app-sign-in-register__branding__starburst"></div><img src="/static/754f2694/images/svgs/logo.svg" alt="logo"></section><main id="main-panel" class="app-sign-in-register__content"><div class="app-sign-in-register__content-inner"><h1>Sign in to Jenkins</h1><form method="post" name="login" action="j_spring_security_check"><div><label class="app-sign-in-register__form-label" for="j_username">Username</label><input autocorrect="off" autocomplete="off" name="j_username" id="j_username" type="text" autofocus="autofocus" class="jenkins-input " autocapitalize="off"></div><div><label class="app-sign-in-register__form-label" for="j_password">Password</label><input name="j_password" id="j_password" type="password" class="jenkins-input "></div><div class="jenkins-checkbox"><input type="checkbox" id="remember_me" name="remember_me"><label for="remember_me">Keep me signed in</label></div><input name="from" type="hidden"><button type="submit" name="Submit" class="jenkins-button jenkins-button--primary">Sign in</button></form><div class="footer"></div></div></main></body></html>[root@k8s-2 jenkins-install]#
[root@k8s-2 jenkins-install]#
[root@k8s-2 jenkins-install]# curl http://localhost:30001/login





    <!DOCTYPE html><html lang="en"><head resURL="/static/754f2694" data-rooturl="" data-resurl="/static/754f2694" data-imagesurl="/static/754f2694/images"><title>Sign in [Jenkins]</title><meta name="ROBOTS" content="NOFOLLOW"><meta name="viewport" content="width=device-width, initial-scale=1"><link rel="icon" href="/static/754f2694/favicon.svg" type="image/svg+xml"><link sizes="any" rel="alternate icon" href="/static/754f2694/favicon.ico"><link rel="stylesheet" href="/static/754f2694/jsbundles/simple-page.css" type="text/css"></head><body class="app-sign-in-register"><section class="app-sign-in-register__branding"><div class="app-sign-in-register__branding__starburst"></div><img src="/static/754f2694/images/svgs/logo.svg" alt="logo"></section><main id="main-panel" class="app-sign-in-register__content"><div class="app-sign-in-register__content-inner"><h1>Sign in to Jenkins</h1><form method="post" name="login" action="j_spring_security_check"><div><label class="app-sign-in-register__form-label" for="j_username">Username</label><input autocorrect="off" autocomplete="off" name="j_username" id="j_username" type="text" autofocus="autofocus" class="jenkins-input " autocapitalize="off"></div><div><label class="app-sign-in-register__form-label" for="j_password">Password</label><input name="j_password" id="j_password" type="password" class="jenkins-input "></div><div class="jenkins-checkbox"><input type="checkbox" id="remember_me" name="remember_me"><label for="remember_me">Keep me signed in</label></div><input name="from" type="hidden"><button type="submit" name="Submit" class="jenkins-button jenkins-button--primary">Sign in</button></form><div class="footer"></div></div></main></body></html>[root@k8s-2 jenkins-install]#
  1. Remove all the probe checks and serviceAccountName from the deployment definition file and remove the annotations from the service definition then apply the deployment and service.
5

Yes, using Docker as container engine. Did all the steps as suggested, still same result. Sharing logs below. Thanks for your time checking this issue.

[root@k8s-master1 ~]# kubeadm version
kubeadm version: &version.Info{Major:"1", Minor:"28", GitVersion:"v1.28.7", GitCommit:"c8dcb00be9961ec36d141d2e4103f85f92bcf291", GitTreeState:"clean", BuildDate:"2024-02-14T10:39:01Z", GoVersion:"go1.21.7", Compiler:"gc", Platform:"linux/amd64"}
[root@k8s-master1 ~]# docker version
Client: Docker Engine - Community
 Version:           25.0.3
 API version:       1.44
 Go version:        go1.21.6
 Git commit:        4debf41
 Built:             Tue Feb  6 21:15:16 2024
 OS/Arch:           linux/amd64
 Context:           default

Server: Docker Engine - Community
 Engine:
  Version:          25.0.3
  API version:      1.44 (minimum version 1.24)
  Go version:       go1.21.6
  Git commit:       f417435
  Built:            Tue Feb  6 21:14:12 2024
  OS/Arch:          linux/amd64
  Experimental:     false
 containerd:
  Version:          1.6.28
  GitCommit:        ae07eda36dd25f8a1b98dfbf587313b99c0190bb
 runc:
  Version:          1.1.12
  GitCommit:        v1.1.12-0-g51d5e94
 docker-init:
  Version:          0.19.0
  GitCommit:        de40ad0




[root@k8s-worker3 ~]# curl http://localhost:32123/login


    <!DOCTYPE html><html lang="en"><head resURL="/static/b03994f4" data-rooturl="" data-resurl="/static/b03994f4" data-imagesurl="/static/b03994f4/images"><title>Sign in [Jenkins]</title><meta name="ROBOTS" content="NOFOLLOW"><meta name="viewport" content="width=device-width, initial-scale=1"><link rel="icon" href="/static/b03994f4/favicon.svg" type="image/svg+xml"><link sizes="any" rel="alternate icon" href="/static/b03994f4/favicon.ico"><link rel="stylesheet" href="/static/b03994f4/jsbundles/simple-page.css" type="text/css"></head><body class="app-sign-in-register">

  <!DOCTYPE html><html><head resURL="/static/b03994f4" data-rooturl="" data-resurl="/static/b03994f4" data-extensions-available="true" data-unit-test="false" data-imagesurl="/static/b03994f4/images" data-crumb-header="Jenkins-Crumb" data-crumb-value="2f100804181942373b602ca13aa39febcde2efe6ad080fcd44a4b7658027bed5">



    <title>Jenkins [Jenkins]</title><link rel="stylesheet" href="/static/b03994f4/jsbundles/styles.css" type="text/css"><link rel="stylesheet" href="/static/b03994f4/css/responsive-grid.css" type="text/css"><link rel="icon" href="/static/b03994f4/favicon.svg" type="image/svg+xml"><link sizes="any" rel="alternate icon" href="/static/b03994f4/favicon.ico"><link sizes="180x180" rel="apple-touch-icon" href="/static/b03994f4/apple-touch-icon.png"><link color="#191717" rel="mask-icon" href="/static/b03994f4/mask-icon.svg"><meta name="theme-color" content="#ffffff"><script src="/static/b03994f4/scripts/behavior.js" type="text/javascript"></script><script src='/adjuncts/b03994f4/org/kohsuke/stapler/bind.js' type='text/javascript'></script><script src="/static/b03994f4/scripts/yui/yahoo/yahoo-min.js"></script><script src="/static/b03994f4/scripts/yui/dom/dom-min.js"></script><script src="/static/b03994f4/scripts/yui/event/event-min.js"></script><script src="/static/b03994f4/scripts/yui/animation/animation-min.js"></script><script src="/static/b03994f4/scripts/yui/dragdrop/dragdrop-min.js"></script><script src="/static/b03994f4/scripts/yui/container/container-min.js"></script><script src="/static/b03994f4/scripts/yui/connection/connection-min.js"></script><script src="/static/b03994f4/scripts/yui/datasource/datasource-min.js"></script><script src="/static/b03994f4/scripts/yui/autocomplete/autocomplete-min.js"></script><script src="/static/b03994f4/scripts/yui/menu/menu-min.js"></script><script src="/static/b03994f4/scripts/yui/element/element-min.js"></script><script src="/static/b03994f4/scripts/yui/button/button-min.js"></script><script src="/static/b03994f4/scripts/yui/storage/storage-min.js"></script><script src="/static/b03994f4/scripts/hudson-behavior.js" type="text/javascript"></script><script src="/static/b03994f4/scripts/sortable.js" type="text/javascript"></script><link rel="stylesheet" href="/static/b03994f4/scripts/yui/container/assets/container.css" type="text/css"><link rel="stylesheet" href="/static/b03994f4/scripts/yui/container/assets/skins/sam/container.css" type="text/css"><link rel="stylesheet" href="/static/b03994f4/scripts/yui/menu/assets/skins/sam/menu.css" type="text/css"><meta name="ROBOTS" content="INDEX,NOFOLLOW"><meta name="viewport" content="width=device-width, initial-scale=1"><script src="/static/b03994f4/jsbundles/vendors.js" type="text/javascript"></script><script src="/static/b03994f4/jsbundles/sortable-drag-drop.js" type="text/javascript"></script><script defer="true" src="/static/b03994f4/jsbundles/app.js" type="text/javascript"></script></head><body data-model-type="jenkins.install.SetupWizard" id="jenkins" class="yui-skin-sam full-screen jenkins-2.440.3" data-version="2.440.3"><div id="page-body" class="app-page-body app-page-body--full-screen clear"><div id="main-panel"><a id="skip2content"></a><script src="/static/b03994f4/jsbundles/pluginSetupWizard.js" type="text/javascript"></script><link rel="stylesheet" href="/static/b03994f4/jsbundles/pluginSetupWizard.css" type="text/css"><form method="POST" action="j_spring_security_check"><input name="from" type="hidden"><div class="plugin-setup-wizard bootstrap-3"><div class="modal fade in" style="display: block;"><div class="modal-dialog"><div class="modal-content"><div class="modal-header"><h4 class="modal-title">Getting Started</h4></div><div class="modal-body setup-wizard-heading"><div class="jumbotron welcome-panel offline"><h1>Unlock Jenkins</h1><p>To ensure Jenkins is securely set up by the administrator, a password has been written to the log (<small><a href="https://www.jenkins.io/redirect/find-jenkins-logs" rel="noopener noreferrer" target="_blank">not sure where to find it?</a></small>) and this file on the server: <p><small><code>/var/jenkins_home/secrets/initialAdminPassword</code></small></p><p>Please copy the password from either location and paste it below.<div class="form-group "><label class="control-label" for="security-token">Administrator password</label><input name="j_username" type="hidden" value="admin"><input name="j_password" id="security-token" type="password" class="jenkins-input "></div></div></div><div class="modal-footer"><input type="submit" class="btn btn-primary set-security-key" value="Continue"></div></div></div></div></div></form></div></div></body></html></bod



[root@k8s-worker3 ~]# curl -X POST -d "j_username=admin&j_password=cb74114aaeb542faab63dbd2ccd4e007" http://localhost:32123/j_acegi_security_check
<html>
<head>
<meta http-equiv="Content-Type" content="text/html;charset=ISO-8859-1"/>
<title>Error 403 No valid crumb was included in the request</title>
</head>
<body><h2>HTTP ERROR 403 No valid crumb was included in the request</h2>
<table>
<tr><th>URI:</th><td>/j_acegi_security_check</td></tr>
<tr><th>STATUS:</th><td>403</td></tr>
<tr><th>MESSAGE:</th><td>No valid crumb was included in the request</td></tr>
<tr><th>SERVLET:</th><td>Stapler</td></tr>
</table>
<hr/><a href="https://eclipse.org/jetty">Powered by Jetty:// 10.0.20</a><hr/>

</body>
</html>
6

It looks like you have no issues when accessing Jenkins via Nodeport. The response is fine when you curl localhost: nodeport. What exactly is the error you are facing right now? can not log in?

7

Yes, i keep on getting the same response from browser. Although the same Nodeport if i use for a another webapp(deployed on same worker node), then i’m able to access it from browser.

8

Oh,ok. I see. I forgot to tell you to change the Jenkins image… Could you try to use this one and see if it is ok? docker.io/jenkins/jenkins:2.440.3

Btw, could you see the init page(I mean the page Jenkins asks you to install suggested plugins or unlock jenkins that requires you to provide the init password) from the browser in the first beginning?

9

Thanks. I have tried multiple images, but same error. The initial “unlock Jenkins” page itself doesn’t open.
All these images which i tried, are giving expected result in another Kubernetes cluster. And many other applications deployed on this problematic cluster are behaving fine. This is what puzzling me and I’m out of option now on what else to check.

10

Yeah, that is really wired… Could you do a last try to use the empty dir as your pod’s volume? I am guessing the existing volume has unwanted data.

volumes:
- name: jenkins-data
  emptyDir: {}
11

Same outcome :frowning:
Tried to run lightweight Jenkins image that can start in safe mode without any plugins, but that also didn’t help.

FROM jenkins/jenkins:lts

# Disable the installation of suggested plugins
ENV JAVA_OPTS -Djenkins.install.runSetupWizard=false

# Start Jenkins in safe mode (note: this requires Jenkins 2.222+)
CMD ["java", "-Djenkins.model.Jenkins.safeMode=true", "-jar", "/usr/share/jenkins/jenkins.war"]
12

I have something similar manifest working fine on my local cluster. Just could not figure out why it is not working on your side…
You can actually access Jenkins on local host via curl command but not from remote browser. And you could access other application from remote browser…This is really strange… I am sorry that I can not help.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: jenkins-test
spec:
  replicas: 1
  selector:
    matchLabels:
      app: jenkins-server-test
  template:
    metadata:
      labels:
        app: jenkins-server-test
    spec:
      securityContext:
            fsGroup: 1000
            runAsUser: 1000
      containers:
        - name: jenkins
          image: jenkins/jenkins:lts
          resources:
            limits:
              memory: "2Gi"
              cpu: "1000m"
            requests:
              memory: "500Mi"
              cpu: "500m"
          ports:
            - name: httpport
              containerPort: 8080
            - name: jnlpport
              containerPort: 50000
          volumeMounts:
            - name: jenkins-data
              mountPath: /var/jenkins_home
      volumes:
        - name: jenkins-data
          emptyDir: {}
---
apiVersion: v1
kind: Service
metadata:
  name: jenkins-service-test
spec:
  selector:
    app: jenkins-server-test
  type: NodePort
  ports:
    - port: 8080
      targetPort: 8080