SSL Cert update for AD login's

Using Jenkins Ask a question
1

Hi All
I’m pretty new to Jenkins and the system I have was inherited.
Also first time using a application that uses Java primarily to serve.
Our certificate for Jenkins at work has expired and I can’t for the life of me find the existing certs and how to update them.
We are on a domain so user auth is Active Directory.
Our Domain CA has been added in the cacerts.
"/etc/pki/ca-trust/extracted/openssl/ca-bundle.trust.crt"

Other than that i can’t find anything.
All entries for HTTPS in the
/etc/sysconfig/jenkins
are # out

# JENKINS_HTTPS_KEYSTORE="/var/lib/jenkins/keystore/jenkins.p12"
# JENKINS_HTTPS_KEYSTORE="/var/lib/jenkins/keystore/jenkins.jks"
# JENKINS_HTTPS_KEYSTORE_PASSWORD=""
# JENKINS_HTTPS_KEYSTORE_PASSWORD=""
# JENKINS_HTTPS_KEYSTORE_PASSWORD="

Would someone be able to help me find figure this out?

2

I’ve never had to mess with certs in java but I think you need to setup a java Keystore

Which is kinda what Active Directory says with no details

3

so i found this .sh file

#!/bin/bash
openssl pkcs12 -export -out jenkins.p12 -passout 'password' -inkey ./jenkins.key -in ./jenkins.cer -certfile ./ca.crt  -name jenkins.MyDomain.com
keytool -importkeystore -srckeystore jenkins.p12 -srcstorepass 'password' -srcstoretype PKCS12 -srcalias jenkins.MyDomain.com -deststoretype JKS -destkeystore jenkins.jks -deststorepass 'password' -destalias jenkins.MyDomain.com

I queried the keystore.jks and it had the correct cert
I ran the .sh file
This has now broken Jenkins and will not start.
Even restoring from an old snapshot does not resolve this.

Not sure what i’ve done.

4

What do your logs say? Why can’t it start up?

5

It says it’s running but the page is not accessible.
503 Service Unaccessable

Looks like logging is off or it’s not going to the default location as there is nothing after 2022-03-14

[root@jenkins ~]# systemctl status jenkins_hook
\u25cf jenkins_hook.service - JENKINS Webhook Service
   Loaded: loaded (/etc/systemd/system/jenkins_hook.service; disabled; vendor preset: disabled)
   Active: active (running) since Tue 2023-02-21 14:58:18 AEDT; 1h 25min ago
 Main PID: 1310 (jenkins_webhook)
   CGroup: /system.slice/jenkins_hook.service
           \u251c\u25001310 /bin/bash /var/jenkins/jenkins_webhook_startup.sh
           \u2514\u25001312 node /usr/bin/smee -u https://smee.io/NluOBiOH4qNrAIw --target https://jenkins.MyDomain.com:443/bitbucket-scmsource-hook/notify

Feb 21 14:58:18 jenkins systemd[1]: Started JENKINS Webhook Service.
Feb 21 16:16:42 jenkins jenkins_webhook_startup.sh[1310]: { Error: certificate has expired
Feb 21 16:16:42 jenkins jenkins_webhook_startup.sh[1310]: at TLSSocket.onConnectSecure (_tls_wrap.js:1088:34)
Feb 21 16:16:42 jenkins jenkins_webhook_startup.sh[1310]: at TLSSocket.emit (events.js:198:13)
Feb 21 16:16:42 jenkins jenkins_webhook_startup.sh[1310]: at TLSSocket._finishInit (_tls_wrap.js:666:8) code: 'CERT_HAS_EXPIRED', response: undefined }
6

I recommend using journalctl to look at the full log

I think from your logs its actually smee having trouble talking to something. Says the cert is expired. Maybe check it’s config too

Edit I see your edit. Jenkins moved to systemd a while ago. That’s probably why there are no logs in var looks

7

It seems like it’s running.
can’t see anything in the journalctl
Only the cert issue if flagging

Feb 21 15:08:42 jenkins systemd[1]: Started Cleanup of Temporary Directories.
Feb 21 15:08:49 jenkins ntpd[569]: 0.0.0.0 c612 02 freq_set kernel 7.351 PPM
Feb 21 15:08:49 jenkins ntpd[569]: 0.0.0.0 c615 05 clock_sync
Feb 21 15:23:42 jenkins sssd_be[585]: GSSAPI client step 1
Feb 21 15:23:42 jenkins sssd_be[585]: GSSAPI client step 1
Feb 21 15:23:42 jenkins sssd_be[585]: GSSAPI client step 1
Feb 21 15:23:42 jenkins sssd_be[585]: GSSAPI client step 2
Feb 21 15:37:01 jenkins anacron[1336]: Job `cron.daily' started
Feb 21 15:37:01 jenkins run-parts(/etc/cron.daily)[1356]: starting logrotate
Feb 21 15:37:01 jenkins rsyslogd[817]:  [origin software="rsyslogd" swVersion="8.24.0-57.el7_9.1" x-pid="817" x-info="http://www.rsyslog.com"] rsyslogd was HUPed
Feb 21 15:37:01 jenkins run-parts(/etc/cron.daily)[1370]: finished logrotate
Feb 21 15:37:01 jenkins run-parts(/etc/cron.daily)[1372]: starting man-db.cron
Feb 21 15:37:01 jenkins run-parts(/etc/cron.daily)[1381]: finished man-db.cron
Feb 21 15:37:01 jenkins anacron[1336]: Job `cron.daily' terminated
Feb 21 15:38:42 jenkins sssd_be[585]: GSSAPI client step 1
Feb 21 15:38:42 jenkins sssd_be[585]: GSSAPI client step 1
Feb 21 15:38:42 jenkins sssd_be[585]: GSSAPI client step 1
Feb 21 15:38:42 jenkins sssd_be[585]: GSSAPI client step 2
Feb 21 15:53:42 jenkins sssd_be[585]: GSSAPI client step 1
Feb 21 15:53:42 jenkins sssd_be[585]: GSSAPI client step 1
Feb 21 15:53:42 jenkins sssd_be[585]: GSSAPI client step 1
Feb 21 15:53:42 jenkins sssd_be[585]: GSSAPI client step 2
Feb 21 15:57:01 jenkins anacron[1336]: Job `cron.weekly' started
Feb 21 15:57:01 jenkins anacron[1336]: Job `cron.weekly' terminated
Feb 21 15:57:01 jenkins anacron[1336]: Normal exit (2 jobs run)
Feb 21 16:01:01 jenkins systemd[1]: Started Session 3 of user root.
Feb 21 16:01:01 jenkins CROND[1395]: (root) CMD (run-parts /etc/cron.hourly)
Feb 21 16:01:01 jenkins run-parts(/etc/cron.hourly)[1398]: starting 0anacron
Feb 21 16:01:01 jenkins run-parts(/etc/cron.hourly)[1404]: finished 0anacron
Feb 21 16:08:42 jenkins sssd_be[585]: GSSAPI client step 1
Feb 21 16:08:42 jenkins sssd_be[585]: GSSAPI client step 1
Feb 21 16:08:42 jenkins sssd_be[585]: GSSAPI client step 1
Feb 21 16:08:42 jenkins sssd_be[585]: GSSAPI client step 2
Feb 21 16:16:42 jenkins jenkins_webhook_startup.sh[1310]: { Error: certificate has expired
Feb 21 16:16:42 jenkins jenkins_webhook_startup.sh[1310]: at TLSSocket.onConnectSecure (_tls_wrap.js:1088:34)
Feb 21 16:16:42 jenkins jenkins_webhook_startup.sh[1310]: at TLSSocket.emit (events.js:198:13)
Feb 21 16:16:42 jenkins jenkins_webhook_startup.sh[1310]: at TLSSocket._finishInit (_tls_wrap.js:666:8) code: 'CERT_HAS_EXPIRED', response: undefined }
Feb 21 16:22:14 jenkins jenkins_webhook_startup.sh[1310]: { Error: certificate has expired
Feb 21 16:22:14 jenkins jenkins_webhook_startup.sh[1310]: at TLSSocket.onConnectSecure (_tls_wrap.js:1088:34)
Feb 21 16:22:14 jenkins jenkins_webhook_startup.sh[1310]: at TLSSocket.emit (events.js:198:13)
Feb 21 16:22:14 jenkins jenkins_webhook_startup.sh[1310]: at TLSSocket._finishInit (_tls_wrap.js:666:8) code: 'CERT_HAS_EXPIRED', response: undefined }
Feb 21 16:23:42 jenkins sssd_be[585]: GSSAPI client step 1
Feb 21 16:23:42 jenkins sssd_be[585]: GSSAPI client step 1
Feb 21 16:23:42 jenkins sssd_be[585]: GSSAPI client step 1
Feb 21 16:23:42 jenkins sssd_be[585]: GSSAPI client step 2
Feb 21 16:33:13 jenkins dbus[568]: [system] Activating via systemd: service name='org.freedesktop.timedate1' unit='dbus-org.freedesktop.timedate1.service'
Feb 21 16:33:13 jenkins systemd[1]: Starting Time & Date Service...
Feb 21 16:33:13 jenkins dbus[568]: [system] Successfully activated service 'org.freedesktop.timedate1'
Feb 21 16:33:13 jenkins systemd[1]: Started Time & Date Service.
8

smee looks pretty straight forward.

[root@jenkins ~]# cat /usr/bin/smee 
#!/usr/bin/env node

const program = require('commander')
const { version } = require('../package.json')

const Client = require('..')

program
  .version(version, '-v, --version')
  .usage('[options]')
  .option('-u, --url <url>', 'URL of the webhook proxy service. Default: https://smee.io/new')
  .option('-t, --target <target>', 'Full URL (including protocol and path) of the target service the events will forwarded to. Default: http://127.0.0.1:PORT/PATH')
  .option('-p, --port <n>', 'Local HTTP server port', process.env.PORT || 3000)
  .option('-P, --path <path>', 'URL path to post proxied requests to`', '/')
  .parse(process.argv)

let target
if (program.target) {
  target = program.target
} else {
  target = `http://127.0.0.1:${program.port}${program.path}`
}

async function setup () {
  let source = program.url

  if (!source) {
    source = await Client.createChannel()
  }

  const client = new Client({ source, target })
  client.start()
}

setup()
9

That looks like to me that its a javascript console.log error message. But i would take a look at /var/jenkins/jenkins_webhook_startup.sh and see what its doing.

you could check date and see if clock if broken, or if it is smee, curl -v smee.io | Webhook deliveries to see if any errors show up. Smee also could be trying to contact https://localhost/ or something, and that cert is expired.

10

okay updated the certificate.
Still getting the 503 Service Unavailable

Assuming this smee that i took out of the /var/jenkins/jenkins_webhook_startup.sh
is working if it connects as below?

[root@jenkins ~]# smee -u https://smee.io/abc123
Forwarding https://smee.io/abc123 to http://127.0.0.1:3000/
Connected https://smee.io/abc123

no more cert error

Feb 21 20:33:40 jenkins systemd[1]: Stopped JENKINS Webhook Service.
Feb 21 20:33:40 jenkins systemd[1]: Started JENKINS Webhook Service.
Feb 21 20:33:40 jenkins polkitd[576]: Unregistered Authentication Agent for unix-process:1346:132726 (system bus name :1.22, object path /org/freedesktop/PolicyKit1/AuthenticationAgent, locale en_AU.UTF-8) (disc
Feb 21 20:41:45 jenkins sssd_be[598]: GSSAPI client step 1
Feb 21 20:41:45 jenkins sssd_be[598]: GSSAPI client step 1
Feb 21 20:41:45 jenkins sssd_be[598]: GSSAPI client step 1
Feb 21 20:41:45 jenkins sssd_be[598]: GSSAPI client step 2
11

seems to be running now. If you curl localhost does it work?

Can this be something external? like a load balancer or something?

12

all sorted now, needed to start the jenkins.service as well.
Everything is loading.
thanks for all the help.