Webhooks issuerCertifiacte: Circular, response: underfined

Hi all

We updated our Jenkins cert to a .pem using a local CA.
Which is working for us locally with out issues.

However we can’t get webhooks to work correctly.
When the server starts it give us a

Mar 01 14:14:14 jenkins jenkins_webhook_startup.sh[20186]: issuerCertificate: [Circular] } },
Mar 01 14:14:14 jenkins jenkins_webhook_startup.sh[20186]: response: undefined

We have the .PEM location set in NODE_EXTRA_CA_CERTS=
it does contain the ca, key and crt.

If we set the NODE_TLS_REJECT_UNAUTHORIZED=0
It does run.
What are we missing?

Hello @Memorist23 ,

What servlet container do you use in front of Jenkins, if any?
Tomcat, WildFly, something else?
Thanks.

as with last time, knowing more about your custom script would help.

I think last time it involved nodejs, i would recommend adding your custom cert to the system ca certificates instead of telling node to try and use a different list of certificates. Maybe its not having trouble talking to jenkins, but to smee or something.

https://support.kaspersky.com/ScanEngine/1.0/en-US/182984.htm

seems like a half decent tutorial

1 Like

Hey all

Sorry I’m new to this one. It was setup before my time. And I’ve never used Jenkins before.
And there was 0 documentation left on how it was done.
So happy to provide what ever is needed to get help.
I just might need a little guidance on that part too. As I think I’m still a bit fresh to Linux.
Only be using it in anger for the last 1.5 years.

So we have Jenkins, with Webhooks and smee as well. All 3 new to me and this is the only VM we have that uses these apps. All my others are nginx or Apache.

It looks like the update-ca-trust was setup for our CA but updating that doesn’t seem to change anything if i run it now.
Our CA in there is correct and up to date with the .pem I created and applied. i.e. the on prem cert is now happy.

I seem to have locations for certs and keystores in multiple places and not sure which affect what.
I have cd /usr/lib/jenkins/ssl I feel this location is doing nothing but i could be wrong
And cd /etc/haproxy/ssl/ This is where i have been updating and thus far seem to be making a difference in changes.

in my Webhook_startup.sh

export NODE_EXTRA_CA_CERTS=/etc/haproxy/jenkins.pem;
smee -u https://smee.io/MY_Identifier --target https://jenkins.mydoamin.com:443/bitbucket-scmsource-hook/notify >> /tmp/jenkins_webhook_log.txt

We enabled the keystore flags with no success.
Here is my jenkins.service file.

cat /usr/lib/systemd/system/jenkins.service
#
# This file is managed by systemd(1). Do NOT edit this file manually!
# To override these settings, run:
#
#     systemctl edit jenkins
#
# For more information about drop-in files, see:
#
#     https://www.freedesktop.org/software/systemd/man/systemd.unit.html
#

[Unit]
Description=Jenkins Continuous Integration Server
Requires=network.target
After=network.target

[Service]
Type=notify
NotifyAccess=main
ExecStart=/usr/bin/jenkins
Restart=on-failure
SuccessExitStatus=143

# Configures the time to wait for start-up. If Jenkins does not signal start-up
# completion within the configured time, the service will be considered failed
# and will be shut down again. Takes a unit-less value in seconds, or a time span
# value such as "5min 20s". Pass "infinity" to disable the timeout logic.
#TimeoutStartSec=90

# Unix account that runs the Jenkins daemon
# Be careful when you change this, as you need to update the permissions of
# $JENKINS_HOME, $JENKINS_LOG, and (if you have already run Jenkins)
# $JENKINS_WEBROOT.
User=jenkins
Group=jenkins

# Directory where Jenkins stores its configuration and workspaces
Environment="JENKINS_HOME=/var/lib/jenkins"
WorkingDirectory=/var/lib/jenkins

# Location of the Jenkins WAR
#Environment="JENKINS_WAR=/usr/share/java/jenkins.war"

# Location of the exploded WAR
Environment="JENKINS_WEBROOT=%C/jenkins/war"

# Location of the Jenkins log. By default, systemd-journald(8) is used.
#Environment="JENKINS_LOG=%L/jenkins/jenkins.log"

# The Java home directory. When left empty, JENKINS_JAVA_CMD and PATH are consulted.
#Environment="JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64"

# The Java executable. When left empty, JAVA_HOME and PATH are consulted.
#Environment="JENKINS_JAVA_CMD=/etc/alternatives/java"

# Arguments for the Jenkins JVM
Environment="JAVA_OPTS=-Djava.awt.headless=true"
#Environment="JAVA_OPTS=-Djsse.enableSNIExtension=false -Djava.awt.headless=true"

# IP address to listen on for HTTP requests.
# The default is to listen on all interfaces (0.0.0.0).
#Environment="JENKINS_LISTEN_ADDRESS="

# Port to listen on for HTTP requests. Set to -1 to disable.
# To be able to listen on privileged ports (port numbers less than 1024),
# add the CAP_NET_BIND_SERVICE capability to the AmbientCapabilities
# directive below.
Environment="JENKINS_PORT=8080"

# IP address to listen on for HTTPS requests. Default is disabled.
#Environment="JENKINS_HTTPS_LISTEN_ADDRESS="

# Port to listen on for HTTPS requests. Default is disabled.
# To be able to listen on privileged ports (port numbers less than 1024),
# add the CAP_NET_BIND_SERVICE capability to the AmbientCapabilities
# directive below.
#Environment="JENKINS_HTTPS_PORT=443"

# Path to the keystore in JKS format (as created by the JDK's keytool).
# Default is disabled.
#Environment="JENKINS_HTTPS_KEYSTORE=/path/to/keystore.jks"
#Environment="JENKINS_HTTPS_KEYSTORE=/etc/haproxy/ssl/jks/jenkins.jks"

# Password to access the keystore defined in JENKINS_HTTPS_KEYSTORE.
# Default is disabled.
#
# Environment="JENKINS_HTTPS_KEYSTORE_PASSWORD=Old_PasswordHere"
#Environment="JENKINS_HTTPS_KEYSTORE_PASSWORD=mypassword_here"
# IP address to listen on for HTTP2 requests. Default is disabled.
#
# Note: HTTP2 support may require additional configuration.
# See the Winstone documentation for more information.
#Environment="JENKINS_HTTP2_LISTEN_ADDRESS="

# HTTP2 port to listen on. Default is disabled.
# To be able to listen on privileged ports (port numbers less than 1024),
# add the CAP_NET_BIND_SERVICE capability to the AmbientCapabilities
# directive below.
#
# Note: HTTP2 support may require additional configuration.
# See the Winstone documentation for more information.
#Environment="JENKINS_HTTP2_PORT="

# Controls which capabilities to include in the ambient capability set for the
# executed process. Takes a whitespace-separated list of capability names, e.g.
# CAP_SYS_ADMIN, CAP_DAC_OVERRIDE, CAP_SYS_PTRACE. Ambient capability sets are
# useful if you want to execute a process as a non-privileged user but still
# want to give it some capabilities. For example, add the CAP_NET_BIND_SERVICE
# capability to be able to listen on privileged ports (port numbers less than
# 1024).
#AmbientCapabilities=CAP_NET_BIND_SERVICE

# Debug level for logs. The higher the value, the more verbose. 5 is INFO.
#Environment="JENKINS_DEBUG_LEVEL=5"

# Set to true to enable logging to /var/log/jenkins/access_log.
#Environment="JENKINS_ENABLE_ACCESS_LOG=false"

# Folder for additional JAR files to add to the Jetty class loader. Default
# is disabled. See the Winstone documentation for more information.
#Environment="JENKINS_EXTRA_LIB_FOLDER="

# Servlet context (important if you want to use reverse proxying)
#Environment="JENKINS_PREFIX=/jenkins"

# Arbitrary additional arguments to pass to Jenkins.
# Full option list: java -jar jenkins.war --help
#Environment="JENKINS_OPTS="

# Maximum core file size. If unset, the value from the OS is inherited.
#LimitCORE=infinity

# Maximum file size. If unset, the value from the OS is inherited.
#LimitFSIZE=infinity

# File descriptor limit. If unset, the value from the OS is inherited.
#LimitNOFILE=8192

# Maximum number of processes. If unset, the value from the OS is inherited.
#LimitNPROC=32768

# Set the umask to control the permission bits of files that Jenkins creates.
#
# 0027 makes files read-only for group and inaccessible for others, which some
# security sensitive users might consider beneficial, especially if Jenkins
# is running on a server that is used for multiple purposes. Beware that 0027
# permissions would interfere with sudo scripts that run on the controller
# (see JENKINS-25065).
#
# Note also that the particularly sensitive parts of $JENKINS_HOME (such as
# credentials) are always written without 'other' access. So the umask values
# only affect job configuration, build records, etc.
#
# If unset, the value from the OS is inherited, which is normally 0022.
# The default umask comes from pam_umask(8) and /etc/login.defs.
#UMask=0022

I am again going to suggest not using this hack (in my opinion).
Essentially have 2 stores. 1 being the java keystore, the other being the system certificates thing.

Then its a matter of testing

curl -v smee.io | Webhook deliveries
curl -v https://jenkins.mydoamin.com:443/bitbucket-scmsource-hook/notify

Honestly I’d also recommend seperating your smee and your jenkins systemd. It’ll make things simplier and easier to track down errors

as stated, recommend you systemctl edit jenkins, which makes a local override file, and not update the system one which could potentially get overridden by your package manager next upgrade.

Okay just so i have this clear.
use the cat /usr/lib/systemd/system/jenkins.service as a reference in what to add in the
systemctl edit jenkins as currently I only have :

[Service]
Environment="JENKINS_LISTEN_ADDRESS=127.0.0.1"

ADD the flags for the Keystore and Password to this systemctl edit jenkins?

So smee is running in the webhooks systemd with the

export NODE_EXTRA_CA_CERTS=/etc/haproxy/jenkins.pem;
smee -u https://smee.io/MY_Identifier --target https://jenkins.mydoamin.com:443/bitbucket-scmsource-hook/notify >> /tmp/jenkins_webhook_log.txt

You’re saying just # out the NODE_EXTRA_CA_CERTS?
Then that is actually 2 systemd jenkins and webhooks(Smee) is this what you meant?

Not sure where the system certificates plays in and how that affects everything.

You are jumping to conclusions about my statements. I’m saying simplify things as much as possible. Instead of trying to get smee and jenkins to work, and the feedback cycle being slow and hard to follow. Split them up, solve one. Based on the previous conversation, I think you have jenkins working. I’m saying isolate smee part.

Also nodejs will read your system’s ca certificates. I recommend using those, its a lot easier to debug (you can use curl), I’ve never used or seen NODE_EXTRA_CA_CERTS, so i don’t know how it works or what problems it might have (does it completely override certificates, or augment the system ones, etc).

By installing the cert to the system, which is well documented all over the internet, you can test things quickly and easily, then tell smee to use the system ones (by commenting/removing NODE_EXTRA_CA_CERTS)

The only reason i was referencing Jenkins as that seem to be the only place where there is reference to nodejs but the flags are all disabled.

Yes jenkins is working as we don’t have a problem accessing it on prem.
When running those curl command the do return

[root@jenkins ~]# curl -v smee.io | Webhook deliveries
-bash: Webhook: command not found
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0* About to connect() to smee.io port 80 (#0)
*   Trying 20.119.128.0...
* Connected to smee.io (09.153.127.0) port 80 (#0)
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: smee.io
> Accept: */*
> 
< HTTP/1.1 301 Moved Permanently
< Content-Length: 0
< Date: Thu, 02 Mar 2023 22:07:52 GMT
< Location: https://smee.io/
< 
  0     0    0     0    0     0      0      0 --:--:-- --:--:-- --:--:--     0
* Connection #0 to host smee.io left intact

As does the curl -v https://jenkins.mydoamin.com:443/bitbucket-scmsource-hook/notify
too long to post here, Unless I’m looking for something specific here?

It’s only when we run a request to git that we getting the smee error and it’s not completing.
So it is definitely smee that’s the problem. We are doing this all with a local CA.

I’m not sure what the pipe there is supposed to be doing, but its erroring

You want to curl https://smee.io (or curl -L)

Assuming no https errors, I think just removing that NODE_EXTRA_CA_CERTS export should put you back into a good state.

doing curl https://smee.io
looks to give me a generic smee page

If i do curl https://smee.io | Webhook deliveries

-bash: Webhook: command not found
  % Total    % Received % Xferd  Average Speed   Time    Time     Time  Current
                                 Dload  Upload   Total   Spent    Left  Speed
100  3305  100  3305    0     0   3385      0 --:--:-- --:--:-- --:--:--  3382
(23) Failed writing body

So if i comment out NODE_EXTRA_CA_CERTS I get this when we run something.

Mar 03 13:36:47 jenkins systemd[1]: Started JENKINS Webhook Service.
Mar 03 13:43:17 jenkins jenkins_webhook_startup.sh[8162]: { Error: self signed certificate in certificate chain
Mar 03 13:43:17 jenkins jenkins_webhook_startup.sh[8162]: at TLSSocket.onConnectSecure (_tls_wrap.js:1088:34)
Mar 03 13:43:17 jenkins jenkins_webhook_startup.sh[8162]: at TLSSocket.emit (events.js:198:13)
Mar 03 13:43:17 jenkins jenkins_webhook_startup.sh[8162]: at TLSSocket._finishInit (_tls_wrap.js:666:8) code: 'SELF_SIGNED_CERT_IN_CHAIN', response: undefined }