# Webhooks issuerCertifiacte: Circular, response: underfined

**URL:** <https://community.jenkins.io/t/webhooks-issuercertifiacte-circular-response-underfined/6033>\
**Category:** Ask a question\
**Tags:** question\
**Created:** [March 1, 2023, 4:36am UTC](https://community.jenkins.io/t/webhooks-issuercertifiacte-circular-response-underfined/6033 "2023-03-01T04:36:51Z")\
**Posts on this page:** 10\
**Page:** 1

<div class="post-metadata">

**Author:** ![Memorist23](https://dub1.discourse-cdn.com/flex013/user_avatar/community.jenkins.io/memorist23/32/3410_2.png) [@Memorist23](https://community.jenkins.io/u/Memorist23)\
**Post date:** [March 1, 2023, 4:36am UTC](https://community.jenkins.io/t/webhooks-issuercertifiacte-circular-response-underfined/6033/1 "2023-03-01T04:36:51Z")

</div>

Hi all

We updated our Jenkins cert to a .pem using a local CA.  
Which is working for us locally with out issues.

However we can’t get webhooks to work correctly.  
When the server starts it give us a

```auto
Mar 01 14:14:14 jenkins jenkins_webhook_startup.sh[20186]: issuerCertificate: [Circular] } },
Mar 01 14:14:14 jenkins jenkins_webhook_startup.sh[20186]: response: undefined

```

We have the .PEM location set in `NODE_EXTRA_CA_CERTS=`  
it does contain the ca, key and crt.

If we set the `NODE_TLS_REJECT_UNAUTHORIZED=0`  
It does run.  
What are we missing?

---

<div class="post-metadata">

**Author:** ![poddingue](https://dub1.discourse-cdn.com/flex013/user_avatar/community.jenkins.io/poddingue/32/985_2.png) [@poddingue](https://community.jenkins.io/u/poddingue)\
**Post date:** [March 1, 2023, 9:15am UTC](https://community.jenkins.io/t/webhooks-issuercertifiacte-circular-response-underfined/6033/2 "2023-03-01T09:15:52Z")

</div>

Hello @Memorist23 ,

What servlet container do you use in [front](https://www.jenkins.io/doc/book/installing/servlet-containers/) of Jenkins, if any?  
Tomcat, WildFly, something else?  
Thanks.

---

<div class="post-metadata">

**Author:** ![halkeye](https://dub1.discourse-cdn.com/flex013/user_avatar/community.jenkins.io/halkeye/32/8_2.png) [@halkeye](https://community.jenkins.io/u/halkeye)\
**Post date:** [March 1, 2023, 7:13pm UTC](https://community.jenkins.io/t/webhooks-issuercertifiacte-circular-response-underfined/6033/3 "2023-03-01T19:13:56Z")

</div>

> [@Memorist23](#):
>
> `jenkins_webhook_startup`

as with last time, knowing more about your custom script would help.

I think last time it involved nodejs, i would recommend adding your custom cert to the system ca certificates instead of telling node to try and use a different list of certificates. Maybe its not having trouble talking to jenkins, but to smee or something.

[https://support.kaspersky.com/ScanEngine/1.0/en-US/182984.htm](https://support.kaspersky.com/ScanEngine/1.0/en-US/182984.htm)

seems like a half decent tutorial

---

<div class="post-metadata">

**Author:** ![Memorist23](https://dub1.discourse-cdn.com/flex013/user_avatar/community.jenkins.io/memorist23/32/3410_2.png) [@Memorist23](https://community.jenkins.io/u/Memorist23)\
**Post date:** [March 1, 2023, 11:42pm UTC](https://community.jenkins.io/t/webhooks-issuercertifiacte-circular-response-underfined/6033/4 "2023-03-01T23:42:10Z")

</div>

Hey all

Sorry I’m new to this one. It was setup before my time. And I’ve never used Jenkins before.  
And there was 0 documentation left on how it was done.  
So happy to provide what ever is needed to get help.  
I just might need a little guidance on that part too. As I think I’m still a bit fresh to Linux.  
Only be using it in anger for the last 1.5 years.

So we have Jenkins, with Webhooks and smee as well. All 3 new to me and this is the only VM we have that uses these apps. All my others are nginx or Apache.

It looks like the update-ca-trust was setup for our CA but updating that doesn’t seem to change anything if i run it now.  
Our CA in there is correct and up to date with the .pem I created and applied. i.e. the on prem cert is now happy.

I seem to have locations for certs and keystores in multiple places and not sure which affect what.  
I have `cd /usr/lib/jenkins/ssl` I feel this location is doing nothing but i could be wrong  
And `cd /etc/haproxy/ssl/` This is where i have been updating and thus far seem to be making a difference in changes.

in my Webhook\_startup.sh

```auto
export NODE_EXTRA_CA_CERTS=/etc/haproxy/jenkins.pem;
smee -u https://smee.io/MY_Identifier --target https://jenkins.mydoamin.com:443/bitbucket-scmsource-hook/notify >> /tmp/jenkins_webhook_log.txt

```

We enabled the keystore flags with no success.  
Here is my jenkins.service file.

```auto
cat /usr/lib/systemd/system/jenkins.service
#
# This file is managed by systemd(1). Do NOT edit this file manually!
# To override these settings, run:
#
# systemctl edit jenkins
#
# For more information about drop-in files, see:
#
# https://www.freedesktop.org/software/systemd/man/systemd.unit.html
#

[Unit]
Description=Jenkins Continuous Integration Server
Requires=network.target
After=network.target

[Service]
Type=notify
NotifyAccess=main
ExecStart=/usr/bin/jenkins
Restart=on-failure
SuccessExitStatus=143

# Configures the time to wait for start-up. If Jenkins does not signal start-up
# completion within the configured time, the service will be considered failed
# and will be shut down again. Takes a unit-less value in seconds, or a time span
# value such as "5min 20s". Pass "infinity" to disable the timeout logic.
#TimeoutStartSec=90

# Unix account that runs the Jenkins daemon
# Be careful when you change this, as you need to update the permissions of
# $JENKINS_HOME, $JENKINS_LOG, and (if you have already run Jenkins)
# $JENKINS_WEBROOT.
User=jenkins
Group=jenkins

# Directory where Jenkins stores its configuration and workspaces
Environment="JENKINS_HOME=/var/lib/jenkins"
WorkingDirectory=/var/lib/jenkins

# Location of the Jenkins WAR
#Environment="JENKINS_WAR=/usr/share/java/jenkins.war"

# Location of the exploded WAR
Environment="JENKINS_WEBROOT=%C/jenkins/war"

# Location of the Jenkins log. By default, systemd-journald(8) is used.
#Environment="JENKINS_LOG=%L/jenkins/jenkins.log"

# The Java home directory. When left empty, JENKINS_JAVA_CMD and PATH are consulted.
#Environment="JAVA_HOME=/usr/lib/jvm/java-11-openjdk-amd64"

# The Java executable. When left empty, JAVA_HOME and PATH are consulted.
#Environment="JENKINS_JAVA_CMD=/etc/alternatives/java"

# Arguments for the Jenkins JVM
Environment="JAVA_OPTS=-Djava.awt.headless=true"
#Environment="JAVA_OPTS=-Djsse.enableSNIExtension=false -Djava.awt.headless=true"

# IP address to listen on for HTTP requests.
# The default is to listen on all interfaces (0.0.0.0).
#Environment="JENKINS_LISTEN_ADDRESS="

# Port to listen on for HTTP requests. Set to -1 to disable.
# To be able to listen on privileged ports (port numbers less than 1024),
# add the CAP_NET_BIND_SERVICE capability to the AmbientCapabilities
# directive below.
Environment="JENKINS_PORT=8080"

# IP address to listen on for HTTPS requests. Default is disabled.
#Environment="JENKINS_HTTPS_LISTEN_ADDRESS="

# Port to listen on for HTTPS requests. Default is disabled.
# To be able to listen on privileged ports (port numbers less than 1024),
# add the CAP_NET_BIND_SERVICE capability to the AmbientCapabilities
# directive below.
#Environment="JENKINS_HTTPS_PORT=443"

# Path to the keystore in JKS format (as created by the JDK's keytool).
# Default is disabled.
#Environment="JENKINS_HTTPS_KEYSTORE=/path/to/keystore.jks"
#Environment="JENKINS_HTTPS_KEYSTORE=/etc/haproxy/ssl/jks/jenkins.jks"

# Password to access the keystore defined in JENKINS_HTTPS_KEYSTORE.
# Default is disabled.
#
# Environment="JENKINS_HTTPS_KEYSTORE_PASSWORD=Old_PasswordHere"
#Environment="JENKINS_HTTPS_KEYSTORE_PASSWORD=mypassword_here"
# IP address to listen on for HTTP2 requests. Default is disabled.
#
# Note: HTTP2 support may require additional configuration.
# See the Winstone documentation for more information.
#Environment="JENKINS_HTTP2_LISTEN_ADDRESS="

# HTTP2 port to listen on. Default is disabled.
# To be able to listen on privileged ports (port numbers less than 1024),
# add the CAP_NET_BIND_SERVICE capability to the AmbientCapabilities
# directive below.
#
# Note: HTTP2 support may require additional configuration.
# See the Winstone documentation for more information.
#Environment="JENKINS_HTTP2_PORT="

# Controls which capabilities to include in the ambient capability set for the
# executed process. Takes a whitespace-separated list of capability names, e.g.
# CAP_SYS_ADMIN, CAP_DAC_OVERRIDE, CAP_SYS_PTRACE. Ambient capability sets are
# useful if you want to execute a process as a non-privileged user but still
# want to give it some capabilities. For example, add the CAP_NET_BIND_SERVICE
# capability to be able to listen on privileged ports (port numbers less than
# 1024).
#AmbientCapabilities=CAP_NET_BIND_SERVICE

# Debug level for logs. The higher the value, the more verbose. 5 is INFO.
#Environment="JENKINS_DEBUG_LEVEL=5"

# Set to true to enable logging to /var/log/jenkins/access_log.
#Environment="JENKINS_ENABLE_ACCESS_LOG=false"

# Folder for additional JAR files to add to the Jetty class loader. Default
# is disabled. See the Winstone documentation for more information.
#Environment="JENKINS_EXTRA_LIB_FOLDER="

# Servlet context (important if you want to use reverse proxying)
#Environment="JENKINS_PREFIX=/jenkins"

# Arbitrary additional arguments to pass to Jenkins.
# Full option list: java -jar jenkins.war --help
#Environment="JENKINS_OPTS="

# Maximum core file size. If unset, the value from the OS is inherited.
#LimitCORE=infinity

# Maximum file size. If unset, the value from the OS is inherited.
#LimitFSIZE=infinity

# File descriptor limit. If unset, the value from the OS is inherited.
#LimitNOFILE=8192

# Maximum number of processes. If unset, the value from the OS is inherited.
#LimitNPROC=32768

# Set the umask to control the permission bits of files that Jenkins creates.
#
# 0027 makes files read-only for group and inaccessible for others, which some
# security sensitive users might consider beneficial, especially if Jenkins
# is running on a server that is used for multiple purposes. Beware that 0027
# permissions would interfere with sudo scripts that run on the controller
# (see JENKINS-25065).
#
# Note also that the particularly sensitive parts of $JENKINS_HOME (such as
# credentials) are always written without 'other' access. So the umask values
# only affect job configuration, build records, etc.
#
# If unset, the value from the OS is inherited, which is normally 0022.
# The default umask comes from pam_umask(8) and /etc/login.defs.
#UMask=0022

```

---

<div class="post-metadata">

**Author:** ![halkeye](https://dub1.discourse-cdn.com/flex013/user_avatar/community.jenkins.io/halkeye/32/8_2.png) [@halkeye](https://community.jenkins.io/u/halkeye)\
**Post date:** [March 2, 2023, 2:08am UTC](https://community.jenkins.io/t/webhooks-issuercertifiacte-circular-response-underfined/6033/5 "2023-03-02T02:08:39Z")

</div>

> [@Memorist23](#):
>
> `export NODE_EXTRA_CA_CERTS=/etc/haproxy/jenkins.pem;`

I am again going to suggest not using this hack (in my opinion).  
Essentially have 2 stores. 1 being the java keystore, the other being the system certificates thing.

Then its a matter of testing

> [@Memorist23](#):
>
> `smee -u https://smee.io/MY_Identifier --target https://jenkins.mydoamin.com:443/bitbucket-scmsource-hook/notify >> /tmp/jenkins_webhook_log.txt`

curl -v [smee.io | Webhook deliveries](https://smee.io/MY_Identifier)  
curl -v [https://jenkins.mydoamin.com:443/bitbucket-scmsource-hook/notify](https://jenkins.mydoamin.com:443/bitbucket-scmsource-hook/notify)

Honestly I’d also recommend seperating your smee and your jenkins systemd. It’ll make things simplier and easier to track down errors

> [@Memorist23](#):
>
> ```auto
> cat /usr/lib/systemd/system/jenkins.service
> #
> # This file is managed by systemd(1). Do NOT edit this file manually!
> # To override these settings, run:
> #
> # systemctl edit jenkins
> 
> ```

as stated, recommend you systemctl edit jenkins, which makes a local override file, and not update the system one which could potentially get overridden by your package manager next upgrade.

---

<div class="post-metadata">

**Author:** ![Memorist23](https://dub1.discourse-cdn.com/flex013/user_avatar/community.jenkins.io/memorist23/32/3410_2.png) [@Memorist23](https://community.jenkins.io/u/Memorist23)\
**Post date:** [March 2, 2023, 4:08am UTC](https://community.jenkins.io/t/webhooks-issuercertifiacte-circular-response-underfined/6033/6 "2023-03-02T04:08:47Z")

</div>

Okay just so i have this clear.  
use the `cat /usr/lib/systemd/system/jenkins.service` as a reference in what to add in the  
`systemctl edit jenkins` as currently I only have :

```auto
[Service]
Environment="JENKINS_LISTEN_ADDRESS=127.0.0.1"

```

ADD the flags for the Keystore and Password to this `systemctl edit jenkins`?

So smee is running in the webhooks systemd with the

```auto
export NODE_EXTRA_CA_CERTS=/etc/haproxy/jenkins.pem;
smee -u https://smee.io/MY_Identifier --target https://jenkins.mydoamin.com:443/bitbucket-scmsource-hook/notify >> /tmp/jenkins_webhook_log.txt

```

You’re saying just `#` out the `NODE_EXTRA_CA_CERTS`?  
Then that is actually 2 systemd jenkins and webhooks(Smee) is this what you meant?

Not sure where the system certificates plays in and how that affects everything.

---

<div class="post-metadata">

**Author:** ![halkeye](https://dub1.discourse-cdn.com/flex013/user_avatar/community.jenkins.io/halkeye/32/8_2.png) [@halkeye](https://community.jenkins.io/u/halkeye)\
**Post date:** [March 2, 2023, 5:23am UTC](https://community.jenkins.io/t/webhooks-issuercertifiacte-circular-response-underfined/6033/7 "2023-03-02T05:23:50Z")

</div>

You are jumping to conclusions about my statements. I’m saying simplify things as much as possible. Instead of trying to get smee and jenkins to work, and the feedback cycle being slow and hard to follow. Split them up, solve one. Based on the previous conversation, I think you have jenkins working. I’m saying isolate smee part.

Also nodejs will read your system’s ca certificates. I recommend using those, its a lot easier to debug (you can use curl), I’ve never used or seen `NODE_EXTRA_CA_CERTS`, so i don’t know how it works or what problems it might have (does it completely override certificates, or augment the system ones, etc).

By installing the cert to the system, which is well documented all over the internet, you can test things quickly and easily, then tell smee to use the system ones (by commenting/removing `NODE_EXTRA_CA_CERTS`)

---

<div class="post-metadata">

**Author:** ![Memorist23](https://dub1.discourse-cdn.com/flex013/user_avatar/community.jenkins.io/memorist23/32/3410_2.png) [@Memorist23](https://community.jenkins.io/u/Memorist23)\
**Post date:** [March 2, 2023, 10:55pm UTC](https://community.jenkins.io/t/webhooks-issuercertifiacte-circular-response-underfined/6033/8 "2023-03-02T22:55:49Z")

</div>

The only reason i was referencing Jenkins as that seem to be the only place where there is reference to nodejs but the flags are all disabled.

Yes jenkins is working as we don’t have a problem accessing it on prem.  
When running those curl command the do return

```auto
[root@jenkins ~]# curl -v smee.io | Webhook deliveries
-bash: Webhook: command not found
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
  0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0* About to connect() to smee.io port 80 (#0)
* Trying 20.119.128.0...
* Connected to smee.io (09.153.127.0) port 80 (#0)
  0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0> GET / HTTP/1.1
> User-Agent: curl/7.29.0
> Host: smee.io
> Accept: */*
> 
< HTTP/1.1 301 Moved Permanently
< Content-Length: 0
< Date: Thu, 02 Mar 2023 22:07:52 GMT
< Location: https://smee.io/
< 
  0 0 0 0 0 0 0 0 --:--:-- --:--:-- --:--:-- 0
* Connection #0 to host smee.io left intact

```

As does the `curl -v https://jenkins.mydoamin.com:443/bitbucket-scmsource-hook/notify`  
too long to post here, Unless I’m looking for something specific here?

It’s only when we run a request to git that we getting the smee error and it’s not completing.  
So it is definitely smee that’s the problem. We are doing this all with a local CA.

---

<div class="post-metadata">

**Author:** ![halkeye](https://dub1.discourse-cdn.com/flex013/user_avatar/community.jenkins.io/halkeye/32/8_2.png) [@halkeye](https://community.jenkins.io/u/halkeye)\
**Post date:** [March 3, 2023, 12:01am UTC](https://community.jenkins.io/t/webhooks-issuercertifiacte-circular-response-underfined/6033/9 "2023-03-03T00:01:50Z")

</div>

> [@Memorist23](#):
>
> `[root@jenkins ~]# curl -v smee.io | Webhook deliveries`

I’m not sure what the pipe there is supposed to be doing, but its erroring

> [@Memorist23](#):
>
> `< Location: https://smee.io/`

You want to curl [https://smee.io](https://smee.io) (or curl -L)

Assuming no https errors, I think just removing that `NODE_EXTRA_CA_CERTS` export should put you back into a good state.

---

<div class="post-metadata">

**Author:** ![Memorist23](https://dub1.discourse-cdn.com/flex013/user_avatar/community.jenkins.io/memorist23/32/3410_2.png) [@Memorist23](https://community.jenkins.io/u/Memorist23)\
**Post date:** [March 3, 2023, 2:55am UTC](https://community.jenkins.io/t/webhooks-issuercertifiacte-circular-response-underfined/6033/10 "2023-03-03T02:55:25Z")

</div>

doing `curl https://smee.io`  
looks to give me a generic smee page

If i do `curl https://smee.io | Webhook deliveries`

```auto
-bash: Webhook: command not found
  % Total % Received % Xferd Average Speed Time Time Time Current
                                 Dload Upload Total Spent Left Speed
100 3305 100 3305 0 0 3385 0 --:--:-- --:--:-- --:--:-- 3382
(23) Failed writing body

```

So if i comment out `NODE_EXTRA_CA_CERTS` I get this when we run something.

```auto
Mar 03 13:36:47 jenkins systemd[1]: Started JENKINS Webhook Service.
Mar 03 13:43:17 jenkins jenkins_webhook_startup.sh[8162]: { Error: self signed certificate in certificate chain
Mar 03 13:43:17 jenkins jenkins_webhook_startup.sh[8162]: at TLSSocket.onConnectSecure (_tls_wrap.js:1088:34)
Mar 03 13:43:17 jenkins jenkins_webhook_startup.sh[8162]: at TLSSocket.emit (events.js:198:13)
Mar 03 13:43:17 jenkins jenkins_webhook_startup.sh[8162]: at TLSSocket._finishInit (_tls_wrap.js:666:8) code: 'SELF_SIGNED_CERT_IN_CHAIN', response: undefined }

```