Resolved: Can't checkout SCM at top level on K8s

rlratcliffe · August 6, 2023, 2:15pm

Hello! I am updating Jenkins on a Kubernetes based platform. Running into issues that appear to stem from checkout csm, at the top level/core Jenkins pipeline checkout functionality.

I have two kinds of pipelines:

Doesn’t build from code but runs from a library, was previously failing when using checkout functionality INSIDE the steps of the pipeline (which I think was due to this 30600 issue). Switched to using withCredentials and git clone based on some comments I found in that issue and is now fixed
Multibranch & regular pipeline that pulls code at the top level (which I don’t think I have much control over?) and then runs the pipeline in the Jenkinsfile, this is the one that is still not working. I could share the pipeline if that’s helpful, but it looks to me like it’s failing before it even pulls the Jenkinsfile so I think what’s in the Jenkinsfile is irrelevant. Can reproduce pretty simply by just by doing Home → New Item → Pipeline script from SCM → SCM set to Git → Repository URL set to a basic Java repo with no creds → Run pipeline. At first I thought it was the same as the JENKINS-30600 issue, but in this comment it says that checkout csm should still work at the top level. Maybe I’m misunderstanding what top level means.

This is the error I see in the pipeline, but I don’t know if it’s the actual error or just because the initial checkout failed. What does “Not running on the Jenkins controller JVM” mean? Is it relevant?

Is there more info I could provide that would be helpful? Or suggestions of how I can debug further? I’ve tried varying Jenkins versions and plugin versions. I also tried checking the JVM version using the upgrade instructions (won’t let me post another link but: upgrade-java-guidelines/#jvm-version-on-agents) and it looked like the node was also using Java 11.

[Pipeline] {
[Pipeline] stage
[Pipeline] { (Declarative: Checkout SCM)
[Pipeline] checkout
Selected Git installation does not exist. Using Default
The recommended git tool is: NONE
No credentials specified
Cloning the remote Git repository
[Pipeline] }
[Pipeline] // stage
[Pipeline] }
[Pipeline] // node
[Pipeline] }
[Pipeline] // podTemplate
[Pipeline] End of Pipeline
Also:   hudson.remoting.Channel$CallSiteStackTrace: Remote call to JNLP4-connect connection from x.x.1.18/x.x.1.18:37276
		at hudson.remoting.Channel.attachCallSiteStackTrace(Channel.java:1784)
		at hudson.remoting.UserRequest$ExceptionResponse.retrieve(UserRequest.java:356)
		at hudson.remoting.Channel.call(Channel.java:1000)
		at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.execute(RemoteGitImpl.java:153)
		at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
		at java.base/jdk.internal.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
		at java.base/jdk.internal.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
		at java.base/java.lang.reflect.Method.invoke(Method.java:566)
		at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler.invoke(RemoteGitImpl.java:138)
		at com.sun.proxy.$Proxy165.execute(Unknown Source)
		at hudson.plugins.git.GitSCM.retrieveChanges(GitSCM.java:1222)
		at hudson.plugins.git.GitSCM.checkout(GitSCM.java:1305)
		at org.jenkinsci.plugins.workflow.steps.scm.SCMStep.checkout(SCMStep.java:129)
		at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:97)
		at org.jenkinsci.plugins.workflow.steps.scm.SCMStep$StepExecutionImpl.run(SCMStep.java:84)
		at org.jenkinsci.plugins.workflow.steps.SynchronousNonBlockingStepExecution.lambda$start$0(SynchronousNonBlockingStepExecution.java:47)
		at java.base/java.util.concurrent.Executors$RunnableAdapter.call(Executors.java:515)
		at java.base/java.util.concurrent.FutureTask.run(FutureTask.java:264)
		at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(ThreadPoolExecutor.java:1128)
		at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(ThreadPoolExecutor.java:628)
		at java.base/java.lang.Thread.run(Thread.java:829)
Also:   org.jenkinsci.plugins.workflow.actions.ErrorAction$ErrorId: 0e050694-7ff6-4772-890f-cd9b4043889c
java.lang.IllegalStateException: Not running on the Jenkins controller JVM
	at jenkins.util.JenkinsJVM.checkJenkinsJVM(JenkinsJVM.java:46)
	at org.jenkinsci.plugins.credentialsbinding.masking.SecretPatterns.getAggregateSecretPattern(SecretPatterns.java:57)
	at org.csanchez.jenkins.plugins.kubernetes.pipeline.SecretsMasker.lambda$decorate$0(SecretsMasker.java:71)
	at org.jenkinsci.plugins.credentialsbinding.masking.SecretPatterns$MaskingOutputStream.eol(SecretPatterns.java:93)
	at hudson.console.LineTransformationOutputStream.eol(LineTransformationOutputStream.java:61)
	at hudson.console.LineTransformationOutputStream.write(LineTransformationOutputStream.java:57)
	at hudson.console.LineTransformationOutputStream.write(LineTransformationOutputStream.java:75)
	at java.base/java.io.PrintStream.write(Unknown Source)
	at java.base/sun.nio.cs.StreamEncoder.writeBytes(Unknown Source)
	at java.base/sun.nio.cs.StreamEncoder.implFlushBuffer(Unknown Source)
	at java.base/sun.nio.cs.StreamEncoder.flushBuffer(Unknown Source)
	at java.base/java.io.OutputStreamWriter.flushBuffer(Unknown Source)
	at java.base/java.io.PrintStream.newLine(Unknown Source)
	at java.base/java.io.PrintStream.println(Unknown Source)
	at org.jenkinsci.plugins.gitclient.CliGitAPIImpl$2.execute(CliGitAPIImpl.java:807)
	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:170)
	at org.jenkinsci.plugins.gitclient.RemoteGitImpl$CommandInvocationHandler$GitCommandMasterToSlaveCallable.call(RemoteGitImpl.java:161)
	at hudson.remoting.UserRequest.perform(UserRequest.java:211)
	at hudson.remoting.UserRequest.perform(UserRequest.java:54)
	at hudson.remoting.Request$2.run(Request.java:377)
	at hudson.remoting.InterceptingExecutorService.lambda$wrap$0(InterceptingExecutorService.java:78)
	at java.base/java.util.concurrent.FutureTask.run(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor.runWorker(Unknown Source)
	at java.base/java.util.concurrent.ThreadPoolExecutor$Worker.run(Unknown Source)
	at hudson.remoting.Engine$1.lambda$newThread$0(Engine.java:125)
	at java.base/java.lang.Thread.run(Unknown Source)
Finished: FAILURE

Jenkins setup:
Jenkins: 2.416
OS: Linux - 5.15.0-1041-azure
Java: 11.0.19 - Eclipse Adoptium (OpenJDK 64-Bit Server VM)

apache-httpcomponents-client-4-api:4.5.14-150.v7a_b_9d17134a_5
authentication-tokens:1.53.v1c90fd9191a_b_
bootstrap5-api:5.3.0-1
bouncycastle-api:2.29
branch-api:2.1122.v09cb_8ea_8a_724
build-user-vars-plugin:1.7
caffeine-api:3.1.6-115.vb_8b_b_328e59d8
checks-api:2.0.0
cloudbees-folder:6.815.v0dd5a_cb_40e0e
command-launcher:100.v2f6722292ee8
commons-lang3-api:3.12.0-36.vd97de6465d5b_
commons-text-api:1.10.0-68.v0d0b_c439292b_
configuration-as-code:1647.ve39ca_b_829b_42
credentials:1271.v54b_1c2c6388a_
credentials-binding:631.v861c06d062b_4
data-tables-api:1.13.5-1
display-url-api:2.3.8
durable-task:513.vc48a_a_075a_d93
echarts-api:5.4.0-5
font-awesome-api:6.4.0-2
git:5.2.0
git-client:4.4.0
git-server:99.va_0826a_b_cdfa_d
gitea:1.2.1
handy-uri-templates-2-api:2.1.8-22.v77d5b_75e6953
instance-identity:173.va_37c494ec4e5
ionicons-api:56.v1b_1c8c49374e
jackson2-api:2.15.2-350.v0c2f3f8fc595
jakarta-activation-api:2.0.1-3
jakarta-mail-api:2.0.1-3
javax-activation-api:1.2.0-6
javax-mail-api:1.6.2-9
jaxb:2.3.8-1
jdk-tool:66.vd8fa_64ee91b_d
job-dsl:1.78.1
jquery3-api:3.7.0-1
junit:1217.v4297208a_a_b_ce
keycloak:2.3.0
kubernetes:3883.v4d70a_a_a_df034
kubernetes-client-api:6.4.1-215.v2ed17097a_8e9
kubernetes-credentials:0.10.0
kubernetes-credentials-provider:1.208.v128ee9800c04
lockable-resources:1184.va_6f2fc274b_e4
mailer:463.vedf8358e006b_
matrix-project:802.v8013b_40c7edc
metrics:4.2.18-442.v02e107157925
mina-sshd-api-common:2.10.0-69.v28e3e36d18eb_
mina-sshd-api-core:2.10.0-69.v28e3e36d18eb_
okhttp-api:4.11.0-145.vcb_8de402ef81
pipeline-build-step:505.v5f0844d8d126
pipeline-groovy-lib:671.v07c339c842e8
pipeline-input-step:477.v339683a_8d55e
pipeline-milestone-step:111.v449306f708b_7
pipeline-model-api:2.2144.v077a_d1928a_40
pipeline-model-definition:2.2144.v077a_d1928a_40
pipeline-model-extensions:2.2144.v077a_d1928a_40
pipeline-stage-step:305.ve96d0205c1c6
pipeline-stage-tags-metadata:2.2144.v077a_d1928a_40
plain-credentials:143.v1b_df8b_d3b_e48
plugin-util-api:3.3.0
saml:4.429.v9a_781a_61f1da_
scm-api:676.v886669a_199a_a_
script-security:1264.vecf66020eb_7d
snakeyaml-api:1.33-95.va_b_a_e3e47b_fa_4
ssh-credentials:308.ve4497b_ccd8f4
sshd:3.312.v1c601b_c83b_0e
structs:324.va_f5d6774f3a_d
trilead-api:2.84.v72119de229b_7
variant:59.vf075fe829ccb
workflow-aggregator:2.7
workflow-api:1251.vd4889a_b_0a_065
workflow-basic-steps:1042.ve7b_140c4a_e0c
workflow-cps:3731.ve4b_5b_857b_a_d3
workflow-cps-global-lib:609.vd95673f149b_b
workflow-durable-task-step:1278.v94b_dc2b_50c6f
workflow-job:1308.v58d48a_763b_31
workflow-multibranch:756.v891d88f2cd46
workflow-scm-step:415.v434365564324
workflow-step-api:639.v6eca_cd8c04a_a_
workflow-support:848.v5a_383b_d14921

daotoan-hd · August 6, 2023, 2:20pm

Did you wrap the code in container block?

container("container-name") {
 checkout scm
}

rlratcliffe · August 6, 2023, 2:30pm

Hmm. Maybe I’m not being clear, but it’s not the checkout csm in the pipeline itself. It’s this checkout scm in the image, before it even pulls the Jenkinsfile. The code snippet above would be something that goes in the Jenkinsfile, no?

After I save/run the above pipeline, the error above is what occurs.

daotoan-hd · August 6, 2023, 2:36pm

Could you share the Jenkins file?

rlratcliffe · August 6, 2023, 2:55pm

I can reproduce the issue with bare bones Jenkinsfile that’s just:

pipeline {
  agent {
    kubernetes {
      yaml """
apiVersion: v1
kind: Pod
spec:
  containers:
  - name: bash
    image: atlassian/default-image:4.20230726
    command:
    - sleep
    args:
    - infinity
"""
}
  }
  stages {
    stage('Test') {
      steps {
        container('bash') {
          sh """
            echo "I'm running"
          """
        }
      }
    }
  }
}

This is the repo I’m pulling with, but it’s not unique to this specific repo, it’s any pipeline I create that pulls from Git or Gitea.

rlratcliffe · August 6, 2023, 5:12pm

This is resolved. Update in case it helps someone else:

Updating to kubernetes plugin 3971.v94b_4c914ca_75 fixed it. The jenkins operator wasn’t allowing updating to a higher version of the kubernetes plugin before, but it turned out that stopping and starting the jenkins instance allowed the plugin to update properly.

I did the same process when reverting to the old version and it was definitely the plugin update that fixed it.

Topic		Replies	Views
Is there a way to be resilient to git network issues when using scm pipelines? Using Jenkins	1	402	April 7, 2023
Checkout not working in pipeline Jenkins Ask a question question	0	429	October 27, 2023
ERROR: Error cloning remote repo 'origin' Using Jenkins question	1	16234	August 7, 2023
AWS code commit repo url Using Jenkins	1	851	April 18, 2023
How to checkout a pipeline with the same scm object as the caller? Using Jenkins	0	715	June 22, 2022

Resolved: Can't checkout SCM at top level on K8s

Jenkins setup: Jenkins: 2.416 OS: Linux - 5.15.0-1041-azure Java: 11.0.19 - Eclipse Adoptium (OpenJDK 64-Bit Server VM)

Related topics

Jenkins setup:
Jenkins: 2.416
OS: Linux - 5.15.0-1041-azure
Java: 11.0.19 - Eclipse Adoptium (OpenJDK 64-Bit Server VM)