Hey, everybody!
I am using Jenkins installation in kubenrenetes and also use this plugin to automatically create kubernetes agents.
In my cloud kubernetes configuration I have some pod templates like this:
clouds:
- kubernetes:
connectTimeout: 100
containerCap: 10
containerCapStr: "10"
credentialsId: "kubernetes-token"
jenkinsUrl: "https://jenkins.example.com"
name: "k8s-cluster"
namespace: "jenkins-agents"
podLabels:
- key: "app"
value: "jenkins-agent"
readTimeout: 100
serverCertificate: "***"
serverUrl: "***"
templates:
- containers:
- alwaysPullImage: true
args: "-noCertificateCheck -disableHttpsCertValidation"
envVars:
- envVar:
key: "JENKINS_URL"
value: "https://jenkins.example.com"
- envVar:
key: "GIT_SSL_NO_VERIFY"
value: "true"
image: "jenkins/inbound-agent:3273.v4cfe589b_fd83-2"
livenessProbe:
failureThreshold: 0
initialDelaySeconds: 0
periodSeconds: 0
successThreshold: 0
timeoutSeconds: 0
name: "jnlp"
ttyEnabled: true
workingDir: "/home/jenkins/agent"
- command: "/busybox/cat"
envVars:
- envVar:
key: "AWS_SDK_LOAD_CONFIG"
value: "true"
- envVar:
key: "AWS_EC2_METADATA_DISABLED"
value: "true"
- envVar:
key: "CI_SERVER_TLS_CA_FILE"
value: "/example-com-certificates/ca.crt"
image: "gcr.io/kaniko-project/executor:v1.23.0-debug"
name: "kaniko"
ttyEnabled: true
workingDir: "/home/jenkins/agent"
id: "***"
label: "generic"
name: "k8s-cluster-template-generic"
namespace: "jenkins-agents"
nodeUsageMode: "NORMAL"
podRetention: "never"
serviceAccount: "jenkins-agent-generic-irsa"
volumes:
- configMapVolume:
configMapName: "example-com-certificates"
mountPath: "/example-com-certificates"
optional: false
yaml: |-
spec:
topologySpreadConstraints:
- maxSkew: 1
topologyKey: "kubernetes.io/hostname"
whenUnsatisfiable: DoNotSchedule
labelSelector:
matchLabels:
app: jenkins-agent
tolerations:
- key: node-group
operator: Equal
value: jenkins-agents
effect: NoSchedule
affinity:
nodeAffinity:
requiredDuringSchedulingIgnoredDuringExecution:
nodeSelectorTerms:
- matchExpressions:
- key: "node-group"
operator: "In"
values: ["jenkins-agents"]
- key: "karpenter.k8s.aws/instance-memory"
operator: In
values: ["4096"]
yamlMergeStrategy: "merge"
I run agents from the pod templates by inheriting the config from the pod template above
pipeline {
agent {
kubernetes {
inheritFrom 'k8s-cluster-template-generic'
defaultContainer 'kaniko'
}
}
stages {
stage('Test kaniko container') {
steps {
container(name: 'kaniko') {
sh '''
/kaniko/executor --dockerfile Dockerfile --context ...
'''
}
}
}
}
}
And everything works fine, it is created pod by the corresponding cloud template.
But, if I try to override some things via yaml in the pipeline, like this
pipeline {
agent {
kubernetes {
inheritFrom 'k8s-cluster-template-generic'
yaml '''
metadata:
annotations:
karpenter.k8s.aws/instance-family: c6a
karpenter.k8s.aws/instance-size: xlarge
'''
}
}
stages {
stage('Test kaniko container') {
steps {
container(name: 'kaniko') {
sh '''
/kaniko/executor --dockerfile Dockerfile --context ...
'''
}
}
}
}
}
I get a pod without any configuration inheritance from the pod template, it’s a pod with a default jnlp agent (which is automatically integrated by the controller), with my overridden annotations in the pipeline via yaml and nothing else.
No container with kaniko, no custom tolerations/affinity.
Obviously, inheritance just doesn’t work in this case
% kubectl -n jenkins-agents describe pod test-71-chwwr-3mn0v-gng96
Name: test-71-chwwr-3mn0v-gng96
Namespace: jenkins-agents
Priority: 0
Service Account: default
Node: my-node-1
Start Time: Thu, 07 Nov 2024 16:35:49 +0400
Labels: app=jenkins-agent
jenkins/label=test_71-chwwr
jenkins/label-digest=34f4c3fc3a3a06af0b77ea3df5ed0baf6874eebf
kubernetes.jenkins.io/controller=https___jenkins_example_comx
Annotations: buildUrl: https://jenkins.example.com/job/test/71/
karpenter.k8s.aws/instance-family: c6a
karpenter.k8s.aws/instance-size: xlarge
kubernetes.jenkins.io/last-refresh: 1730982949724
runUrl: job/test/71/
Status: Pending
IP:
IPs: <none>
Containers:
jnlp:
Container ID:
Image: jenkins/inbound-agent:3261.v9c670a_4748a_9-2
Image ID:
Port: <none>
Host Port: <none>
State: Waiting
Reason: ContainerCreating
Ready: False
Restart Count: 0
Requests:
cpu: 100m
memory: 256Mi
Environment:
JENKINS_SECRET: ***
JENKINS_AGENT_NAME: test-71-chwwr-3mn0v-gng96
JENKINS_WEB_SOCKET: true
REMOTING_OPTS: -noReconnectAfter 1d
JENKINS_NAME: test-71-chwwr-3mn0v-gng96
JENKINS_AGENT_WORKDIR: /home/jenkins/agent
JENKINS_URL: https://jenkins.example.com/
Mounts:
/home/jenkins/agent from workspace-volume (rw)
Conditions:
Type Status
PodReadyToStartContainers False
Initialized True
Ready False
ContainersReady False
PodScheduled True
Volumes:
workspace-volume:
Type: EmptyDir (a temporary directory that shares a pod's lifetime)
Medium:
SizeLimit: <unset>
QoS Class: Burstable
Node-Selectors: kubernetes.io/os=linux
Tolerations: node.kubernetes.io/not-ready:NoExecute op=Exists for 300s
node.kubernetes.io/unreachable:NoExecute op=Exists for 300s
According to the plugin documentation Kubernetes, I expect that these annotations will simply be added to the pod being created, and everything else will remain consistent with the cloud pod template.
Maybe I’m misunderstanding inherited and somewhere there is an explicit answer to my question or somewhere there is a bug on this topic.
I followed this fix [JENKINS-72112] Allow merge strategy inheritance in PodTemplates by georgblumenschein · Pull Request #1513 · jenkinsci/kubernetes-plugin · GitHub, but unfortunately it didn’t solve my problem (and it looks like this fix is about inheritance between multiple pod templates, not between pod templates and yaml in the pipeline definition).
I would be very grateful for any explanation and recommendations on this problem!
Thanks!
Jenkins setup:
Jenkins: 2.462.2
OS: Linux - 5.10.226-214.880.amzn2.x86_64
Java: 17.0.12 - Eclipse Adoptium (OpenJDK 64-Bit Server VM)
---
ansicolor:1.0.5
antisamy-markup-formatter:162.v0e6ec0fcfcf6
apache-httpcomponents-client-4-api:4.5.14-208.v438351942757
artifact-manager-s3:871.v72f7f642a_245
asm-api:9.7.1-97.v4cc844130d97
authentication-tokens:1.119.v50285141b_7e1
aws-credentials:231.v08a_59f17d742
aws-global-configuration:130.v35b_7b_96f53c3
aws-java-sdk-ec2:1.12.772-474.v7f79a_2046a_fb_
aws-java-sdk-minimal:1.12.772-474.v7f79a_2046a_fb_
basic-branch-build-strategies:81.v05e333931c7d
blueocean:1.27.16
blueocean-bitbucket-pipeline:1.27.16
blueocean-commons:1.27.16
blueocean-config:1.27.16
blueocean-core-js:1.27.16
blueocean-dashboard:1.27.16
blueocean-display-url:2.4.3
blueocean-events:1.27.16
blueocean-git-pipeline:1.27.16
blueocean-github-pipeline:1.27.16
blueocean-i18n:1.27.16
blueocean-jwt:1.27.16
blueocean-personalization:1.27.16
blueocean-pipeline-api-impl:1.27.16
blueocean-pipeline-editor:1.27.16
blueocean-pipeline-scm-api:1.27.16
blueocean-rest:1.27.16
blueocean-rest-impl:1.27.16
blueocean-web:1.27.16
bootstrap5-api:5.3.3-1
bouncycastle-api:2.30.1.78.1-248.ve27176eb_46cb_
branch-api:2.1178.v969d9eb_c728e
build-discarder:139.v05696a_7fe240
caffeine-api:3.1.8-133.v17b_1ff2e0599
checks-api:2.2.1
cloudbees-bitbucket-branch-source:888.v8e6d479a_1730
cloudbees-folder:6.955.v81e2a_35c08d3
commons-compress-api:1.26.1-2
commons-lang3-api:3.17.0-84.vb_b_938040b_078
commons-text-api:1.12.0-129.v99a_50df237f7
configuration-as-code:1850.va_a_8c31d3158b_
credentials:1380.va_435002fa_924
credentials-binding:681.vf91669a_32e45
display-url-api:2.204.vf6fddd8a_8b_e9
durable-task:577.v2a_8a_4b_7c0247
echarts-api:5.5.1-2
email-ext:1844.v3ea_a_b_842374a_
favorite:2.221.v19ca_666b_62f5
font-awesome-api:6.6.0-2
generic-webhook-trigger:2.2.3
git:5.5.2
git-client:5.0.0
git-parameter:0.9.19
github:1.40.0
github-api:1.321-468.v6a_9f5f2d5a_7e
github-branch-source:1797.v86fdb_4d57d43
gson-api:2.11.0-85.v1f4e87273c33
handy-uri-templates-2-api:2.1.8-30.v7e777411b_148
hashicorp-vault-plugin:370.v946b_53544a_30
htmlpublisher:1.36
http_request:1.19
instance-identity:201.vd2a_b_5a_468a_a_6
ionicons-api:74.v93d5eb_813d5f
jackson2-api:2.17.0-379.v02de8ec9f64c
jakarta-activation-api:2.1.3-1
jakarta-mail-api:2.1.3-1
javax-activation-api:1.2.0-7
javax-mail-api:1.6.2-10
jaxb:2.3.9-1
jenkins-design-language:1.27.16
jjwt-api:0.11.5-112.ve82dfb_224b_a_d
job-dsl:1.89
joda-time-api:2.13.0-93.v9934da_29b_a_e9
jquery3-api:3.7.1-2
json-api:20240303-101.v7a_8666713110
json-path-api:2.9.0-118.v7f23ed82a_8b_8
junit:1304.vc85a_b_ca_96613
kubernetes:4292.v11898cf8fa_66
kubernetes-client-api:6.10.0-240.v57880ce8b_0b_2
kubernetes-credentials:190.v03c305394deb_
mailer:488.v0c9639c1a_eb_3
matrix-project:839.vff91cd7e3a_b_2
metrics:4.2.21-451.vd51df8df52ec
mina-sshd-api-common:2.14.0-133.vcc091215a_358
mina-sshd-api-core:2.14.0-133.vcc091215a_358
oidc-provider:79.v46f0066a_d813
okhttp-api:4.11.0-172.vda_da_1feeb_c6e
pipeline-build-step:540.vb_e8849e1a_b_d8
pipeline-graph-analysis:216.vfd8b_ece330ca_
pipeline-groovy-lib:740.va_2701257fe8d
pipeline-input-step:495.ve9c153f6067b_
pipeline-milestone-step:119.vdfdc43fc3b_9a_
pipeline-model-api:2.2214.vb_b_34b_2ea_9b_83
pipeline-model-definition:2.2214.vb_b_34b_2ea_9b_83
pipeline-model-extensions:2.2214.vb_b_34b_2ea_9b_83
pipeline-rest-api:2.34
pipeline-stage-step:312.v8cd10304c27a_
pipeline-stage-tags-metadata:2.2214.vb_b_34b_2ea_9b_83
pipeline-stage-view:2.34
pipeline-utility-steps:2.18.0
plain-credentials:183.va_de8f1dd5a_2b_
plugin-util-api:5.1.0
prism-api:1.29.0-17
pubsub-light:1.18
resource-disposer:0.24
role-strategy:743.v142ea_b_d5f1d3
saml:4.464.vea_cb_75d7f5e0
scm-api:696.v778d637b_a_762
script-security:1365.v4778ca_84b_de5
slack:751.v2e44153c8fe1
snakeyaml-api:2.3-123.v13484c65210a_
sse-gateway:1.27
ssh-credentials:343.v884f71d78167
structs:338.v848422169819
timestamper:1.27
token-macro:400.v35420b_922dcb_
variant:60.v7290fc0eb_b_cd
workflow-api:1336.vee415d95c521
workflow-basic-steps:1058.vcb_fc1e3a_21a_9
workflow-cps:3975.v567e2a_1ffa_22
workflow-durable-task-step:1371.vb_7cec8f3b_95e
workflow-job:1436.vfa_244484591f
workflow-multibranch:795.ve0cb_1f45ca_9a_
workflow-scm-step:427.v4ca_6512e7df1
workflow-step-api:678.v3ee58b_469476
workflow-support:926.v9f4f9b_b_98c19
ws-cleanup:0.47