Hi,
This is a question related to the retry() and eviction of pod of the Jenkins Kubernetes plugin.
I have pipeline that are using
options {
...
retry(conditions: [kubernetesAgent(), nonresumable()], count: 3)
...
}
I confirm that a pipeline is restarted when we manually evict a pod from a node.
But today I faced a different kind of eviction due to node pressure. Naïvely I thought the pod would be recreated on an other node, but no retry was performed.
This is the log I got
We clearly see
Pod just failed. Reason: Evicted, Message: Pod was rejected: The node had condition: [DiskPressure]. .
[13:21:51.022+01:00] - Current branch: renovate/workflows
[13:21:52.094+01:00] - [Pipeline] podTemplate
[13:21:52.097+01:00] - [Pipeline] {
[13:21:52.112+01:00] - [Pipeline] node
[13:21:53.281+01:00] - Created Pod: openshift *******************************-2fworkflows-1-hzpd-j2hqg
[13:21:53.362+01:00] - *******************************-2fworkflows-1-hzpd-j2hqg Pod just failed. Reason: Evicted, Message: Pod was rejected: The node had condition: [DiskPressure]. .
[13:21:53.367+01:00] - [PodInfo] *******************************-2fworkflows-1-hzpd-j2hqg
[13:21:53.367+01:00] - Container [kroki] waiting [PodInitializing] No message
[13:21:53.367+01:00] - Container [kroki-excalidraw] waiting [PodInitializing] No message
[13:21:53.367+01:00] - Container [kroki-mermaid] waiting [PodInitializing] No message
[13:21:53.367+01:00] - Container [mc] waiting [PodInitializing] No message
[13:21:53.367+01:00] - Container [techdoc] waiting [PodInitializing] No message
[13:21:53.367+01:00] - Pod [Failed][ContainersNotInitialized] containers with incomplete status: [set-up-jenkins-agent]
[13:21:53.367+01:00] - Pod [Failed][PodFailed] No message
[13:21:53.369+01:00] - [PodInfo] *******************************-2fworkflows-1-hzpd-j2hqg
[13:21:53.369+01:00] - Container [kroki] waiting [PodInitializing] No message
[13:21:53.369+01:00] - Container [kroki-excalidraw] waiting [PodInitializing] No message
[13:21:53.369+01:00] - Container [kroki-mermaid] waiting [PodInitializing] No message
[13:21:53.369+01:00] - Container [mc] waiting [PodInitializing] No message
[13:21:53.369+01:00] - Container [techdoc] waiting [PodInitializing] No message
[13:21:53.369+01:00] - Pod [Failed][ContainersNotInitialized] containers with incomplete status: [set-up-jenkins-agent]
[13:21:53.369+01:00] - Pod [Failed][PodFailed] No message
[13:21:53.370+01:00] - [PodInfo] *******************************-2fworkflows-1-hzpd-j2hqg
[13:21:53.370+01:00] - Container [kroki] waiting [PodInitializing] No message
[13:21:53.370+01:00] - Container [kroki-excalidraw] waiting [PodInitializing] No message
[13:21:53.370+01:00] - Container [kroki-mermaid] waiting [PodInitializing] No message
[13:21:53.370+01:00] - Container [mc] waiting [PodInitializing] No message
[13:21:53.370+01:00] - Container [techdoc] waiting [PodInitializing] No message
[13:21:53.370+01:00] - Pod [Failed][ContainersNotInitialized] containers with incomplete status: [set-up-jenkins-agent]
[13:21:53.370+01:00] - Pod [Failed][PodFailed] No message
[13:21:53.370+01:00] - [PodInfo] *******************************-2fworkflows-1-hzpd-j2hqg
[13:21:53.370+01:00] - Container [kroki] waiting [PodInitializing] No message
[13:21:53.370+01:00] - Container [kroki-excalidraw] waiting [PodInitializing] No message
[13:21:53.370+01:00] - Container [kroki-mermaid] waiting [PodInitializing] No message
[13:21:53.370+01:00] - Container [mc] waiting [PodInitializing] No message
[13:21:53.370+01:00] - Container [techdoc] waiting [PodInitializing] No message
[13:21:53.370+01:00] - Pod [Failed][ContainersNotInitialized] containers with incomplete status: [set-up-jenkins-agent]
[13:21:53.370+01:00] - Pod [Failed][PodFailed] No message
[13:21:53.371+01:00] - [PodInfo] *******************************-2fworkflows-1-hzpd-j2hqg
[13:21:53.371+01:00] - Container [kroki] waiting [PodInitializing] No message
[13:21:53.371+01:00] - Container [kroki-excalidraw] waiting [PodInitializing] No message
[13:21:53.371+01:00] - Container [kroki-mermaid] waiting [PodInitializing] No message
[13:21:53.371+01:00] - Container [mc] waiting [PodInitializing] No message
[13:21:53.371+01:00] - Container [techdoc] waiting [PodInitializing] No message
[13:21:53.371+01:00] - Pod [Failed][ContainersNotInitialized] containers with incomplete status: [set-up-jenkins-agent]
[13:21:53.371+01:00] - Pod [Failed][PodFailed] No message
[13:21:53.371+01:00] - [PodInfo] *******************************-2fworkflows-1-hzpd-j2hqg
[13:21:53.371+01:00] - Container [kroki] waiting [PodInitializing] No message
[13:21:53.371+01:00] - Container [kroki-excalidraw] waiting [PodInitializing] No message
[13:21:53.371+01:00] - Container [kroki-mermaid] waiting [PodInitializing] No message
[13:21:53.371+01:00] - Container [mc] waiting [PodInitializing] No message
[13:21:53.371+01:00] - Container [techdoc] waiting [PodInitializing] No message
[13:21:53.371+01:00] - Pod [Failed][ContainersNotInitialized] containers with incomplete status: [set-up-jenkins-agent]
[13:21:53.371+01:00] - Pod [Failed][PodFailed] No message
[13:21:53.372+01:00] - [PodInfo] *******************************-2fworkflows-1-hzpd-j2hqg
[13:21:53.372+01:00] - Container [kroki] waiting [PodInitializing] No message
[13:21:53.372+01:00] - Container [kroki-excalidraw] waiting [PodInitializing] No message
[13:21:53.372+01:00] - Container [kroki-mermaid] waiting [PodInitializing] No message
[13:21:53.372+01:00] - Container [mc] waiting [PodInitializing] No message
[13:21:53.372+01:00] - Container [techdoc] waiting [PodInitializing] No message
[13:21:53.372+01:00] - Pod [Failed][ContainersNotInitialized] containers with incomplete status: [set-up-jenkins-agent]
[13:21:53.372+01:00] - Pod [Failed][PodFailed] No message
[13:21:53.373+01:00] - [PodInfo] *******************************-2fworkflows-1-hzpd-j2hqg
[13:21:53.373+01:00] - Container [kroki] waiting [PodInitializing] No message
[13:21:53.373+01:00] - Container [kroki-excalidraw] waiting [PodInitializing] No message
[13:21:53.373+01:00] - Container [kroki-mermaid] waiting [PodInitializing] No message
[13:21:53.373+01:00] - Container [mc] waiting [PodInitializing] No message
[13:21:53.373+01:00] - Container [techdoc] waiting [PodInitializing] No message
[13:21:53.373+01:00] - Pod [Failed][ContainersNotInitialized] containers with incomplete status: [set-up-jenkins-agent]
[13:21:53.373+01:00] - Pod [Failed][PodFailed] No message
[13:21:53.375+01:00] - [PodInfo] *******************************-2fworkflows-1-hzpd-j2hqg
[13:21:53.375+01:00] - Container [kroki] waiting [PodInitializing] No message
[13:21:53.375+01:00] - Container [kroki-excalidraw] waiting [PodInitializing] No message
[13:21:53.375+01:00] - Container [kroki-mermaid] waiting [PodInitializing] No message
[13:21:53.375+01:00] - Container [mc] waiting [PodInitializing] No message
[13:21:53.375+01:00] - Container [techdoc] waiting [PodInitializing] No message
[13:21:53.375+01:00] - Pod [Failed][ContainersNotInitialized] containers with incomplete status: [set-up-jenkins-agent]
[13:21:53.375+01:00] - Pod [Failed][PodFailed] No message
[13:21:53.378+01:00] - [PodInfo] *******************************-2fworkflows-1-hzpd-j2hqg
[13:21:53.378+01:00] - Container [kroki] waiting [PodInitializing] No message
[13:21:53.378+01:00] - Container [kroki-excalidraw] waiting [PodInitializing] No message
[13:21:53.378+01:00] - Container [kroki-mermaid] waiting [PodInitializing] No message
[13:21:53.378+01:00] - Container [mc] waiting [PodInitializing] No message
[13:21:53.378+01:00] - Container [techdoc] waiting [PodInitializing] No message
[13:21:53.378+01:00] - Pod [Failed][ContainersNotInitialized] containers with incomplete status: [set-up-jenkins-agent]
[13:21:53.378+01:00] - Pod [Failed][PodFailed] No message
[13:21:53.378+01:00] - [PodInfo] *******************************-2fworkflows-1-hzpd-j2hqg
[13:21:53.378+01:00] - Container [kroki] waiting [PodInitializing] No message
[13:21:53.378+01:00] - Container [kroki-excalidraw] waiting [PodInitializing] No message
[13:21:53.378+01:00] - Container [kroki-mermaid] waiting [PodInitializing] No message
[13:21:53.378+01:00] - Container [mc] waiting [PodInitializing] No message
[13:21:53.378+01:00] - Container [techdoc] waiting [PodInitializing] No message
[13:21:53.378+01:00] - Pod [Failed][ContainersNotInitialized] containers with incomplete status: [set-up-jenkins-agent]
[13:21:53.378+01:00] - Pod [Failed][PodFailed] No message
[13:21:53.378+01:00] - [PodInfo] *******************************-2fworkflows-1-hzpd-j2hqg
[13:21:53.378+01:00] - Container [kroki] waiting [PodInitializing] No message
[13:21:53.378+01:00] - Container [kroki-excalidraw] waiting [PodInitializing] No message
[13:21:53.378+01:00] - Container [kroki-mermaid] waiting [PodInitializing] No message
[13:21:53.378+01:00] - Container [mc] waiting [PodInitializing] No message
[13:21:53.378+01:00] - Container [techdoc] waiting [PodInitializing] No message
[13:21:53.378+01:00] - Pod [Failed][ContainersNotInitialized] containers with incomplete status: [set-up-jenkins-agent]
[13:21:53.378+01:00] - Pod [Failed][PodFailed] No message
[13:21:53.379+01:00] - [PodInfo] *******************************-2fworkflows-1-hzpd-j2hqg
[13:21:53.379+01:00] - Container [kroki] waiting [PodInitializing] No message
[13:21:53.379+01:00] - Container [kroki-excalidraw] waiting [PodInitializing] No message
[13:21:53.379+01:00] - Container [kroki-mermaid] waiting [PodInitializing] No message
[13:21:53.379+01:00] - Container [mc] waiting [PodInitializing] No message
[13:21:53.379+01:00] - Container [techdoc] waiting [PodInitializing] No message
[13:21:53.379+01:00] - Pod [Failed][ContainersNotInitialized] containers with incomplete status: [set-up-jenkins-agent]
[13:21:53.379+01:00] - Pod [Failed][PodFailed] No message
[13:21:53.380+01:00] - [PodInfo] *******************************-2fworkflows-1-hzpd-j2hqg
[13:21:53.380+01:00] - Container [kroki] waiting [PodInitializing] No message
[13:21:53.380+01:00] - Container [kroki-excalidraw] waiting [PodInitializing] No message
[13:21:53.380+01:00] - Container [kroki-mermaid] waiting [PodInitializing] No message
[13:21:53.380+01:00] - Container [mc] waiting [PodInitializing] No message
[13:21:53.380+01:00] - Container [techdoc] waiting [PodInitializing] No message
[13:21:53.380+01:00] - Pod [Failed][ContainersNotInitialized] containers with incomplete status: [set-up-jenkins-agent]
[13:21:53.380+01:00] - Pod [Failed][PodFailed] No message
[13:21:53.381+01:00] - [PodInfo] *******************************-2fworkflows-1-hzpd-j2hqg
[13:21:53.381+01:00] - Container [kroki] waiting [PodInitializing] No message
[13:21:53.381+01:00] - Container [kroki-excalidraw] waiting [PodInitializing] No message
[13:21:53.381+01:00] - Container [kroki-mermaid] waiting [PodInitializing] No message
[13:21:53.381+01:00] - Container [mc] waiting [PodInitializing] No message
[13:21:53.381+01:00] - Container [techdoc] waiting [PodInitializing] No message
[13:21:53.381+01:00] - Pod [Failed][ContainersNotInitialized] containers with incomplete status: [set-up-jenkins-agent]
[13:21:53.381+01:00] - Pod [Failed][PodFailed] No message
[13:21:53.381+01:00] - [PodInfo] *******************************-2fworkflows-1-hzpd-j2hqg
[13:21:53.381+01:00] - Container [kroki] waiting [PodInitializing] No message
[13:21:53.381+01:00] - Container [kroki-excalidraw] waiting [PodInitializing] No message
[13:21:53.381+01:00] - Container [kroki-mermaid] waiting [PodInitializing] No message
[13:21:53.381+01:00] - Container [mc] waiting [PodInitializing] No message
[13:21:53.381+01:00] - Container [techdoc] waiting [PodInitializing] No message
[13:21:53.381+01:00] - Pod [Failed][ContainersNotInitialized] containers with incomplete status: [set-up-jenkins-agent]
[13:21:53.381+01:00] - Pod [Failed][PodFailed] No message
[13:21:53.386+01:00] - [PodInfo] *******************************-2fworkflows-1-hzpd-j2hqg
[13:21:53.386+01:00] - Container [kroki] waiting [PodInitializing] No message
[13:21:53.386+01:00] - Container [kroki-excalidraw] waiting [PodInitializing] No message
[13:21:53.386+01:00] - Container [kroki-mermaid] waiting [PodInitializing] No message
[13:21:53.386+01:00] - Container [mc] waiting [PodInitializing] No message
[13:21:53.386+01:00] - Container [techdoc] waiting [PodInitializing] No message
[13:21:53.386+01:00] - Pod [Failed][ContainersNotInitialized] containers with incomplete status: [set-up-jenkins-agent]
[13:21:53.386+01:00] - Pod [Failed][PodFailed] No message
[13:21:53.500+01:00] - [Pipeline] // node
[13:21:53.510+01:00] - [Pipeline] }
[13:21:53.523+01:00] - [Pipeline] // podTemplate
[13:21:53.596+01:00] - [Pipeline] stage
[13:21:53.598+01:00] - [Pipeline] { (Declarative: Post Actions)
[13:21:53.892+01:00] - [Pipeline] echo
[13:21:53.892+01:00] - Skip email according to configuration.
[13:21:53.911+01:00] - [Pipeline] }
[13:21:53.959+01:00] - [Pipeline] // stage
[13:21:53.983+01:00] - [Pipeline] End of Pipeline
[13:21:54.385+01:00] - Queue task was cancelled
[13:21:54.385+01:00] - org.jenkinsci.plugins.workflow.actions.ErrorAction$ErrorId: 730a7b53-ba05-416d-a625-1bbbfe9d4ef2
[13:21:54.385+01:00] - [Bitbucket] Notifying commit build result
[13:21:54.453+01:00] - [Bitbucket] Build result notified
[13:21:54.471+01:00] - Finished: ABORTED
Meaning it fail before starting the first stage (implicit checkout).
Is it normal the pipeline didn’t restart? Is this the expected behavior of the Kuberentes plugin? If this is expected, do we have a ways to restart such pipeline? (Maybe on the cloud config?)
Thanks!