How to get process-level memory detail when a Kubernetes agent pod is OOMKilled?

Using Jenkins
1

Hi all,
We’re running Jenkins with the Kubernetes plugin (agents provisioned as pods on EKS), and I’m trying to find a reliable way to identify which specific process inside an agent pod consumed memory before it gets OOMKilled — not just the fact that it happened.

Jenkins setup:

1.Jenkins Kubernetes plugin, pod templates with explicit memory limits set (e.g., 256Mi)

2.Agent pods run the standard JNLP container plus whatever the pipeline’s sh steps spawn

What I’ve already tried, and where each one falls short:

  1. kubectl describe pod — shows Reason: OOMKilled, exit code 137, and the limit that was hit, but only at the container level. Doesn’t tell me which process inside the container was responsible. Also only works if the pod hasn’t been deleted yet.
  2. Jenkins’ own [PodInfo] log listener (via a k8s-agent-debug log recorder on org.csanchez.jenkins.plugins.kubernetes) — reliably reports Container [jnlp] terminated [OOMKilled] in the build console log every time, but again, no process-level breakdown.
  3. OpenTelemetry-based log export (we ship agent logs via the OTel plugin to an external log platform) — confirmed this cannot capture the OOMKilled event at all, since the JNLP agent process is what would normally emit/export the log, and it’s the process being killed. Logs simply stop at the last line printed before death.
  4. Kubernetes Events (kubectl get events) — confirmed no Warning event gets generated for a container hitting its own cgroup memory limit (this differs from node-level SystemOOM eviction, which does generate an event). So an events-based exporter wouldn’t catch this either.
  5. Manual node-level kernel log (journalctl -k via a privileged debug pod with the node’s root filesystem mounted) — this is the only method that worked. The kernel’s OOM-killer log shows exactly which PID/process name/anon-rss was killed. But this requires manually creating a debug pod after the fact and reading the log before it rotates out — not something that scales across ~200 jobs without a human noticing and reacting each time.

My question: Is there an existing, more native way to get this level of detail — either through the Kubernetes plugin itself, a companion plugin, or a recommended pattern others use? Or is the general approach here to build a standing collection layer (e.g., shipping node kernel/journald logs continuously via a DaemonSet) since Jenkins has no visibility into subprocess-level memory by design?

Jenkins version: 2.555.1

Eks version: 1.35

Kubernetes Pluggin: 4547.v52f3080db_8cd