I have observed some weird behaviour when submitting parametrized pipeline job (buildWithParameters) and I would like to troubleshoot it further.
I run Jenkins controller in a docker container with docker-compose and logs are available until restart with docker-compose logs.
However, master seem to log mostly mundane Java things, and errors like 500 exceptions (they happen if wrong parameters are submitted) appear unrelated to job activities.
I was thinking about forwarding the log to some external logging facility (Splunk maybe) - I have seen some examples how to modify Java logging to forward events to external collectors. I am not interesting in forwarding text log contents, I want to have structured logging - one Java exception should be one event and not multiple lines.
My question is what are the best practices to:
- Collect logs from the controller
- Collect logs from the agents
- I’d like also have logs from Kubernetes related to the Kubernetes agents
- Collect build events and logs (I have Splunk plugin for that for example)
- Let applications running as Jenkins pipelines log centrally
- And, most importantly, how to correlate builds with events from (1), (2), (3) and (5)?
What I’d like to have is some permanent setup for better observability of the system, not just a one-time debugging thing.