Jenkins build logs aggregation valuable?

Using Jenkins
1

Guys, I am looking for strong advice about solution to monitor jenkins build logs. I do have in place monitoring jenkins app logs and all functional data related to running the jenkins on the infra. However, some people come into the conclusion to ship logs from builds, as well. In very complex setup is not worth to perform such action, due to the fact that each case is different (mostly) + cost efficiency about storing them on RDS in AWS(massive).
Idea of monitoring is to catch up the trends and metrics related or infra app layer from functional point of view, do you agree ?
Based on, theory that is need to monitor build logs, means cost are massive + filters are not so unique and you need to deep dive to troubleshoot, anyway. Effort become pointless.
Have you heard about consideration, previously such a thing ? What tools might be useful ?

Thanks for discussion.

2

Several different groups are finding ways to more widely use the logs from Jenkins jobs. You might consider their approaches in your investigation.

The OpenTelemetry plugin can export Jenkins build logs to storage and analysis services like ElasticSearch, Jaeger, and more. Elastic

@darinpope has created a video that shows how to use OpenTelemetry and Jaeger to trace Jenkins pipelines.

Elastic includes a section on CI/CD observability in their documentation

Logz.io has a blog post that describes their use of observability on their CI/CD Pipelines:

3

@MarkEWaite
That’s true, the conclusion from @darinpope . However, would name it more user friendly interface in Jaeger + time tracker add-on. Nevertheless, re-writing pipeline in my case, as outcome of implementation would be case to address to an author of the pipeline. That also make sense, not sure if people would react even on data provided to them, about produced problem by them. Nothing to carry on that thread.
In very short, looking for advantages of this implementation shared by you. I am seeing
-measuring the time of particular step in the pipe +

  • evaluation of interface,
    but manual action is required anyway for review.
4

Hey @MarkEWaite
I was recently setting up Cloudbees CI rather than Jenkins-LTS.
I was following the video which you shared and did follow all the steps.
The traces for opentelemetry plugin is not getting generated for cloudbees CI and working for jenkins-LTS.
Opentelemetry plugin is there in its marketplace.
version -
3.1138.v80fc844ed246Latest
on Mar 18