Hi ,
We are running Jenkins core 2.387.1 on Linux VM shipping with 24G Ram with jdk 11.018 version.
We have around 30+ Linux, Windows and Mac nodes under the Jenkins , the jenkins service config is as belows:
=====
JAVA_HEAP="-Xms24g -Xmx24g -Xmn4g -XX:MaxNewSize=4g"
JAVA_G1GC="-XX:+UseG1GC -XX:+ExplicitGCInvokesConcurrent -XX:+ParallelRefProcEnabled -XX:+UseStringDeduplication -XX:+UnlockExperimentalVMOptions -XX:G1NewSizePercent=20 -XX:+UnlockDiagnosticVMOptions -XX:G1SummarizeRSetStatsPeriod=1"
=====
We noticed performance hit recently with normal work load and $jenkinsurl/montoring show up 90% of the ram usage as below screenshot:
Somehow I could not upload the full gc log here ( system says that I am the new user ), so I paste the last several lines of log entries here:
=================
42711 [514497.706s][info][gc] GC(30943) Concurrent Cycle
42712 [514503.605s][info][gc] GC(30944) Pause Young (Normal) (G1 Evacuation Pause) 22609M->18633M(24576M) 84.032ms
42713 [514539.930s][info][gc] GC(30943) Pause Remark 19189M->19053M(24576M) 36183.931ms
42714 [514541.142s][info][gc] GC(30945) Pause Young (Normal) (G1 Evacuation Pause) 22513M->18538M(24576M) 64.978ms
42715 [514547.632s][info][gc] GC(30946) Pause Young (Normal) (G1 Evacuation Pause) 22602M->18654M(24576M) 78.366ms
42716 [514548.428s][info][gc] GC(30943) Pause Cleanup 19173M->19173M(24576M) 2.937ms
42717 [514548.447s][info][gc] GC(30943) Concurrent Cycle 50741.134ms
42718 [514551.575s][info][gc] GC(30947) Pause Young (Concurrent Start) (G1 Humongous Allocation) 21280M->18687M(24576M) 70.152ms
42719 [514551.576s][info][gc] GC(30948) Concurrent Cycle
42720 [514593.582s][info][gc] GC(30948) Pause Remark 21263M->21015M(24576M) 36208.755ms
42721 [514594.269s][info][gc] GC(30949) Pause Young (Normal) (G1 Evacuation Pause) 22455M->18510M(24576M) 111.931ms
42722 [514601.599s][info][gc] GC(30950) Pause Young (Normal) (G1 Evacuation Pause) 22582M->18658M(24576M) 84.088ms
42723 [514602.852s][info][gc] GC(30948) Pause Cleanup 19186M->19186M(24576M) 3.034ms
42724 [514602.869s][info][gc] GC(30948) Concurrent Cycle 51293.526ms
42725 [514611.217s][info][gc] GC(30951) Pause Young (Normal) (G1 Evacuation Pause) 22586M->18670M(24576M) 59.382ms
42726 [514618.669s][info][gc] GC(30952) Pause Young (Concurrent Start) (G1 Humongous Allocation) 20916M->18692M(24576M) 73.119ms
42727 [514618.669s][info][gc] GC(30953) Concurrent Cycle
42728 [514626.925s][info][gc] GC(30953) Pause Remark 22003M->21811M(24576M) 128.246ms
42729 [514627.548s][info][gc] GC(30954) Pause Young (Normal) (G1 Evacuation Pause) 22540M->18635M(24576M) 85.216ms
42730 [514632.921s][info][gc] GC(30955) Pause Young (Normal) (G1 Evacuation Pause) 22651M->18746M(24576M) 83.755ms
42731 [514636.242s][info][gc] GC(30953) Pause Cleanup 20209M->20209M(24576M) 3.641ms
42732 [514636.260s][info][gc] GC(30953) Concurrent Cycle 17590.345ms
42733 [514636.414s][info][gc] GC(30956) Pause Young (Concurrent Start) (G1 Humongous Allocation) 20228M->18785M(24576M) 101.124ms
42734 [514636.414s][info][gc] GC(30957) Concurrent Cycle
42735 [514678.613s][info][gc] GC(30957) Pause Remark 22100M->21812M(24576M) 36297.902ms
42736 [514679.026s][info][gc] GC(30958) Pause Young (Normal) (GCLocker Initiated GC) 22657M->18700M(24576M) 90.272ms
42737 [514684.609s][info][gc] GC(30959) Pause Young (Normal) (G1 Evacuation Pause) 22732M->18835M(24576M) 112.319ms
42738 [514687.890s][info][gc] GC(30957) Pause Cleanup 20576M->20576M(24576M) 3.738ms
42739 [514687.907s][info][gc] GC(30957) Concurrent Cycle 51493.623ms
42740 [514688.108s][info][gc] GC(30960) Pause Young (Concurrent Start) (G1 Humongous Allocation) 20610M->18855M(24576M) 86.960ms
42741 [514688.108s][info][gc] GC(30961) Concurrent Cycle
42742 [514695.738s][info][gc] GC(30962) Pause Young (Normal) (G1 Evacuation Pause) 22815M->18905M(24576M) 67.546ms
42743 [514696.363s][info][gc] GC(30961) Pause Remark 19087M->18759M(24576M) 107.369ms
42744 [514705.274s][info][gc] GC(30961) Pause Cleanup 22091M->22091M(24576M) 3.337ms
42745 [514705.292s][info][gc] GC(30961) Concurrent Cycle 17184.066ms
42746 [514705.796s][info][gc] GC(30963) Pause Young (Concurrent Start) (G1 Humongous Allocation) 22302M->18669M(24576M) 87.089ms
42747 [514705.796s][info][gc] GC(30964) Concurrent Cycle
42748 [514747.664s][info][gc] GC(30964) Pause Remark 20284M->20188M(24576M) 36027.053ms
42749 [514749.902s][info][gc] GC(30965) Pause Young (Normal) (G1 Evacuation Pause) 22517M->18661M(24576M) 145.408ms
42750 [514756.874s][info][gc] GC(30964) Pause Cleanup 20812M->20812M(24576M) 4.527ms
42751 [514756.891s][info][gc] GC(30964) Concurrent Cycle 51094.858ms
42752 [514760.400s][info][gc] GC(30966) Pause Young (Concurrent Start) (G1 Humongous Allocation) 21657M->18682M(24576M) 90.502ms
42753 [514760.400s][info][gc] GC(30967) Concurrent Cycle
42754 [514767.273s][info][gc] GC(30967) Pause Remark 20689M->20689M(24576M) 0.710ms
42755 [514773.014s][info][gc] GC(30968) Pause Young (Normal) (G1 Evacuation Pause) 22666M->18784M(24576M) 72.750ms
42756 [514773.872s][info][gc] GC(30967) Pause Remark 19194M->19090M(24576M) 103.981ms
42757 [514782.734s][info][gc] GC(30967) Pause Cleanup 21562M->21562M(24576M) 3.252ms
42758 [514782.752s][info][gc] GC(30967) Concurrent Cycle 22352.195ms
42759 [514788.180s][info][gc] GC(30969) Pause Young (Normal) (G1 Evacuation Pause) 22600M->18672M(24576M) 66.497ms
42760 [514791.238s][info][gc] GC(30970) Pause Young (Concurrent Start) (G1 Humongous Allocation) 20061M->18670M(24576M) 86.174ms
42761 [514791.238s][info][gc] GC(30971) Concurrent Cycle
=================
Meanwhile, we upload the gc log against GC log online analysis tool and has the report here :
We are wondering if there is anything noticable from above that we can tune up ?