I need to run identical steps (manipulation of files) on different servers (agents) that have different files. E.g.:
- server A has files A1-A3
- server B has files B1-B6
- server C has a file C1
Every server has to run the steps with its own filelist independently and in parallel of other servers. Every file has to be processed in 3 steps. Step1 → then step2 → then send to network share (step3).
So my ideas were:
Idea1 . Create a map:
filelist = [ [serverhostname: serverA, files: [A1, A2, A3]], [serverhostname: serverB, files: [B1, B2, B3, B4, B5, B6]], [serverhostname: serverC, files: [C1]] ]
Idea2 . Generate parallel stages and pass the map into these stages. I read an article Generating parallel stages in Jenkinsfile according to passed parameters but couldn’t combine examples into a working code.
Idea3 . In order to limit servers loading (step2 eats CPU, step3 eats network bandwidth) I want to process only 1 file at the moment by every server but not whole fileset.