Programmatic CI plan generation and execution via plugin

Contributing
1

Greetings all. I’ve seen a few related posts and questions on the web, but nothing verbose or particularly concrete.

Objective

I want a CI pipeline entirely defined by my plugin, without consuming config from userspace, such as a Jenkinsfile. The only initial build step is my-plugin, and it creates a CI plan (collection of tasks), allocates nodes/executors, and feeds work to workers (a pull system vs a push system) until all of the work distributed and completed.

Specifically, the plugin shall host the following capabilities:

  • derive the tasks needed to execute. Consider this work a black-box for now. I’ll shell out to some toolkit for this.
  • Reserve N nodes from my compute pool
  • For node n in N nodes, for each executor e in n, request work from the controller and execute it in parallel, until the the controller has no work remaining.

Discussion

  • The ability to schedule work into the current build/run isn’t entirely clear to me yet. Some internet searches have suggested “the best you can do is trigger another job”, but surely their must be a way to programmatically add steps into the current run?
    • (p.s. I’m a baby jenkins plugin dev if it’s not already obvious. I’ve merely completed the HelloWorld plugin dev-ux, but happy to report that it went off without a hitch :slight_smile: )
  • How does one inspect the jenkins agent pool and those agents’ capabilities in-plugin?
  • A stretch goal would be to be able to efficiently transport build artifacts between child tasks running on different nodes/workspaces without some intermediate cache provider!

I’m not looking for anyone to dev this for me (obviously), but, I am seeking advice from plugin author veterans. Help me with APIs, or even an opinionated roadmap for what low-level jenkins resources I’ll need to stitch together, what lifecycle hooks to plug into, points of friction you think I’ll encounter… etc etc.

Thanks for your time!

2

You can’t reserve nodes in Jenkins. The best you get is to work with cloud agents that get instantiated when you have the need for one.

An executor can’t request work from the controller, it is the other way round. The controller assigns a task to an executor.

In Jenkins wou work with nodes (aka agent). An nodes provides one or more executors that will be able to execute a task. Usually you assign labels to your nodes to define what kind of tasks can be executed, e.g. when you have to support different platforms you would give your windows nodes the label windows and your linux nodes the label linux. You can add more labels to further distinguish the agents if they have special abilities or tools

What you describe sounds to be something that should be implemented as a shared library and not as a plugin. Plugins have the big disadvantage that you need to restart Jenkins when you need to update the plugin. For a shared library you just need to push the new version to your repo and depending how you define it, it becomes instantly active for newly started runs.

You can probably implement this in a plugin but you will need to repeat all the work that has already been done in core and other plugins. And this requires deep knowledge about Jenkins internals.

1 Like
3

Thanks for pitching in.

Right. However, inverting that behavior is really the primary objective of the project. The impl may not literally invert it, but the practical outcome shall. Eg the controller maybe sends to the executor “get work from me and loop.” Practically it would be getting work from my lib, not the controllers core impl.

should be implemented as a shared library and not as a plugin

Thanks. Sounds reasonable to me!