How do I avoid plugin mirror sites X during downloading?

Using Jenkins Ask a question
1

Hi all,

I am automatically downloading Jenkins plugins into an air gapped environment. That worked pretty good over the last years. Because of regulatory reasons I am no longer allowed to download plugins from a one or more mirror servers (aka block list).

So my two questions are:

  1. From which alternative location should I download the plugin?
  2. And how can I determine the location automatically?

What I tried until now:

  • I know the ...plugins/git/5.2.2/git.hpi?mirrorlist trick but that only provides an HTML page and we all know parsing HTML for links is more than unpleasant.
  • I tried to curl the plugin URL with a proper Accept: application/json header as can be seen in the mirrorbits source but it too returns only HTML.
  • URLs like pkg.jenkins.io/plugins/git/5.2.2/git.hpi or jenkins.io/plugins/git/5.2.2/git.hpi do not work

What will not work for me:

  • parsing HTML because it depends on HTML URL location knowledge and is too brittle for the future
  • browsing the HTML page by hand and copying the URL

What I hope for:

  • a static URL to fall back to (bypassing mirrors at all)
  • a JSON result from which I can select the next mirror (choose a different mirror)

For your understanding, my workflow looks like this:

  1. load next plugin metadata from the official update center JSON => url and so on
  2. GET https://get.jenkins.io/plugins/${plugin}/${version}/${plugin}.hpi
  3. analyze response
    • if response Code is 302 and Location is not in ${UrlBlockList}: yay
    • else: download from original source or other mirror (HELP)

Please note: Doing a full mirror is out of the question.

Thanks and cheers

2

Since you’re running an offline installation, it seems like you can safely assume that all the mirrors have all the files that are interesting to you. If the selected mirror fails, fallback to another hard-coded mirror (based on your location) and try again. If that assumption is proven invalid, then you delay your update for 24 hours and try again.

We strongly prefer that people not access it directly, because it bypasses the mirrors, but archives.jenkins.io retains copies of all the Jenkins plugin releases and core releases. Please, please only use it as a fallback, not as your primary download location.

3

Thank you Mark for your swift reply. I fully understand your concerns regarding archives.jenkins.io as a main download target. After all the mirrors are there for a reason (mostly that server fee money doesn’t grow on trees :smile: )

I monitored the mirror redirects locations and found that more than often not even the nearest (even though) unwanted mirror is chosen which cues the idea that not all mirrors are the fully in sync. I am not sure but for now I will rewrite the mirror URL with a secondary near mirror and will watch the results.

I’ll let you know how that worked out. Thanks again :pray: