Infrastructure Team Meeting - August 08, 2022

Attendees :busts_in_silhouette:

Announcements :loudspeaker:

  1. Weekly 2.363
    • Complete, checklist complete, Docker images confirmed built
    • Revision to the changelog is submitted
    • Git client plugin security fix for host key verification broke the packaging step
  2. Costs
    • July expenses high on all platforms in July
      • May have missed autoscaling on kubernetes, manually scaled to complete a release, forgot to autoscale
      • Had machines running but idle
      • Fixed last Friday, reduced Azure cost estimate
    • AWS expenses at $12k per month
      • Needs more research to understand costs
    • DigitalOcean credits are low
      • Email notice to the DigitalOcean sponsors to increase
  3. Salad Cloud
  4. Launchable
    • Kohsuke Kawaguchi would like to allow the Jenkins project to use Launchable
    • Would likely extend buildPlugin() to send test duration and result info to Launchable service

Upcoming Calendar :calendar:

  • Next Weekly: 2.364 - 16th of August 2022
  • Next LTS: Tomorrow - 10th of August 2022 (infra stable tomorrow)
    • Alex Brandes is release lead
  • Next Security Release: N.A.
  • Next major event: N.A.

Notes :book:

1 Like

Hi! it’s Bob from Salad here. Apologies for missing this, I wasn’t sure of the logistics and just got an email notification. Would love to join next week (please let me know how)…and I see the cost of AWS is an agenda item :wink:

1 Like

Are there (public) stats of AWS costs for comparision?

Tim and I cleared the backlog of dependency updates of the ATH earlier in July, which runs on highmem nodes only, plus we did a bunch of core PR testing, I’m wondering if there’s a coincidence.

Hi Bob, no problem. Feel free to join us today: the meeting is referenced in this page: Events (all of our meeting are public).

That could be linked to the overall cost increase (at least make sense).

Spending credits is not a bad things: in this case it means people worked on the topic.

But from infrastructure prospective, there are still 2 issues:

  • A lot of failing agents triggered unwanted costs (by autoscaling worker nodes while pod where failing). Infra need to improve on this topic.
  • Infra team forgot to disable manual scaling on Azure, which also costed few bucks.