Jenkins in docker intermittent problem with failed to fetch anonymous token

I am running Jenkins in a Docker environment, which in turn is running in a virtual machine. This works very well most of the time, but every now and then all jobs fail with this error message (actual address replaced with xxxx):
Dockerfile:1

1 | >>> FROM ubuntu:22.04

ERROR: failed to solve: failed to fetch anonymous token: Get “https://auth.docker.io/token?scope=repository%3Alibrary%2Fubuntu%3Apull&service=registry.docker.io”: dial tcp [xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx:xxxx]:443: connect: cannot assign requested address

When it happens it usually continues to happen for a period of minutes, until it suddenly starts working again.
I have disabled IPv6 on the docker host to no avail.

Jenkins setup:
Jenkins: 2.414.3
OS: Linux - 5.15.0-87-generic
Java: 11.0.18 - Eclipse Adoptium (OpenJDK 64-Bit Server VM)

analysis-model-api:11.8.0
antisamy-markup-formatter:159.v25b_c67cd35fb_
apache-httpcomponents-client-4-api:4.5.14-208.v438351942757
apache-httpcomponents-client-5-api:5.2.1-1.0
authentication-tokens:1.4
bootstrap5-api:5.3.0-1
bouncycastle-api:2.29
branch-api:2.1105.v472604208c55
build-monitor-plugin:1.14-744.v35fd6fa_a_26b_2
build-timeout:1.28
caffeine-api:3.1.8-133.v17b_1ff2e0599
checks-api:2.0.0
cloudbees-folder:6.815.v0dd5a_cb_40e0e
command-launcher:107.v773860566e2e
commons-lang3-api:3.13.0-62.v7d18e55f51e2
commons-text-api:1.10.0-78.v3e7b_ea_d5a_fe1
cppcheck:1.26
credentials:1271.v54b_1c2c6388a_
credentials-binding:626.v8d9034b_8ea_cc
data-tables-api:1.13.6-4
discard-old-build:1.07
display-url-api:2.3.9
docker-commons:419.v8e3cd84ef49c
docker-java-api:3.3.1-79.v20b_53427e041
docker-plugin:1.3.0
docker-workflow:572.v950f58993843
durable-task:523.va_a_22cf15d5e0
echarts-api:5.4.0-5
email-ext:2.99
font-awesome-api:6.4.2-1
forensics-api:2.3.0
git:5.0.2
git-client:4.3.0
github:1.37.2
github-api:1.314-431.v78d72a_3fe4c3
github-branch-source:1728.v859147241f49
htmlpublisher:1.32
instance-identity:173.va_37c494ec4e5
ionicons-api:56.v1b_1c8c49374e
jackson2-api:2.15.2-350.v0c2f3f8fc595
jakarta-activation-api:2.0.1-3
jakarta-mail-api:2.0.1-3
javax-activation-api:1.2.0-6
javax-mail-api:1.6.2-9
jaxb:2.3.8-1
jdk-tool:73.vddf737284550
jjwt-api:0.11.5-77.v646c772fddb_0
jquery3-api:3.7.0-1
junit:1207.va_09d5100410f
ldap:694.vc02a_69c9787f
mailer:457.v3f72cb_e015e5
mapdb-api:1.0.9-28.vf251ce40855d
matrix-auth:3.1.9
matrix-project:802.v8013b_40c7edc
memory-map:2.2.1 (53614)
mina-sshd-api-common:2.10.0-69.v28e3e36d18eb_
mina-sshd-api-core:2.10.0-69.v28e3e36d18eb_
okhttp-api:4.11.0-145.vcb_8de402ef81
pam-auth:1.10
pipeline-build-step:491.v1fec530da_858
pipeline-github-lib:42.v0739460cda_c4
pipeline-graph-analysis:202.va_d268e64deb_3
pipeline-groovy-lib:687.v62591d623759
pipeline-input-step:477.v339683a_8d55e
pipeline-milestone-step:111.v449306f708b_7
pipeline-model-api:2.2133.ve46a_6113dfc3
pipeline-model-definition:2.2133.ve46a_6113dfc3
pipeline-model-extensions:2.2133.ve46a_6113dfc3
pipeline-rest-api:2.33
pipeline-stage-step:305.ve96d0205c1c6
pipeline-stage-tags-metadata:2.2133.ve46a_6113dfc3
pipeline-stage-view:2.33
pipeline-utility-steps:2.16.0
plain-credentials:143.v1b_df8b_d3b_e48
plugin-util-api:3.3.0
popper2-api:2.11.6-2
prism-api:1.29.0-8
resource-disposer:0.23
robot:3.4.0
schedule-build:502.v9379e178e65b_
scm-api:676.v886669a_199a_a_
script-security:1269.v639888f5e366
snakeyaml-api:2.2-111.vc6598e30cc65
ssh-credentials:305.v8f4381501156
ssh-slaves:2.877.v365f5eb_a_b_eec
sshd:3.312.v1c601b_c83b_0e
structs:325.vcb_307d2a_2782
subversion:2.17.3
timestamper:1.25
token-macro:359.vb_cde11682e0c
trilead-api:2.84.v72119de229b_7
variant:60.v7290fc0eb_b_cd
warnings-ng:10.4.0
workflow-aggregator:596.v8c21c963d92d
workflow-api:1281.vca_5fddb_3fceb_
workflow-basic-steps:1042.ve7b_140c4a_e0c
workflow-cps:3673.v5b_dd74276262
workflow-durable-task-step:1289.v4d3e7b_01546b_
workflow-job:1326.ve643e00e9220
workflow-multibranch:746.v05814d19c001
workflow-scm-step:408.v7d5b_135a_b_d49
workflow-step-api:639.v6eca_cd8c04a_a_
workflow-support:839.v35e2736cfd5c
ws-cleanup:0.45