Enterprise Manager package mirror DNS-Cache

[jbender81]

Moin,
Today we had issues with our VBR's not being able to download update information via the enterprise manager's package "mirror".
The error message was "Curl error (28): Timeout was reached for https://<EM-FQDN>/vsa/9.6/vbr/13.1/mandatory/repodata/repomd.xml [Operation too slow. Less than 1000 bytes/sec transferred the last 30 seconds]
22463 Error: Failed to download metadata for repo 'veeam-backup-mandatory': Cannot download repomd.xml: Cannot download repodata/repomd.xml: All mirrors were tried".
Although at the same the update-check on the EM worked fine, as well as a manual curl for "https://repository.veeam.com/vsa/9.6/vb ... repomd.xml" on the EM itself.
Also, this had already worked yesterday perfectly fine.
As such I had a look in our firewall log, and I noticed, that the EM used different destination IP-Addresses depending on whether I triggered the update-check on the EM itself or on the VBR.

Digging through the configuration how the package "mirror", or more precisely cache, works, I found, that this is done using nginx proxy_pass and proxy_cache mechanism to implement this cache. In this setup are the lines:
> proxy_pass https://repository.veeam.com$request_uri;
and
> proxy_pass https://repository.veeam.com/vsa/;
Unfortunately, this has the effect, that the FQDN's therein are resolved only _once_ during startup of nginx and used indefinitely until restart of the service.
For this to work properly, nginx has to be configured to re-resolve the FQDN's according to TTL using a configured resolver, and instruct proxy_pass to do a resolution for that FQDN.

This behaviour is described here https://joshua.hu/nginx-dns-caching

This could be confirmed by restarting the whole EM or nginx.service

PS:
Please do _not_ hardcode a resolver like Google, CloudFlare or Quad9, but try to use systemd-resolved as local resolver, as this allows to use 127.0.0.53 as resolver for anything, that requires a "hardcoded" resolver.

Cheers
Jörg