Excessive DNS requests from VeeamONE server

[EdwardsCP]

I have an existing case open with support (case #00820404). I'm posting here to see if anyone has the appropriate logging set up to see if the same issue exists in their environment.

A brief description of my environment: 2 sites. Site A has a server running VeeamONE, and a Veeam B&R server (veeam01) that's backing up a vSphere environment. Site B has a server (veeam02) that's the target for Backup Copy jobs from Site A. Veeam01 and Veeam02 are each functioning as Proxy, Repository, and WAN Accelerator for each site. VeeamONE and Veeam B&R are all at current patch levels (8.0 + Patch 1). All three servers (VeeamONE, veeam01, and veeam02) are Windows 2008 R2.

The issue: Our VeeamONE server sends approximately 190 queries (sometimes a few more, sometimes a few less) to DNS, every single minute, to resolve the IP of Veeam01. It also sends approximately 100 queries, every minute, to resolve the IP of Veeam02. This happens regardless of there being entries in the VeeamONE server's local DNS cache. Originally, I thought the flood of queries was happening only when the TTL for the cached lookup expired on the VeeamONE host, but I pulled my DNS logs into Splunk to drill into the data and have proven that to be false (and I just shared that last piece of info with Support today).

I've run a packet capture on the VeeamONE host to take a look at the traffic and confirm that responses from our DNS server are happening in a timely manner, and the host does not continue to send requests because it's not getting a reply. Looking at the DNS traffic in the pcap, each request is followed immediately by the response (so, in chronological order it is always Request, Response, Request, Response, etc.).

If I stop the Veeam ONE Monitor Server service, the queries stop. As soon as the service is started again, the flood of 300ish queries per minute starts again.

This isn't causing any particular problem in my environment. It's not creating a DoS condition on the DNS server or anything like that. But it is certainly not normal for an application to be doing this.

If anyone reading this logs DNS requests in their environment, can you check to see if your VeeamONE server is also exhibiting the same problem? Or if you're not logging DNS requests, but have some time to kill and feel like running a packet capture on the VeeamONE server to look at the traffic.

Support has told me that they don't see the same behavior in their environment.

Thank you,
Colin

[Vitaliy S.]

Hi Colin,

Let me ask our QC team to check their installations, as the behavior you observe is definitely not expected.

Thanks!

[EdwardsCP]

Vitaly - any chance you've heard anything from QC?

No one else out there can quickly check their environment?

[Vitaliy S.]

It seems like our dev team has found the possible reason for these extensive queries and is working on the pilot fix for our next major update. Also I believe our support team should be able to confirm this fix in your infrastructure, but to be on the safe side I would suggest double-checking it with your support engineer directly.

I see that you're still running version 7 in your infrastructure, do you have any reason not to upgrade to Veeam ONE v8 U1?

[EdwardsCP]

We were on 7 when the ticket was opened. It has since been updated to 8 patch 1, and that info was provided to support.

[Vitaliy S.]

Got it! Didn't see it in the notes of the ticket. Thanks!

[jdrch]

We were on 7 when the ticket was opened. It has since been updated to 8 patch 1, and that info was provided to support.

Did you ever resolve this? Check to ensure your DHCP and DNS server aren't forwarding hostname lookups to your router (assuming they are all separate machines.) Doing so causes a loop.

[EdwardsCP]

Yes, it was resolved about a year after the problem was reported. Veeam QA eventually confirmed this behavior with Veeam One installed on 2008r2 where it sends tons of WMI requests that are not using DNS cache. Resolution was to migrate to a newer version of Windows Server. When we moved Veeam One to 2012R2, the problem did not occur anymore.