Backup of windows based backup proxy always fails on 1st run

[kjstech]

I have two Server 2008 R2 VM's that are in my Backup Proxies. I don't backup the one in any job, but the other one does other things besides act as a backup proxy, so I have that in a backup job.

We have a backup job for each NFS filesystem that contains VM's. Their names are
vm1-sas
vm2-sas
vm3-sas
vnxfs1
vnxfs2

This backup proxy windows server is on vnxfs1. We have email reports go out after each job and I noticed the first attempt is always a failure and it says Error: Can't find datastore 'vm2-sas' in vm configuration.

Its not really a big deal because on the second attempt its backed up successfully (last nights duration was 0:09:16).

Just odd that it always fails on the first run with a reference to filesystem vm2-sas, which it does not even reside on (remember it lives on filesystem named vnxfs1).

Again no real huge deal because its always backed up on second run. Just strange IMO.

4 active ESXi 5.0.0, 1489271
I just updated to 7.0.0.839 now, so lets see if that fixes it. However last night I was on the version prior to that... I think 7.0.0.833 was the number before .839? Anyway lets see what happens tonight.

[kjstech]

Same error last night even with the latest patch.

Well it always works on its second attempt. Just weird.

[foggy]

Keith, the described behavior looks strange, indeed. If you'd like to discover the reasons, I recommend contacting support directly so they could examine your setup closer and review job debug logs, if required.

Btw, which proxy server is used to back this VM up and what transport mode is used? Are both the same during the first (failed) and the second (successful) job runs?

[kjstech]

Whats the quickest way to find that out? I am looking at the email reports.

The proxy servers are named: veeam and c3p0.

The server that fails on the first run is c3p0 with "Can't find datastore 'vm2-sas' in vm configuration. It only takes 23 seconds to determine this. Then it retries and gets a successful backup. Last nights statistics for this machine Size: 80 GB Read 76 GB Transfered 1.5 GB Duration 0:08:25.

So this is all over the network, backups written to cifs share which is an older server re-purposed with big drives and FreeNAS software.

So backing up the proxy itself fails on the first attempt. I'm not sure what the reference with that filesystem is for because its C drive exists on a filesystem called vnxfs1.

[foggy]

This information is available in the job statistics window if you select the particular VM to the left.

[kjstech]

It says "Using backup proxy VMware Backup Proxy for disk Hard disk 1 [hotadd]"

I checked Backup Infrastructure and vm named veeam is "VMware Backup Proxy" since host says "This server".

vm "veeam" has no ties to filesystem vm2-sas either, unless it is backing up a machine using hotadd that exists on that filesystem.

The job for filesystem vm2-sas finished at 5:24:09 AM.
The job for filesystem vnxfs1 finished with that error at 4:55:19 AM
The job for filesystem vnxfs1 retry finished successfully at 5:14:36 AM

[foggy]

Is there a chance that at the time of the first attempt to backup that proxy, it is still doing the backup of some VMs residing on the vm2-sas datastore and has some of their disks attached to it, which causes the said error message?

[kjstech]

Is there a chance that at the time of the first attempt to backup that proxy, it is still doing the backup of some VMs residing on the vm2-sas datastore and has some of their disks attached to it, which causes the said error message?

Sure is a good chance.

Job vnxfs1 where this proxy resides starts at 10 PM and it finished at 2:03 AM. It backed up 21 VM's
Job vm2-sas starts at 11 PM and finished at 2:51 AM backed up 3 machines
Job vnxfs1 (retry) for this proxy started at 2:13 AM and finished at 2:25 AM
Job vm2-sas (retry) for our mail server (oddly last night first run errord on "An existing connection was forcibly closed by the remote host) - Began 3:01 AM ended 6:33 AM.

So job vm2-sas was still running until 2:51 AM, but by 2:13 AM the retry backing up of this proxy worked. Maybe whatever backup from 2:13 to 2:51 was handled by the other proxy.

I would like to add a feature request that Veeam takes this into consideration and maybe doesn't fail the job, or determines if the proxy is busy just move that vm to the end when maybe its not busy. Or perhaps logic that allows a proxy to be backed up when it has other hotadd disks attached. Maybe it can say "Hey those are just veeam hotadd disks, I will ignore them and just capture the c: drive."

[foggy]

This was just a guess, but I need to check whether this could really be the cause of the observed behavior. Actually, I've just found another existing thread where I state that this situation is already handled by Veeam B&R, so this shouldn't be the case.

[foggy]

So yes, such cases (attempts to backup proxy server VM while it is backing up other VMs) should be handled by Veeam B&R automatically, so shouldn't cause the observed behavior. Please work with support to investigate this further. Thanks.