Backup of VCENTER server backs up wrong VMs disk

[Frosty]

I have a really strange problem. Hoping someone can shed some light on it. I created a backup job for my vCenter server, configured to run through the ESXi host, scheduled at 5:10pm daily. I thought it was running fine. I discovered today that it wasn't.

In the backup report, under Processed Objects, I see that about 2 months ago it switched from backing up the correct VM (CBMVC40) to another VM altogether (CBMDESKTOP), and last night it changed again to yet another VM (TESTCBMDB). I never noticed that it changed until last night, mainly because the backup of both CBMVC40 and CBMDESKTOP are both 40GB C: drives, whereas TESTCBMDB is an 800GB C: drive and my backup size/duration blew out.

It just so happens that yesterday I migrated a lot of servers (vMotion) because I was patching/rebooting hosts. There is every likelihood that the first time this backup went wrong it was also related to a lot of vMotion activity and host patching/rebooting. Is this a known fault? Is it something 'special' because it is the vCenter VM?

The VM itself is fine (boots fine, runs fine, the vSphere client shows all the correct stuff). It just looks like Veeam Backup has gotten a corrupted configuration and instead of pointing to the CBMVC40 disk it now points to another VM's disk instead. I rebooted both CBMVC40 and my Veeam Backup VM, then tried running a Full Backup ... it still processed the wrong VM's disk.

My best guess is that the vMotion of CBMVC40 away from ESXi3 to another ESXi host, then the vMotion back again (because of the host patching/rebooting) changed some kind of GUID and the Veeam software no longer 'knew' which disk to back up.

[Frosty]

Confirmed:
-- I had to delete and re-create the backup job to fix this ... meaning that there's nothing wrong with the VM itself, but something wrong in the way that Veeam is identifying the resource which is to be backed up

[rbrambley]

Frosty,

That's not normal at all.

Please explain your Veeam Server config, job set up, and upgrades a little more.

Did you add vCenter to the Veeam Server? or individual ESX hosts (which you upgraded recently from what version to what version?)
Did you upgrade vCenter as well?

How did you add the VMs to the job? - VM by VM or by a parent object such as a vCenter, Host, resource pool ,folder, datastore, etc?

Did you upgrade VMs / VMware tools / hardware too?

[Frosty]

My backup server is a VM (4 vCPUs and 4GB RAM) running VBR v5.0.2.224 (64-bit).
I have added my vCenter server (CBMVC40) and my 3 x ESXi v4.1.0 build-433742 hosts (ESX1, ESX2 and ESX3) to VBR.
Yesterday I applied two patches to the ESXi hosts, requiring a reboot (so I had to vMotion Vms off the hosts first).
The problem pre-dates the application of these patches.
No, we didn't update vCenter software at all.
The single vCenter VM (CBMVC40) was added to the job on its own (we do all our backups as single VM backups, not using parent objects).
I did not upgrade any VMs yesterday, though I did upgrade VMware Tools to a more recent build number today (quite unrelated to this problem though).
No hardware upgrades.

Note that last bit: TESTCBMDB Success.
Weird!

[Vitaliy S.]

Basically, when you add standalone hosts to the backup console, VM gets identified by a unique moref number. Once you've migrated a VM then it will receive a new moref ID and will be treated as a new one, meaning that you should re-add it to the backup job. All of this is not the case with backup jobs configured through vCenter Server though, as VM moref ID is kept during migration/vMotion tasks.

We never had this issue reported before, so please open a support ticket with our technical team, there must be something else that have caused such a behavior. Aslo I've delete the log snippets, please avoid posting those as requested when you click New Topic. Thanks.

[mooreka777]

>> Basically, when you add standalone hosts to the backup console, VM gets identified by a unique moref number. Once you've migrated a VM then it will receive a new moref ID and will be treated as a new one, meaning that you should re-add it to the backup job. All of this is not the case with backup jobs configured through vCenter Server though, as VM moref ID is kept during migration/vMotion tasks.

------------------------------

I have seen this behavior with my own eyes in our environment and this was my issue. If you don't schedule jobs through vcenter odd things can happen.

-kelly

[Frosty]

Thanks ... I have opened a support ticket for this issue.

[DSmith]

Forsty, I have picked up your ticket and we can continue troubleshooting through there.