Comprehensive data protection for all workloads
Post Reply
Frosty
Expert
Posts: 164
Liked: 35 times
Joined: Dec 22, 2009 9:00 pm
Full Name: Stephen Frost
Contact:

Backup of VCENTER server backs up wrong VMs disk

Post by Frosty » Aug 04, 2011 11:02 pm

I have a really strange problem. Hoping someone can shed some light on it. I created a backup job for my vCenter server, configured to run through the ESXi host, scheduled at 5:10pm daily. I thought it was running fine. I discovered today that it wasn't.

In the backup report, under Processed Objects, I see that about 2 months ago it switched from backing up the correct VM (CBMVC40) to another VM altogether (CBMDESKTOP), and last night it changed again to yet another VM (TESTCBMDB). I never noticed that it changed until last night, mainly because the backup of both CBMVC40 and CBMDESKTOP are both 40GB C: drives, whereas TESTCBMDB is an 800GB C: drive and my backup size/duration blew out.

It just so happens that yesterday I migrated a lot of servers (vMotion) because I was patching/rebooting hosts. There is every likelihood that the first time this backup went wrong it was also related to a lot of vMotion activity and host patching/rebooting. Is this a known fault? Is it something 'special' because it is the vCenter VM?

The VM itself is fine (boots fine, runs fine, the vSphere client shows all the correct stuff). It just looks like Veeam Backup has gotten a corrupted configuration and instead of pointing to the CBMVC40 disk it now points to another VM's disk instead. I rebooted both CBMVC40 and my Veeam Backup VM, then tried running a Full Backup ... it still processed the wrong VM's disk.

My best guess is that the vMotion of CBMVC40 away from ESXi3 to another ESXi host, then the vMotion back again (because of the host patching/rebooting) changed some kind of GUID and the Veeam software no longer 'knew' which disk to back up.

Frosty
Expert
Posts: 164
Liked: 35 times
Joined: Dec 22, 2009 9:00 pm
Full Name: Stephen Frost
Contact:

Re: Backup of VCENTER server backs up wrong VMs disk

Post by Frosty » Aug 04, 2011 11:47 pm

Confirmed:
-- I had to delete and re-create the backup job to fix this ... meaning that there's nothing wrong with the VM itself, but something wrong in the way that Veeam is identifying the resource which is to be backed up

rbrambley
Veeam Software
Posts: 459
Liked: 54 times
Joined: Jun 16, 2009 1:23 pm
Full Name: Rich Brambley
Contact:

Re: Backup of VCENTER server backs up wrong VMs disk

Post by rbrambley » Aug 05, 2011 12:43 am

Frosty,

That's not normal at all.

Please explain your Veeam Server config, job set up, and upgrades a little more.

Did you add vCenter to the Veeam Server? or individual ESX hosts (which you upgraded recently from what version to what version?)
Did you upgrade vCenter as well?

How did you add the VMs to the job? - VM by VM or by a parent object such as a vCenter, Host, resource pool ,folder, datastore, etc?

Did you upgrade VMs / VMware tools / hardware too?

Frosty
Expert
Posts: 164
Liked: 35 times
Joined: Dec 22, 2009 9:00 pm
Full Name: Stephen Frost
Contact:

Re: Backup of VCENTER server backs up wrong VMs disk

Post by Frosty » Aug 05, 2011 5:32 am

My backup server is a VM (4 vCPUs and 4GB RAM) running VBR v5.0.2.224 (64-bit).
I have added my vCenter server (CBMVC40) and my 3 x ESXi v4.1.0 build-433742 hosts (ESX1, ESX2 and ESX3) to VBR.
Yesterday I applied two patches to the ESXi hosts, requiring a reboot (so I had to vMotion Vms off the hosts first).
The problem pre-dates the application of these patches.
No, we didn't update vCenter software at all.
The single vCenter VM (CBMVC40) was added to the job on its own (we do all our backups as single VM backups, not using parent objects).
I did not upgrade any VMs yesterday, though I did upgrade VMware Tools to a more recent build number today (quite unrelated to this problem though).
No hardware upgrades.

Note that last bit: TESTCBMDB Success.
Weird!

Vitaliy S.
Product Manager
Posts: 22771
Liked: 1520 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Backup of VCENTER server backs up wrong VMs disk

Post by Vitaliy S. » Aug 05, 2011 7:58 am

Basically, when you add standalone hosts to the backup console, VM gets identified by a unique moref number. Once you've migrated a VM then it will receive a new moref ID and will be treated as a new one, meaning that you should re-add it to the backup job. All of this is not the case with backup jobs configured through vCenter Server though, as VM moref ID is kept during migration/vMotion tasks.

We never had this issue reported before, so please open a support ticket with our technical team, there must be something else that have caused such a behavior. Aslo I've delete the log snippets, please avoid posting those as requested when you click New Topic. Thanks.

mooreka777
Influencer
Posts: 18
Liked: 1 time
Joined: Jan 20, 2011 1:05 pm
Full Name: K Moore
Contact:

Re: Backup of VCENTER server backs up wrong VMs disk

Post by mooreka777 » Aug 07, 2011 9:27 pm

>> Basically, when you add standalone hosts to the backup console, VM gets identified by a unique moref number. Once you've migrated a VM then it will receive a new moref ID and will be treated as a new one, meaning that you should re-add it to the backup job. All of this is not the case with backup jobs configured through vCenter Server though, as VM moref ID is kept during migration/vMotion tasks.

------------------------------

I have seen this behavior with my own eyes in our environment and this was my issue. If you don't schedule jobs through vcenter odd things can happen.

-kelly

Frosty
Expert
Posts: 164
Liked: 35 times
Joined: Dec 22, 2009 9:00 pm
Full Name: Stephen Frost
Contact:

Re: Backup of VCENTER server backs up wrong VMs disk

Post by Frosty » Aug 07, 2011 10:07 pm

Thanks ... I have opened a support ticket for this issue.

DSmith
Veeam Software
Posts: 21
Liked: never
Joined: Aug 31, 2010 8:50 pm
Full Name: Dewain Smith
Contact:

Re: Backup of VCENTER server backs up wrong VMs disk

Post by DSmith » Aug 08, 2011 1:38 am

Forsty, I have picked up your ticket and we can continue troubleshooting through there.

Post Reply

Who is online

Users browsing this forum: Baidu [Spider], Bing [Bot], Dima P., Google [Bot], RubinCompServ and 29 guests