Discussions specific to the VMware vSphere hypervisor
luckyinfil
Enthusiast
Posts: 91
Liked: 10 times
Joined: Aug 30, 2013 8:25 pm
Contact:

Storage vMotion during backup

Post by luckyinfil » Oct 01, 2013 3:08 pm

What is the best practices for designing Veeam around datastores that are clustered with svmotion enabled? Currently, I have my Veeam backup jobs to generate the list of backups based on a VM folder. However, what I am noticing is that Veeam has a ton of issues if svmotion occurs during the backup of a VM.

What I am noticing is that the virtual backup proxy hotadds the disks (thus locking it), but if a svmotion occurs, the disks get copied onto the new datastore. Normally, the vmdks on the original datastore would be deleted after the svmotion has successfully completed, but in this case, since it is locked by the virtual appliance, they don't delete these files. What I end up with is a lot of garbage VMDK files which are not linked to any VMs within the inventory. Is there any way to avoid this? I've had to set svmotion to manual mode in order to avoid this scenario.

dellock6
Veeam Software
Posts: 5487
Liked: 1510 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Storage vMotion during backup

Post by dellock6 » Oct 01, 2013 3:22 pm

From the VDDK 5.1 release notes (libraries used by Veeam):

"To prevent orphaned virtual disk caused by Storage vMotion during backup, VDDK 5.0 added the VixDiskLib_PrepareForAccess and VixDiskLib_EndAccess functions, which disable and re-enable the RelocateVM_Task method. Storage vMotion can wait for the backup to finish, or relocate a different VM. See the Virtual Disk API Programming Guide for details."

I remeember Anton Gostev confirmed Veeam is using those API calls when running backups. This means storage vmotion is temporarely stopped when a disk is under a backup operation.

Luca.
Luca Dell'Oca
EMEA Cloud Architect @ Veeam Software

@dellock6
http://www.virtualtothecore.com/en/
vExpert 2011-2012-2013-2014-2015-2016-2017-2018
Veeam VMCE #1

luckyinfil
Enthusiast
Posts: 91
Liked: 10 times
Joined: Aug 30, 2013 8:25 pm
Contact:

Re: Storage vMotion during backup

Post by luckyinfil » Oct 01, 2013 3:37 pm

My vCenter Server is 5.1U1 and I'm on veeam 7. This behaviour you listed is not the behaviour I'm seeing. Is there any specific setting I have to configure?

dellock6
Veeam Software
Posts: 5487
Liked: 1510 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Storage vMotion during backup

Post by dellock6 » Oct 01, 2013 3:48 pm

No setting that I'm aware of, it is leveraged automatically by Veeam.

I'm not from Veeam, so just wait for some of them to jump into this, but since the behaviour is not expected, I think they would ask you to open a support ticket.

Luca.
Luca Dell'Oca
EMEA Cloud Architect @ Veeam Software

@dellock6
http://www.virtualtothecore.com/en/
vExpert 2011-2012-2013-2014-2015-2016-2017-2018
Veeam VMCE #1

v.Eremin
Veeam Software
Posts: 15140
Liked: 1141 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Storage vMotion during backup

Post by v.Eremin » Oct 01, 2013 3:51 pm

Hi, John.

First of all, it might be worth understanding whether this issue can be reproduced without Veeam presence. In order to check it, you can take a snapshot of a given VM. Then, mount its disks manually to the other VM, using SCSI Hot Add functionality. After that, migrate the first VM via vMotion and see whether disks in question are left on the original datastore or not.

Thanks.

luckyinfil
Enthusiast
Posts: 91
Liked: 10 times
Joined: Aug 30, 2013 8:25 pm
Contact:

Re: Storage vMotion during backup

Post by luckyinfil » Oct 01, 2013 4:10 pm

v.Eremin wrote:Hi, John.

First of all, it might be worth understanding whether this issue can be reproduced without Veeam presence. In order to check it, you can take a snapshot of a given VM. Then, mount its disks manually to the other VM, using SCSI Hot Add functionality. After that, migrate the first VM via vMotion and see whether disks in question are left on the original datastore or not.

Thanks.
I've just done this and can verify that the vmdks stay in both the original datastore (due to the scsi hotadd which locks the files) and the new location.

luckyinfil
Enthusiast
Posts: 91
Liked: 10 times
Joined: Aug 30, 2013 8:25 pm
Contact:

Re: Storage vMotion during backup

Post by luckyinfil » Oct 01, 2013 4:13 pm

Perhaps when veeam removes the scsi disk, it should select the "remove and delete the virtual disks?". I also noticed that when this situation occurs during the backup, the VMDKs are not removed from the proxy server. I have to manually go in to detach those disks.

v.Eremin
Veeam Software
Posts: 15140
Liked: 1141 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Storage vMotion during backup

Post by v.Eremin » Oct 01, 2013 4:41 pm

I wouldn’t expect this feature to be implemented in the foreseeable future, since under certain circumstances it might produce several dangerous issues. Thanks.

Vitaliy S.
Veeam Software
Posts: 21527
Liked: 1281 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Storage vMotion during backup

Post by Vitaliy S. » Oct 01, 2013 4:42 pm

As a potential workaround for this behavior, I would suggest using Veeam ONE "garbage files" report to scan your datastores on regular basis and remove orphaned virtual disks from the datastores.

luckyinfil
Enthusiast
Posts: 91
Liked: 10 times
Joined: Aug 30, 2013 8:25 pm
Contact:

Re: Storage vMotion during backup

Post by luckyinfil » Oct 01, 2013 4:59 pm

Vitaliy S. wrote:As a potential workaround for this behavior, I would suggest using Veeam ONE "garbage files" report to scan your datastores on regular basis and remove orphaned virtual disks from the datastores.
That's what I used to do before removing svmotion alltogether but seeing how this is a reproducable bug with Veeam/VMware, what are the next steps to resolving it?

Also,

What I'm interested is how this behaviour affects Direct Attach SAN mode.

foggy
Veeam Software
Posts: 16822
Liked: 1359 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Storage vMotion during backup

Post by foggy » Oct 02, 2013 8:42 am

John, we do indeed utilize those calls, mentioned by Luca above, that disable and re-enable the RelocateVM_Task method during backup. As the result, the corresponding Migrate command on the VM's shortcut menu in the vSphere client gets disabled for the duration of backup so that you cannot start svMotion manually and could avoid the behavior you are experiencing. May I ask you to check whether the Migrate command is disabled during Veeam backup? If it is disabled in UI but still can be called by a cluster in case of automatic VM relocation, then the next step might be contacting VMware for clarification.

foggy
Veeam Software
Posts: 16822
Liked: 1359 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Storage vMotion during backup

Post by foggy » Oct 02, 2013 8:50 am

luckyinfil wrote:What I'm interested is how this behaviour affects Direct Attach SAN mode.
Those APIs allow to disable migration tasks for the whole duration of backup operation, irrespective of the transport method. However, since there's a chance that migration still can be performed in a cluster environment, I cannot predict the effect it has on direct SAN backups.

tntteam
Enthusiast
Posts: 57
Liked: 4 times
Joined: Aug 28, 2015 12:40 pm
Full Name: tntteam
Contact:

Re: Storage vMotion during backup

Post by tntteam » Oct 20, 2016 1:30 pm

Hi there,

Sorry for diging this old topic, but I m not sure about some behavior between veeam backups and VMware storage DRS.

What you explained (very well btw) is that during a backup of a specific VM, veeam locks svmotion so SDRS will not touch it.

Okay. What about the reverse case ? What happens if veeam starts a backup of a VM that is currently under svmotion by SDRS ?

Thanks

foggy
Veeam Software
Posts: 16822
Liked: 1359 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Storage vMotion during backup

Post by foggy » Oct 20, 2016 3:05 pm 1 person likes this post

vSphere will not allow to take a snapshot of such VM, so the job will either fail or wait for svMotion completion.

Andreas Neufert
Veeam Software
Posts: 2871
Liked: 499 times
Joined: May 04, 2011 8:36 am
Full Name: @AndyandtheVMs Veeam PM
Location: Germany
Contact:

Re: Storage vMotion during backup

Post by Andreas Neufert » Oct 21, 2016 6:33 pm

Just wanted to add that sometimes customer give us too low rights on vcenter and this flag can not be set. Look at the vcenter rights document if you gave the veeam user all needed rights.

Post Reply

Who is online

Users browsing this forum: No registered users and 23 guests