Storage vMotion during backup

VMware specific discussions

Storage vMotion during backup

Veeam Logoby luckyinfil » Tue Oct 01, 2013 3:08 pm

What is the best practices for designing Veeam around datastores that are clustered with svmotion enabled? Currently, I have my Veeam backup jobs to generate the list of backups based on a VM folder. However, what I am noticing is that Veeam has a ton of issues if svmotion occurs during the backup of a VM.

What I am noticing is that the virtual backup proxy hotadds the disks (thus locking it), but if a svmotion occurs, the disks get copied onto the new datastore. Normally, the vmdks on the original datastore would be deleted after the svmotion has successfully completed, but in this case, since it is locked by the virtual appliance, they don't delete these files. What I end up with is a lot of garbage VMDK files which are not linked to any VMs within the inventory. Is there any way to avoid this? I've had to set svmotion to manual mode in order to avoid this scenario.
luckyinfil
Enthusiast
 
Posts: 91
Liked: 10 times
Joined: Fri Aug 30, 2013 8:25 pm

Re: Storage vMotion during backup

Veeam Logoby dellock6 » Tue Oct 01, 2013 3:22 pm

From the VDDK 5.1 release notes (libraries used by Veeam):

"To prevent orphaned virtual disk caused by Storage vMotion during backup, VDDK 5.0 added the VixDiskLib_PrepareForAccess and VixDiskLib_EndAccess functions, which disable and re-enable the RelocateVM_Task method. Storage vMotion can wait for the backup to finish, or relocate a different VM. See the Virtual Disk API Programming Guide for details."

I remeember Anton Gostev confirmed Veeam is using those API calls when running backups. This means storage vmotion is temporarely stopped when a disk is under a backup operation.

Luca.
Luca Dell'Oca
EMEA Cloud Architect @ Veeam Software

@dellock6
http://www.virtualtothecore.com
vExpert 2011-2012-2013-2014-2015-2016
Veeam VMCE #1
dellock6
Veeam Software
 
Posts: 5052
Liked: 1330 times
Joined: Sun Jul 26, 2009 3:39 pm
Location: Varese, Italy
Full Name: Luca Dell'Oca

Re: Storage vMotion during backup

Veeam Logoby luckyinfil » Tue Oct 01, 2013 3:37 pm

My vCenter Server is 5.1U1 and I'm on veeam 7. This behaviour you listed is not the behaviour I'm seeing. Is there any specific setting I have to configure?
luckyinfil
Enthusiast
 
Posts: 91
Liked: 10 times
Joined: Fri Aug 30, 2013 8:25 pm

Re: Storage vMotion during backup

Veeam Logoby dellock6 » Tue Oct 01, 2013 3:48 pm

No setting that I'm aware of, it is leveraged automatically by Veeam.

I'm not from Veeam, so just wait for some of them to jump into this, but since the behaviour is not expected, I think they would ask you to open a support ticket.

Luca.
Luca Dell'Oca
EMEA Cloud Architect @ Veeam Software

@dellock6
http://www.virtualtothecore.com
vExpert 2011-2012-2013-2014-2015-2016
Veeam VMCE #1
dellock6
Veeam Software
 
Posts: 5052
Liked: 1330 times
Joined: Sun Jul 26, 2009 3:39 pm
Location: Varese, Italy
Full Name: Luca Dell'Oca

Re: Storage vMotion during backup

Veeam Logoby v.Eremin » Tue Oct 01, 2013 3:51 pm

Hi, John.

First of all, it might be worth understanding whether this issue can be reproduced without Veeam presence. In order to check it, you can take a snapshot of a given VM. Then, mount its disks manually to the other VM, using SCSI Hot Add functionality. After that, migrate the first VM via vMotion and see whether disks in question are left on the original datastore or not.

Thanks.
v.Eremin
Veeam Software
 
Posts: 13281
Liked: 970 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin

Re: Storage vMotion during backup

Veeam Logoby luckyinfil » Tue Oct 01, 2013 4:10 pm

v.Eremin wrote:Hi, John.

First of all, it might be worth understanding whether this issue can be reproduced without Veeam presence. In order to check it, you can take a snapshot of a given VM. Then, mount its disks manually to the other VM, using SCSI Hot Add functionality. After that, migrate the first VM via vMotion and see whether disks in question are left on the original datastore or not.

Thanks.


I've just done this and can verify that the vmdks stay in both the original datastore (due to the scsi hotadd which locks the files) and the new location.
luckyinfil
Enthusiast
 
Posts: 91
Liked: 10 times
Joined: Fri Aug 30, 2013 8:25 pm

Re: Storage vMotion during backup

Veeam Logoby luckyinfil » Tue Oct 01, 2013 4:13 pm

Perhaps when veeam removes the scsi disk, it should select the "remove and delete the virtual disks?". I also noticed that when this situation occurs during the backup, the VMDKs are not removed from the proxy server. I have to manually go in to detach those disks.
luckyinfil
Enthusiast
 
Posts: 91
Liked: 10 times
Joined: Fri Aug 30, 2013 8:25 pm

Re: Storage vMotion during backup

Veeam Logoby v.Eremin » Tue Oct 01, 2013 4:41 pm

I wouldn’t expect this feature to be implemented in the foreseeable future, since under certain circumstances it might produce several dangerous issues. Thanks.
v.Eremin
Veeam Software
 
Posts: 13281
Liked: 970 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin

Re: Storage vMotion during backup

Veeam Logoby Vitaliy S. » Tue Oct 01, 2013 4:42 pm

As a potential workaround for this behavior, I would suggest using Veeam ONE "garbage files" report to scan your datastores on regular basis and remove orphaned virtual disks from the datastores.
Vitaliy S.
Veeam Software
 
Posts: 19564
Liked: 1104 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Storage vMotion during backup

Veeam Logoby luckyinfil » Tue Oct 01, 2013 4:59 pm

Vitaliy S. wrote:As a potential workaround for this behavior, I would suggest using Veeam ONE "garbage files" report to scan your datastores on regular basis and remove orphaned virtual disks from the datastores.


That's what I used to do before removing svmotion alltogether but seeing how this is a reproducable bug with Veeam/VMware, what are the next steps to resolving it?

Also,

What I'm interested is how this behaviour affects Direct Attach SAN mode.
luckyinfil
Enthusiast
 
Posts: 91
Liked: 10 times
Joined: Fri Aug 30, 2013 8:25 pm

Re: Storage vMotion during backup

Veeam Logoby foggy » Wed Oct 02, 2013 8:42 am

John, we do indeed utilize those calls, mentioned by Luca above, that disable and re-enable the RelocateVM_Task method during backup. As the result, the corresponding Migrate command on the VM's shortcut menu in the vSphere client gets disabled for the duration of backup so that you cannot start svMotion manually and could avoid the behavior you are experiencing. May I ask you to check whether the Migrate command is disabled during Veeam backup? If it is disabled in UI but still can be called by a cluster in case of automatic VM relocation, then the next step might be contacting VMware for clarification.
foggy
Veeam Software
 
Posts: 14743
Liked: 1081 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Storage vMotion during backup

Veeam Logoby foggy » Wed Oct 02, 2013 8:50 am

luckyinfil wrote:What I'm interested is how this behaviour affects Direct Attach SAN mode.

Those APIs allow to disable migration tasks for the whole duration of backup operation, irrespective of the transport method. However, since there's a chance that migration still can be performed in a cluster environment, I cannot predict the effect it has on direct SAN backups.
foggy
Veeam Software
 
Posts: 14743
Liked: 1081 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Storage vMotion during backup

Veeam Logoby tntteam » Thu Oct 20, 2016 1:30 pm

Hi there,

Sorry for diging this old topic, but I m not sure about some behavior between veeam backups and VMware storage DRS.

What you explained (very well btw) is that during a backup of a specific VM, veeam locks svmotion so SDRS will not touch it.

Okay. What about the reverse case ? What happens if veeam starts a backup of a VM that is currently under svmotion by SDRS ?

Thanks
tntteam
Enthusiast
 
Posts: 46
Liked: 4 times
Joined: Fri Aug 28, 2015 12:40 pm
Full Name: tntteam

Re: Storage vMotion during backup

Veeam Logoby foggy » Thu Oct 20, 2016 3:05 pm 1 person likes this post

vSphere will not allow to take a snapshot of such VM, so the job will either fail or wait for svMotion completion.
foggy
Veeam Software
 
Posts: 14743
Liked: 1081 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Storage vMotion during backup

Veeam Logoby Andreas Neufert » Fri Oct 21, 2016 6:33 pm

Just wanted to add that sometimes customer give us too low rights on vcenter and this flag can not be set. Look at the vcenter rights document if you gave the veeam user all needed rights.
Andreas Neufert
Veeam Software
 
Posts: 2201
Liked: 360 times
Joined: Wed May 04, 2011 8:36 am
Location: Germany
Full Name: @AndyandtheVMs Veeam PM

Next

Return to VMware vSphere



Who is online

Users browsing this forum: No registered users and 13 guests