Host-based backup of VMware vSphere VMs.
Post Reply
luckyinfil
Enthusiast
Posts: 91
Liked: 10 times
Joined: Aug 30, 2013 8:25 pm
Contact:

Storage vMotion during backup

Post by luckyinfil »

What is the best practices for designing Veeam around datastores that are clustered with svmotion enabled? Currently, I have my Veeam backup jobs to generate the list of backups based on a VM folder. However, what I am noticing is that Veeam has a ton of issues if svmotion occurs during the backup of a VM.

What I am noticing is that the virtual backup proxy hotadds the disks (thus locking it), but if a svmotion occurs, the disks get copied onto the new datastore. Normally, the vmdks on the original datastore would be deleted after the svmotion has successfully completed, but in this case, since it is locked by the virtual appliance, they don't delete these files. What I end up with is a lot of garbage VMDK files which are not linked to any VMs within the inventory. Is there any way to avoid this? I've had to set svmotion to manual mode in order to avoid this scenario.
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Storage vMotion during backup

Post by dellock6 »

From the VDDK 5.1 release notes (libraries used by Veeam):

"To prevent orphaned virtual disk caused by Storage vMotion during backup, VDDK 5.0 added the VixDiskLib_PrepareForAccess and VixDiskLib_EndAccess functions, which disable and re-enable the RelocateVM_Task method. Storage vMotion can wait for the backup to finish, or relocate a different VM. See the Virtual Disk API Programming Guide for details."

I remeember Anton Gostev confirmed Veeam is using those API calls when running backups. This means storage vmotion is temporarely stopped when a disk is under a backup operation.

Luca.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
luckyinfil
Enthusiast
Posts: 91
Liked: 10 times
Joined: Aug 30, 2013 8:25 pm
Contact:

Re: Storage vMotion during backup

Post by luckyinfil »

My vCenter Server is 5.1U1 and I'm on veeam 7. This behaviour you listed is not the behaviour I'm seeing. Is there any specific setting I have to configure?
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Storage vMotion during backup

Post by dellock6 »

No setting that I'm aware of, it is leveraged automatically by Veeam.

I'm not from Veeam, so just wait for some of them to jump into this, but since the behaviour is not expected, I think they would ask you to open a support ticket.

Luca.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Storage vMotion during backup

Post by veremin »

Hi, John.

First of all, it might be worth understanding whether this issue can be reproduced without Veeam presence. In order to check it, you can take a snapshot of a given VM. Then, mount its disks manually to the other VM, using SCSI Hot Add functionality. After that, migrate the first VM via vMotion and see whether disks in question are left on the original datastore or not.

Thanks.
luckyinfil
Enthusiast
Posts: 91
Liked: 10 times
Joined: Aug 30, 2013 8:25 pm
Contact:

Re: Storage vMotion during backup

Post by luckyinfil »

v.Eremin wrote:Hi, John.

First of all, it might be worth understanding whether this issue can be reproduced without Veeam presence. In order to check it, you can take a snapshot of a given VM. Then, mount its disks manually to the other VM, using SCSI Hot Add functionality. After that, migrate the first VM via vMotion and see whether disks in question are left on the original datastore or not.

Thanks.
I've just done this and can verify that the vmdks stay in both the original datastore (due to the scsi hotadd which locks the files) and the new location.
luckyinfil
Enthusiast
Posts: 91
Liked: 10 times
Joined: Aug 30, 2013 8:25 pm
Contact:

Re: Storage vMotion during backup

Post by luckyinfil »

Perhaps when veeam removes the scsi disk, it should select the "remove and delete the virtual disks?". I also noticed that when this situation occurs during the backup, the VMDKs are not removed from the proxy server. I have to manually go in to detach those disks.
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Storage vMotion during backup

Post by veremin »

I wouldn’t expect this feature to be implemented in the foreseeable future, since under certain circumstances it might produce several dangerous issues. Thanks.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Storage vMotion during backup

Post by Vitaliy S. »

As a potential workaround for this behavior, I would suggest using Veeam ONE "garbage files" report to scan your datastores on regular basis and remove orphaned virtual disks from the datastores.
luckyinfil
Enthusiast
Posts: 91
Liked: 10 times
Joined: Aug 30, 2013 8:25 pm
Contact:

Re: Storage vMotion during backup

Post by luckyinfil »

Vitaliy S. wrote:As a potential workaround for this behavior, I would suggest using Veeam ONE "garbage files" report to scan your datastores on regular basis and remove orphaned virtual disks from the datastores.
That's what I used to do before removing svmotion alltogether but seeing how this is a reproducable bug with Veeam/VMware, what are the next steps to resolving it?

Also,

What I'm interested is how this behaviour affects Direct Attach SAN mode.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Storage vMotion during backup

Post by foggy »

John, we do indeed utilize those calls, mentioned by Luca above, that disable and re-enable the RelocateVM_Task method during backup. As the result, the corresponding Migrate command on the VM's shortcut menu in the vSphere client gets disabled for the duration of backup so that you cannot start svMotion manually and could avoid the behavior you are experiencing. May I ask you to check whether the Migrate command is disabled during Veeam backup? If it is disabled in UI but still can be called by a cluster in case of automatic VM relocation, then the next step might be contacting VMware for clarification.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Storage vMotion during backup

Post by foggy »

luckyinfil wrote:What I'm interested is how this behaviour affects Direct Attach SAN mode.
Those APIs allow to disable migration tasks for the whole duration of backup operation, irrespective of the transport method. However, since there's a chance that migration still can be performed in a cluster environment, I cannot predict the effect it has on direct SAN backups.
tntteam
Enthusiast
Posts: 68
Liked: 5 times
Joined: Aug 28, 2015 12:40 pm
Full Name: tntteam
Contact:

Re: Storage vMotion during backup

Post by tntteam »

Hi there,

Sorry for diging this old topic, but I m not sure about some behavior between veeam backups and VMware storage DRS.

What you explained (very well btw) is that during a backup of a specific VM, veeam locks svmotion so SDRS will not touch it.

Okay. What about the reverse case ? What happens if veeam starts a backup of a VM that is currently under svmotion by SDRS ?

Thanks
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Storage vMotion during backup

Post by foggy » 1 person likes this post

vSphere will not allow to take a snapshot of such VM, so the job will either fail or wait for svMotion completion.
Andreas Neufert
VP, Product Management
Posts: 6707
Liked: 1401 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: Storage vMotion during backup

Post by Andreas Neufert »

Just wanted to add that sometimes customer give us too low rights on vcenter and this flag can not be set. Look at the vcenter rights document if you gave the veeam user all needed rights.
marius roma
Veteran
Posts: 459
Liked: 5 times
Joined: Feb 01, 2012 12:04 pm
Full Name: Mario
Contact:

[MERGED] Concurrent tasks

Post by marius roma »

I'm quite sure that the answer is "don't worry", but I'd like to get the answer from anybody more expert than myself...
If a Veeam backup and a VMware task moving a VM from a datastore to another datastore run at the same time, is there any risk for the integrity of the VM and/or of the backup?
Regards
marius
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

[MERGED] Re: Concurrent tasks

Post by dellock6 »

If you mean that a storage vmotion is happening during a Veeam backup, there is a running discussion about this topic, this post is worth lookig at.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
jjlethiers
Service Provider
Posts: 13
Liked: 1 time
Joined: Apr 11, 2016 9:55 am
Full Name: Jean-Jacques Lethiers
Contact:

Re: Storage vMotion during backup

Post by jjlethiers »

Hi all,
I got a similar question related to svMotion and replication. At the target site, I can see that replica VM being automatically added to the "VM Overrides" list with SDRS set to Manual Mode so that corresponding VMDKs cannot be moved during the replication process. Perfect!
Problem is that our Veeam Proxy appliances are also part of the SDRS cluster (set to fully automated) meaning that:
1/ Replication process starts
2/ Replica VM is added to Storage DRS "VM Overrides"
3/ Snapshot of Replica VM is created
4/ Replica VM disks are attached to Proxy VM >>> VMDKs are now "owned" by Proxy VM and CAN be svMotioned

Knowing that, I guess I should disable SDRS for all Proxy VMs to avoid any issue. But I don't really know how to do it easily as we got 4 different storage tiers and proxy VMs are hosted on only one tier.
Thus I cannot add them to the "VM overrides" list of all other tiers (unless I add a tiny disk of each tier to each proxy...).
How do you guys deal with that? Aren't there any best practices related to SDRS & Virtual Appliance mode? Thanks.

Regards,
JJ
Post Reply

Who is online

Users browsing this forum: Google [Bot], Semrush [Bot] and 81 guests