Storage vMotion during backup

[luckyinfil]

What is the best practices for designing Veeam around datastores that are clustered with svmotion enabled? Currently, I have my Veeam backup jobs to generate the list of backups based on a VM folder. However, what I am noticing is that Veeam has a ton of issues if svmotion occurs during the backup of a VM.

What I am noticing is that the virtual backup proxy hotadds the disks (thus locking it), but if a svmotion occurs, the disks get copied onto the new datastore. Normally, the vmdks on the original datastore would be deleted after the svmotion has successfully completed, but in this case, since it is locked by the virtual appliance, they don't delete these files. What I end up with is a lot of garbage VMDK files which are not linked to any VMs within the inventory. Is there any way to avoid this? I've had to set svmotion to manual mode in order to avoid this scenario.

[dellock6]

From the VDDK 5.1 release notes (libraries used by Veeam):

"To prevent orphaned virtual disk caused by Storage vMotion during backup, VDDK 5.0 added the VixDiskLib_PrepareForAccess and VixDiskLib_EndAccess functions, which disable and re-enable the RelocateVM_Task method. Storage vMotion can wait for the backup to finish, or relocate a different VM. See the Virtual Disk API Programming Guide for details."

I remeember Anton Gostev confirmed Veeam is using those API calls when running backups. This means storage vmotion is temporarely stopped when a disk is under a backup operation.

Luca.

[luckyinfil]

My vCenter Server is 5.1U1 and I'm on veeam 7. This behaviour you listed is not the behaviour I'm seeing. Is there any specific setting I have to configure?

[dellock6]

No setting that I'm aware of, it is leveraged automatically by Veeam.

I'm not from Veeam, so just wait for some of them to jump into this, but since the behaviour is not expected, I think they would ask you to open a support ticket.

Luca.

[veremin]

Hi, John.

First of all, it might be worth understanding whether this issue can be reproduced without Veeam presence. In order to check it, you can take a snapshot of a given VM. Then, mount its disks manually to the other VM, using SCSI Hot Add functionality. After that, migrate the first VM via vMotion and see whether disks in question are left on the original datastore or not.

Thanks.

[luckyinfil]

Hi, John.

First of all, it might be worth understanding whether this issue can be reproduced without Veeam presence. In order to check it, you can take a snapshot of a given VM. Then, mount its disks manually to the other VM, using SCSI Hot Add functionality. After that, migrate the first VM via vMotion and see whether disks in question are left on the original datastore or not.

Thanks.

I've just done this and can verify that the vmdks stay in both the original datastore (due to the scsi hotadd which locks the files) and the new location.

[luckyinfil]

Perhaps when veeam removes the scsi disk, it should select the "remove and delete the virtual disks?". I also noticed that when this situation occurs during the backup, the VMDKs are not removed from the proxy server. I have to manually go in to detach those disks.

[veremin]

I wouldn’t expect this feature to be implemented in the foreseeable future, since under certain circumstances it might produce several dangerous issues. Thanks.

[Vitaliy S.]

As a potential workaround for this behavior, I would suggest using Veeam ONE "garbage files" report to scan your datastores on regular basis and remove orphaned virtual disks from the datastores.

[luckyinfil]

As a potential workaround for this behavior, I would suggest using Veeam ONE "garbage files" report to scan your datastores on regular basis and remove orphaned virtual disks from the datastores.

That's what I used to do before removing svmotion alltogether but seeing how this is a reproducable bug with Veeam/VMware, what are the next steps to resolving it?

Also,

What I'm interested is how this behaviour affects Direct Attach SAN mode.

[foggy]

John, we do indeed utilize those calls, mentioned by Luca above, that disable and re-enable the RelocateVM_Task method during backup. As the result, the corresponding Migrate command on the VM's shortcut menu in the vSphere client gets disabled for the duration of backup so that you cannot start svMotion manually and could avoid the behavior you are experiencing. May I ask you to check whether the Migrate command is disabled during Veeam backup? If it is disabled in UI but still can be called by a cluster in case of automatic VM relocation, then the next step might be contacting VMware for clarification.

[foggy]

What I'm interested is how this behaviour affects Direct Attach SAN mode.

Those APIs allow to disable migration tasks for the whole duration of backup operation, irrespective of the transport method. However, since there's a chance that migration still can be performed in a cluster environment, I cannot predict the effect it has on direct SAN backups.

[tntteam]

Hi there,

Sorry for diging this old topic, but I m not sure about some behavior between veeam backups and VMware storage DRS.

What you explained (very well btw) is that during a backup of a specific VM, veeam locks svmotion so SDRS will not touch it.

Okay. What about the reverse case ? What happens if veeam starts a backup of a VM that is currently under svmotion by SDRS ?

Thanks

[foggy]

vSphere will not allow to take a snapshot of such VM, so the job will either fail or wait for svMotion completion.

[Andreas Neufert]

Just wanted to add that sometimes customer give us too low rights on vcenter and this flag can not be set. Look at the vcenter rights document if you gave the veeam user all needed rights.

[marius roma]

I'm quite sure that the answer is "don't worry", but I'd like to get the answer from anybody more expert than myself...
If a Veeam backup and a VMware task moving a VM from a datastore to another datastore run at the same time, is there any risk for the integrity of the VM and/or of the backup?
Regards
marius

[dellock6]

If you mean that a storage vmotion is happening during a Veeam backup, there is a running discussion about this topic, this post is worth lookig at.

[jjlethiers]

Hi all,
I got a similar question related to svMotion and replication. At the target site, I can see that replica VM being automatically added to the "VM Overrides" list with SDRS set to Manual Mode so that corresponding VMDKs cannot be moved during the replication process. Perfect!
Problem is that our Veeam Proxy appliances are also part of the SDRS cluster (set to fully automated) meaning that:
1/ Replication process starts
2/ Replica VM is added to Storage DRS "VM Overrides"
3/ Snapshot of Replica VM is created
4/ Replica VM disks are attached to Proxy VM >>> VMDKs are now "owned" by Proxy VM and CAN be svMotioned

Knowing that, I guess I should disable SDRS for all Proxy VMs to avoid any issue. But I don't really know how to do it easily as we got 4 different storage tiers and proxy VMs are hosted on only one tier.
Thus I cannot add them to the "VM overrides" list of all other tiers (unless I add a tiny disk of each tier to each proxy...).
How do you guys deal with that? Aren't there any best practices related to SDRS & Virtual Appliance mode? Thanks.

Regards,
JJ