-
- Enthusiast
- Posts: 91
- Liked: 10 times
- Joined: Aug 30, 2013 8:25 pm
- Contact:
Storage vMotion during backup
What is the best practices for designing Veeam around datastores that are clustered with svmotion enabled? Currently, I have my Veeam backup jobs to generate the list of backups based on a VM folder. However, what I am noticing is that Veeam has a ton of issues if svmotion occurs during the backup of a VM.
What I am noticing is that the virtual backup proxy hotadds the disks (thus locking it), but if a svmotion occurs, the disks get copied onto the new datastore. Normally, the vmdks on the original datastore would be deleted after the svmotion has successfully completed, but in this case, since it is locked by the virtual appliance, they don't delete these files. What I end up with is a lot of garbage VMDK files which are not linked to any VMs within the inventory. Is there any way to avoid this? I've had to set svmotion to manual mode in order to avoid this scenario.
What I am noticing is that the virtual backup proxy hotadds the disks (thus locking it), but if a svmotion occurs, the disks get copied onto the new datastore. Normally, the vmdks on the original datastore would be deleted after the svmotion has successfully completed, but in this case, since it is locked by the virtual appliance, they don't delete these files. What I end up with is a lot of garbage VMDK files which are not linked to any VMs within the inventory. Is there any way to avoid this? I've had to set svmotion to manual mode in order to avoid this scenario.
-
- VeeaMVP
- Posts: 6166
- Liked: 1971 times
- Joined: Jul 26, 2009 3:39 pm
- Full Name: Luca Dell'Oca
- Location: Varese, Italy
- Contact:
Re: Storage vMotion during backup
From the VDDK 5.1 release notes (libraries used by Veeam):
"To prevent orphaned virtual disk caused by Storage vMotion during backup, VDDK 5.0 added the VixDiskLib_PrepareForAccess and VixDiskLib_EndAccess functions, which disable and re-enable the RelocateVM_Task method. Storage vMotion can wait for the backup to finish, or relocate a different VM. See the Virtual Disk API Programming Guide for details."
I remeember Anton Gostev confirmed Veeam is using those API calls when running backups. This means storage vmotion is temporarely stopped when a disk is under a backup operation.
Luca.
"To prevent orphaned virtual disk caused by Storage vMotion during backup, VDDK 5.0 added the VixDiskLib_PrepareForAccess and VixDiskLib_EndAccess functions, which disable and re-enable the RelocateVM_Task method. Storage vMotion can wait for the backup to finish, or relocate a different VM. See the Virtual Disk API Programming Guide for details."
I remeember Anton Gostev confirmed Veeam is using those API calls when running backups. This means storage vmotion is temporarely stopped when a disk is under a backup operation.
Luca.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software
@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
Principal EMEA Cloud Architect @ Veeam Software
@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
-
- Enthusiast
- Posts: 91
- Liked: 10 times
- Joined: Aug 30, 2013 8:25 pm
- Contact:
Re: Storage vMotion during backup
My vCenter Server is 5.1U1 and I'm on veeam 7. This behaviour you listed is not the behaviour I'm seeing. Is there any specific setting I have to configure?
-
- VeeaMVP
- Posts: 6166
- Liked: 1971 times
- Joined: Jul 26, 2009 3:39 pm
- Full Name: Luca Dell'Oca
- Location: Varese, Italy
- Contact:
Re: Storage vMotion during backup
No setting that I'm aware of, it is leveraged automatically by Veeam.
I'm not from Veeam, so just wait for some of them to jump into this, but since the behaviour is not expected, I think they would ask you to open a support ticket.
Luca.
I'm not from Veeam, so just wait for some of them to jump into this, but since the behaviour is not expected, I think they would ask you to open a support ticket.
Luca.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software
@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
Principal EMEA Cloud Architect @ Veeam Software
@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
-
- Product Manager
- Posts: 20415
- Liked: 2302 times
- Joined: Oct 26, 2012 3:28 pm
- Full Name: Vladimir Eremin
- Contact:
Re: Storage vMotion during backup
Hi, John.
First of all, it might be worth understanding whether this issue can be reproduced without Veeam presence. In order to check it, you can take a snapshot of a given VM. Then, mount its disks manually to the other VM, using SCSI Hot Add functionality. After that, migrate the first VM via vMotion and see whether disks in question are left on the original datastore or not.
Thanks.
First of all, it might be worth understanding whether this issue can be reproduced without Veeam presence. In order to check it, you can take a snapshot of a given VM. Then, mount its disks manually to the other VM, using SCSI Hot Add functionality. After that, migrate the first VM via vMotion and see whether disks in question are left on the original datastore or not.
Thanks.
-
- Enthusiast
- Posts: 91
- Liked: 10 times
- Joined: Aug 30, 2013 8:25 pm
- Contact:
Re: Storage vMotion during backup
I've just done this and can verify that the vmdks stay in both the original datastore (due to the scsi hotadd which locks the files) and the new location.v.Eremin wrote:Hi, John.
First of all, it might be worth understanding whether this issue can be reproduced without Veeam presence. In order to check it, you can take a snapshot of a given VM. Then, mount its disks manually to the other VM, using SCSI Hot Add functionality. After that, migrate the first VM via vMotion and see whether disks in question are left on the original datastore or not.
Thanks.
-
- Enthusiast
- Posts: 91
- Liked: 10 times
- Joined: Aug 30, 2013 8:25 pm
- Contact:
Re: Storage vMotion during backup
Perhaps when veeam removes the scsi disk, it should select the "remove and delete the virtual disks?". I also noticed that when this situation occurs during the backup, the VMDKs are not removed from the proxy server. I have to manually go in to detach those disks.
-
- Product Manager
- Posts: 20415
- Liked: 2302 times
- Joined: Oct 26, 2012 3:28 pm
- Full Name: Vladimir Eremin
- Contact:
Re: Storage vMotion during backup
I wouldn’t expect this feature to be implemented in the foreseeable future, since under certain circumstances it might produce several dangerous issues. Thanks.
-
- VP, Product Management
- Posts: 27377
- Liked: 2800 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: Storage vMotion during backup
As a potential workaround for this behavior, I would suggest using Veeam ONE "garbage files" report to scan your datastores on regular basis and remove orphaned virtual disks from the datastores.
-
- Enthusiast
- Posts: 91
- Liked: 10 times
- Joined: Aug 30, 2013 8:25 pm
- Contact:
Re: Storage vMotion during backup
That's what I used to do before removing svmotion alltogether but seeing how this is a reproducable bug with Veeam/VMware, what are the next steps to resolving it?Vitaliy S. wrote:As a potential workaround for this behavior, I would suggest using Veeam ONE "garbage files" report to scan your datastores on regular basis and remove orphaned virtual disks from the datastores.
Also,
What I'm interested is how this behaviour affects Direct Attach SAN mode.
-
- Veeam Software
- Posts: 21139
- Liked: 2141 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: Storage vMotion during backup
John, we do indeed utilize those calls, mentioned by Luca above, that disable and re-enable the RelocateVM_Task method during backup. As the result, the corresponding Migrate command on the VM's shortcut menu in the vSphere client gets disabled for the duration of backup so that you cannot start svMotion manually and could avoid the behavior you are experiencing. May I ask you to check whether the Migrate command is disabled during Veeam backup? If it is disabled in UI but still can be called by a cluster in case of automatic VM relocation, then the next step might be contacting VMware for clarification.
-
- Veeam Software
- Posts: 21139
- Liked: 2141 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: Storage vMotion during backup
Those APIs allow to disable migration tasks for the whole duration of backup operation, irrespective of the transport method. However, since there's a chance that migration still can be performed in a cluster environment, I cannot predict the effect it has on direct SAN backups.luckyinfil wrote:What I'm interested is how this behaviour affects Direct Attach SAN mode.
-
- Enthusiast
- Posts: 68
- Liked: 5 times
- Joined: Aug 28, 2015 12:40 pm
- Full Name: tntteam
- Contact:
Re: Storage vMotion during backup
Hi there,
Sorry for diging this old topic, but I m not sure about some behavior between veeam backups and VMware storage DRS.
What you explained (very well btw) is that during a backup of a specific VM, veeam locks svmotion so SDRS will not touch it.
Okay. What about the reverse case ? What happens if veeam starts a backup of a VM that is currently under svmotion by SDRS ?
Thanks
Sorry for diging this old topic, but I m not sure about some behavior between veeam backups and VMware storage DRS.
What you explained (very well btw) is that during a backup of a specific VM, veeam locks svmotion so SDRS will not touch it.
Okay. What about the reverse case ? What happens if veeam starts a backup of a VM that is currently under svmotion by SDRS ?
Thanks
-
- Veeam Software
- Posts: 21139
- Liked: 2141 times
- Joined: Jul 11, 2011 10:22 am
- Full Name: Alexander Fogelson
- Contact:
Re: Storage vMotion during backup
vSphere will not allow to take a snapshot of such VM, so the job will either fail or wait for svMotion completion.
-
- VP, Product Management
- Posts: 7081
- Liked: 1511 times
- Joined: May 04, 2011 8:36 am
- Full Name: Andreas Neufert
- Location: Germany
- Contact:
Re: Storage vMotion during backup
Just wanted to add that sometimes customer give us too low rights on vcenter and this flag can not be set. Look at the vcenter rights document if you gave the veeam user all needed rights.
-
- Veteran
- Posts: 459
- Liked: 5 times
- Joined: Feb 01, 2012 12:04 pm
- Full Name: Mario
- Contact:
[MERGED] Concurrent tasks
I'm quite sure that the answer is "don't worry", but I'd like to get the answer from anybody more expert than myself...
If a Veeam backup and a VMware task moving a VM from a datastore to another datastore run at the same time, is there any risk for the integrity of the VM and/or of the backup?
Regards
marius
If a Veeam backup and a VMware task moving a VM from a datastore to another datastore run at the same time, is there any risk for the integrity of the VM and/or of the backup?
Regards
marius
-
- VeeaMVP
- Posts: 6166
- Liked: 1971 times
- Joined: Jul 26, 2009 3:39 pm
- Full Name: Luca Dell'Oca
- Location: Varese, Italy
- Contact:
[MERGED] Re: Concurrent tasks
If you mean that a storage vmotion is happening during a Veeam backup, there is a running discussion about this topic, this post is worth lookig at.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software
@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
Principal EMEA Cloud Architect @ Veeam Software
@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
-
- Service Provider
- Posts: 13
- Liked: 1 time
- Joined: Apr 11, 2016 9:55 am
- Full Name: Jean-Jacques Lethiers
- Contact:
Re: Storage vMotion during backup
Hi all,
I got a similar question related to svMotion and replication. At the target site, I can see that replica VM being automatically added to the "VM Overrides" list with SDRS set to Manual Mode so that corresponding VMDKs cannot be moved during the replication process. Perfect!
Problem is that our Veeam Proxy appliances are also part of the SDRS cluster (set to fully automated) meaning that:
1/ Replication process starts
2/ Replica VM is added to Storage DRS "VM Overrides"
3/ Snapshot of Replica VM is created
4/ Replica VM disks are attached to Proxy VM >>> VMDKs are now "owned" by Proxy VM and CAN be svMotioned
Knowing that, I guess I should disable SDRS for all Proxy VMs to avoid any issue. But I don't really know how to do it easily as we got 4 different storage tiers and proxy VMs are hosted on only one tier.
Thus I cannot add them to the "VM overrides" list of all other tiers (unless I add a tiny disk of each tier to each proxy...).
How do you guys deal with that? Aren't there any best practices related to SDRS & Virtual Appliance mode? Thanks.
Regards,
JJ
I got a similar question related to svMotion and replication. At the target site, I can see that replica VM being automatically added to the "VM Overrides" list with SDRS set to Manual Mode so that corresponding VMDKs cannot be moved during the replication process. Perfect!
Problem is that our Veeam Proxy appliances are also part of the SDRS cluster (set to fully automated) meaning that:
1/ Replication process starts
2/ Replica VM is added to Storage DRS "VM Overrides"
3/ Snapshot of Replica VM is created
4/ Replica VM disks are attached to Proxy VM >>> VMDKs are now "owned" by Proxy VM and CAN be svMotioned
Knowing that, I guess I should disable SDRS for all Proxy VMs to avoid any issue. But I don't really know how to do it easily as we got 4 different storage tiers and proxy VMs are hosted on only one tier.
Thus I cannot add them to the "VM overrides" list of all other tiers (unless I add a tiny disk of each tier to each proxy...).
How do you guys deal with that? Aren't there any best practices related to SDRS & Virtual Appliance mode? Thanks.
Regards,
JJ
Who is online
Users browsing this forum: AdsBot [Google] and 29 guests