Hyper-v vs VMWare Backup Performance

[collinp]

I started looking further into why I can backup 80 VMWare vm's in 5-6 hours but 23 Hyper-v VM's take me 14 hours. Both job types show oan avg transfer rate of 60 mb/s. But what I am seeing is that on the VMWare jobs, it will list the hard drive sizes as let's say [100GB] and Read size as let's say 2GB. On the Hyper-v jobs, most of the time it is having to read the entire Hard Drive. So if the hard drive size is [100GB] the read size will also show 100GB. A few of the [100GB] hard drives showed as 84GB read size. I clicked on manage volumes on the Hyper-v hosts and verified that CBT was enabled on the Hyper-v jobs.

Is this just a difference in technologies or is something wrong with my config? If so, I can open up a support case.

[Gostev]

Hi Collin, are you comparing full or incremental backup here?

[collinp]

I am seeing this on both incremental and fulls. I think what happened is there were hyper-v snapshots applied on all the vm's. The Veeam email status usually alerts me when there are vm snapshots on vmware with a warning message, but the same alerts aren't happening with hyper-v backups.

Isn't it true that vm snapshots essentially disable CBT for that particular vm?

[Gostev]

What you are thinking about is true only with VMware (you cannot enable CBT on a VM that has snapshot). Our Hyper-V CBT can be enabled even if VM has snapshots.

[Gostev]

Try removing snapshots from one of the VMs and see if it helps... at least, it's plausible. Could be either by design, or bug (will confirm with devs next week).

[collinp]

Hyper-v snapshots seem to have no bearing on this behavior. I guess I don't know how you can have an environment where you are backing up 100's of Hyper-v VM's successfully within a night with Veeam when each backup has to read the entire .vhd. Some of the VMWare backups take seconds if nothing has changed. Whereas the Hyper-v backups take much much much longer. Yes, I am using the hardware vss provider and am running multiple jobs at the same time.

[tsightler]

You should open a case with support. It seems obvious that the Veeam Hyper-V CBT filter driver is not functioning correctly in your environment. Are you perhaps using a filesystem such as MelioFS which doesn't allow our CBT driver to function?

[tietzjd]

Collinp- I always seen this issue with Hyper-V and backups.

[collinp]

I have a ticket open for this. Rebooting the Hyper-v hosts fixes it temporarily and then CBT works on Hyper-v vm's on all jobs. Then after a few days CBT stops working for specific vm's, one at a time until CBT stops working again for the entire job. Another interesting symptom when this is happening is the dedup ratio falsely shows a high ratio of 18 to 1 when CBT isn't working correctly on the Hyper-v backups. Will let you know the outcome. CBT on the vmware backups seems to me much more reliable.

[Gostev]

Hi Collin, what is the ticket - I want to draw lead dev's attention to this issue. Thanks!

[collinp]

Gostev,

Thank you! The ticket # is 5211474. I purchased 12 veeam hyper-v cpu licenses and there is no way I am going make my backup windows without CBT working. It is sporadic and I am seeing it across different Hyper-v clusters and different underlying storage. The one Hyper-v job that CBT always worked on, finally has reverted back to not working. Sounds like others have had this issue to. Don't know how widespread it is. I also had a special hotfix installed a couple months ago for the Hyper-v host BSOD issues. Don't know if this special hotfix affected anything. I have upgraded since then.

[Gostev]

OK, so to answer the original statement of this topic - no, there is not performance difference in backup performance between VMware and Hyper-V. Likewise, all original ideas of why performance may be different (such as VM snapshot presence) were incorrect.

According to the logs, slow backup was caused by the fact that changed block tracking driver was not working, possibly because Hyper-V hosts required reboot after product update, or due to some 3rd party conflict, or something else. As logs show, drivers did start working couple of weeks later (without any changes on product side - likely, after host reboot). This particular case will be investigated further.

If you have similar issue, especially after product update - try rebooting your Hyper-V hosts first.

[collinp]

Yes, technical support had me reboot the hosts a week ago and CBT started working again for a few days but now it isn't working again. They are investigating further.

[tsightler]

OK, so to answer the original statement of this topic - no, there is not performance difference in backup performance between VMware and Hyper-V. Likewise, all original ideas of why performance may be different (such as VM snapshot presence) were incorrect.

OK, not "all original ideas" since my post specifically indicated that CBT was obviously not working in his environment and that the reason needed to be investigated.

[collinp]

Just an update on this post. It looks as though there were major improvements in the Veeam Filter driver in version 6.5. I had issues before in multiple Hyper-v environments where CBT would not work reliably on every vm. All of these problems are fixed and backups are now working perfectly! With both CBT and the Netapp hardware VSS provider working, we are now able to back up a large number of Hyper-v vm's within our backup window without any issues.

[Vitaliy S.]

Thanks for the update and for such positive feedback!