Host-based backup of Microsoft Hyper-V VMs.
Post Reply
collinp
Expert
Posts: 230
Liked: 13 times
Joined: Feb 14, 2012 8:56 pm
Full Name: Collin P
Contact:

Hyper-v vs VMWare Backup Performance

Post by collinp »

I started looking further into why I can backup 80 VMWare vm's in 5-6 hours but 23 Hyper-v VM's take me 14 hours. Both job types show oan avg transfer rate of 60 mb/s. But what I am seeing is that on the VMWare jobs, it will list the hard drive sizes as let's say [100GB] and Read size as let's say 2GB. On the Hyper-v jobs, most of the time it is having to read the entire Hard Drive. So if the hard drive size is [100GB] the read size will also show 100GB. A few of the [100GB] hard drives showed as 84GB read size. I clicked on manage volumes on the Hyper-v hosts and verified that CBT was enabled on the Hyper-v jobs.

Is this just a difference in technologies or is something wrong with my config? If so, I can open up a support case.
Gostev
Chief Product Officer
Posts: 31459
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Hyper-v vs VMWare Backup Performance

Post by Gostev »

Hi Collin, are you comparing full or incremental backup here?
collinp
Expert
Posts: 230
Liked: 13 times
Joined: Feb 14, 2012 8:56 pm
Full Name: Collin P
Contact:

Re: Hyper-v vs VMWare Backup Performance

Post by collinp »

I am seeing this on both incremental and fulls. I think what happened is there were hyper-v snapshots applied on all the vm's. The Veeam email status usually alerts me when there are vm snapshots on vmware with a warning message, but the same alerts aren't happening with hyper-v backups.

Isn't it true that vm snapshots essentially disable CBT for that particular vm?
Gostev
Chief Product Officer
Posts: 31459
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Hyper-v vs VMWare Backup Performance

Post by Gostev »

What you are thinking about is true only with VMware (you cannot enable CBT on a VM that has snapshot). Our Hyper-V CBT can be enabled even if VM has snapshots.
Gostev
Chief Product Officer
Posts: 31459
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Hyper-v vs VMWare Backup Performance

Post by Gostev »

Try removing snapshots from one of the VMs and see if it helps... at least, it's plausible. Could be either by design, or bug (will confirm with devs next week).
collinp
Expert
Posts: 230
Liked: 13 times
Joined: Feb 14, 2012 8:56 pm
Full Name: Collin P
Contact:

Re: Hyper-v vs VMWare Backup Performance

Post by collinp »

Hyper-v snapshots seem to have no bearing on this behavior. I guess I don't know how you can have an environment where you are backing up 100's of Hyper-v VM's successfully within a night with Veeam when each backup has to read the entire .vhd. Some of the VMWare backups take seconds if nothing has changed. Whereas the Hyper-v backups take much much much longer. Yes, I am using the hardware vss provider and am running multiple jobs at the same time.
tsightler
VP, Product Management
Posts: 6009
Liked: 2843 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Hyper-v vs VMWare Backup Performance

Post by tsightler »

You should open a case with support. It seems obvious that the Veeam Hyper-V CBT filter driver is not functioning correctly in your environment. Are you perhaps using a filesystem such as MelioFS which doesn't allow our CBT driver to function?
tietzjd
Influencer
Posts: 16
Liked: 2 times
Joined: Nov 09, 2011 3:00 pm
Full Name: Joe Tietz
Contact:

Re: Hyper-v vs VMWare Backup Performance

Post by tietzjd »

Collinp- I always seen this issue with Hyper-V and backups.
collinp
Expert
Posts: 230
Liked: 13 times
Joined: Feb 14, 2012 8:56 pm
Full Name: Collin P
Contact:

Re: Hyper-v vs VMWare Backup Performance

Post by collinp »

I have a ticket open for this. Rebooting the Hyper-v hosts fixes it temporarily and then CBT works on Hyper-v vm's on all jobs. Then after a few days CBT stops working for specific vm's, one at a time until CBT stops working again for the entire job. Another interesting symptom when this is happening is the dedup ratio falsely shows a high ratio of 18 to 1 when CBT isn't working correctly on the Hyper-v backups. Will let you know the outcome. CBT on the vmware backups seems to me much more reliable.
Gostev
Chief Product Officer
Posts: 31459
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Hyper-v vs VMWare Backup Performance

Post by Gostev »

Hi Collin, what is the ticket - I want to draw lead dev's attention to this issue. Thanks!
collinp
Expert
Posts: 230
Liked: 13 times
Joined: Feb 14, 2012 8:56 pm
Full Name: Collin P
Contact:

Re: Hyper-v vs VMWare Backup Performance

Post by collinp »

Gostev,

Thank you! The ticket # is 5211474. I purchased 12 veeam hyper-v cpu licenses and there is no way I am going make my backup windows without CBT working. It is sporadic and I am seeing it across different Hyper-v clusters and different underlying storage. The one Hyper-v job that CBT always worked on, finally has reverted back to not working. Sounds like others have had this issue to. Don't know how widespread it is. I also had a special hotfix installed a couple months ago for the Hyper-v host BSOD issues. Don't know if this special hotfix affected anything. I have upgraded since then.
Gostev
Chief Product Officer
Posts: 31459
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Hyper-v vs VMWare Backup Performance

Post by Gostev »

OK, so to answer the original statement of this topic - no, there is not performance difference in backup performance between VMware and Hyper-V. Likewise, all original ideas of why performance may be different (such as VM snapshot presence) were incorrect.

According to the logs, slow backup was caused by the fact that changed block tracking driver was not working, possibly because Hyper-V hosts required reboot after product update, or due to some 3rd party conflict, or something else. As logs show, drivers did start working couple of weeks later (without any changes on product side - likely, after host reboot). This particular case will be investigated further.

If you have similar issue, especially after product update - try rebooting your Hyper-V hosts first.
collinp
Expert
Posts: 230
Liked: 13 times
Joined: Feb 14, 2012 8:56 pm
Full Name: Collin P
Contact:

Re: Hyper-v vs VMWare Backup Performance

Post by collinp »

Yes, technical support had me reboot the hosts a week ago and CBT started working again for a few days but now it isn't working again. They are investigating further.
tsightler
VP, Product Management
Posts: 6009
Liked: 2843 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Hyper-v vs VMWare Backup Performance

Post by tsightler »

Gostev wrote:OK, so to answer the original statement of this topic - no, there is not performance difference in backup performance between VMware and Hyper-V. Likewise, all original ideas of why performance may be different (such as VM snapshot presence) were incorrect.
OK, not "all original ideas" since my post specifically indicated that CBT was obviously not working in his environment and that the reason needed to be investigated. :mrgreen:
collinp
Expert
Posts: 230
Liked: 13 times
Joined: Feb 14, 2012 8:56 pm
Full Name: Collin P
Contact:

Re: Hyper-v vs VMWare Backup Performance

Post by collinp » 3 people like this post

Just an update on this post. It looks as though there were major improvements in the Veeam Filter driver in version 6.5. I had issues before in multiple Hyper-v environments where CBT would not work reliably on every vm. All of these problems are fixed and backups are now working perfectly! With both CBT and the Netapp hardware VSS provider working, we are now able to back up a large number of Hyper-v vm's within our backup window without any issues.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Hyper-v vs VMWare Backup Performance

Post by Vitaliy S. »

Thanks for the update and for such positive feedback!
Post Reply

Who is online

Users browsing this forum: No registered users and 19 guests