Testing Veeam Backup Free Edition: Ridiculously SLOW

[gsaunders]

[UPDATE] This turned out to be defective network switch issue.

We are looking to test Veeam Backup Free Edition as a possible gateway to the full version, but we are hitting such a roadblock I am not sure I could ever recommend even using this product. This could very well be due to not understanding best practices for the install so I will give as much detail as I can here.

We have 2 Dell T320 servers with 32GB RAM and 6 core processor, 2x1GB NICS teamed and tied to Dell 1GB switch. One server has 6x600GB 15k SAS drives with Perc H710P in RAID10. The second server is identical, but with 8x2TB Nearline SAS in RAID10. So there is no SAN... both units have very fast local storage.

They both have a fresh install of vSphere 5.0 Essentials latest updates. There is ONLY 1 VM installed at the moment.

As a simple test we wanted to backup that VM... and then also simply copy it from one ESXi host to the next.

I downloaded the Veeam Backup Free edition and installed to a new Optiplex 990 Quad Core unit with 4GB of RAM and 1GB NIC. So at this point we have 3 physical units (2 ESXi 5 hosts and 1 Windows 7 Desktop).

After installing the software I added the two hosts to Veeam and told it to backup to a local drive on the Windows 7 unit. It was backing up a VCSA (Vcenter Appliance) that was off and thin provisioned). It was about 5GB in size and should have backed up very quickly, but it took over 2 HOURS. It had a section called bottleneck and it showed SOURCE. Again we just did a simple VeeamZip. The processor on the Windows 7 unit barely ever moved and the memory was not taxed... so I know the client didn't have any processor issues or memory issues.

So what is going on. We have such a simple setup and fairly powerful units yet a simple backup is taking 2 hours when a simple copy of 5GB over typical GB network and typical drives would have only taken minutes.

I then attempted to do a copy where I copied the VM folder from one ESXi host to the next. It was going so slow that I decided to tell it to stop, but it has been sitting in stopping mode for the last 2 hours.

Simply put... this is so slow I could not see how I could recommend this product to someone... much less pay for the full version. BUT... This is our first look at Veeam and understand there must be something we are doing wrong or configuring wrong and look forward to your advice / recommendation.

Thanks,

Greg

[dellock6]

It had a section called bottleneck and it showed SOURCE. Again we just did a simple VeeamZip. The processor on the Windows 7 unit barely ever moved and the memory was not taxed... so I know the client didn't have any processor issues or memory issues.

For sure, if bottleneck is source, the backup server is not going to suffer on its cpu, you need to find out what problem could be in the source. For example:

when a simple copy of 5GB over typical GB network and typical drives would have only taken minutes.

Is it a supposition, or you really tested it? At least, if you find out even a file copy from the datastore to the local backup server takes so much time, it's more related to source rather than the backup server itself.

Luca.

[foggy]

Yes, the issue is clearly the source data retrieval speed. I would pay attention to the storage itself/connection to the storage.

[gsaunders]

Is it a supposition, or you really tested it? At least, if you find out even a file copy from the datastore to the local backup server takes so much time, it's more related to source rather than the backup server itself.

Luca.

My point in that statement was simply that a 5GB file copy (in the normal windows or linux world) would only take minutes and so I would NEVER expect the Veeam product to take so long. I understand the storage is VMSF-5 and this is vSphere and not a windows box, but the point is it should NOT take this long. However I will be doing more testing today.

What I am wondering is... is there something I need to be installing on the ESXi hosts from Veeam??? I see information talking about backup proxies and wondering if just installing this on a physical windows 7 box means I need to install someting on the ESXi host.

When I downloaded the product there seem to be 3 Setup files:
Veeam_B&R_Setup_x64.exe (which I installed on Windows 7 physical box.
Veeam_B&R_SearchServer_Setup_x64.exe (Didn't do anything with this)
Veeam_B&R_EnterpriseManager_Setup_x64.exe (Didn't do anything with this)

Is something needed from the 2nd and 3rd setup files above? I didn't do anything with those.

Greg

[foggy]

Installing a proxy on the VM having access to the datastore with VMs you are backing up, will allow for using hotadd source data retrieval mode, which provides better performance. However, since Veeam B&R Free does not provide distributed architecture and is limited to a single backup proxy server (installed as a part of the product), the only way to benefit from that is to install Veeam B&R Free itself on a VM.

Aside from that, even using network transport mode you could expect much better speeds, so the real issue is in something else.

[tsightler]

The simplest way to test the network is to install the vCenter client on the same Windows 7 box and use the Datastore browser to download the same VMDK from the powered off VM. We effectively use the very same method for data retrieval so if this is faster that would tell us if the network was working correctly.

[gsaunders]

Installing a proxy on the VM having access to the datastore with VMs you are backing up, will allow for using hotadd source data retrieval mode, which provides better performance. However, since Veeam B&R Free does not provide distributed architecture and is limited to a single backup proxy server (installed as a part of the product), the only way to benefit from that is to install Veeam B&R Free itself on a VM.

Aside from that, even using network transport mode you could expect much better speeds, so the real issue is in something else.

Thanks for the input...

As an additional test I switched to advanced mode and then right clicked on a folder and chose copy... then pasted the folder in second host and right now I am seeing the following: 6MB per sec transfer rate which is extremely slow.

What would you recommend using as a test (outside of Veeam) to try and isolate the problem.

I can do some IOPS tests and create some VM's to do transfer from Host1/VM to Host2/VM and see if that suffers as well. If there are some useful tools you would recommend please share.

Thanks

[foggy]

What would you recommend using as a test (outside of Veeam) to try and isolate the problem.

The first thing to do is what Tom has suggested above.

[gsaunders]

The simplest way to test the network is to install the vCenter client on the same Windows 7 box and use the Datastore browser to download the same VMDK from the powered off VM. We effectively use the very same method for data retrieval so if this is faster that would tell us if the network was working correctly.

I cancelled the Veeam file copy... which took a while to cancel. Now trying Datastore browser... download to local machine option. It seems to be going much faster... but can't tell until it completely finishes and then can do the math to determine the transfer rate.

[gsaunders]

The simplest way to test the network is to install the vCenter client on the same Windows 7 box and use the Datastore browser to download the same VMDK from the powered off VM. We effectively use the very same method for data retrieval so if this is faster that would tell us if the network was working correctly.

Interesting... while the transfer is running the vSphere client is losing connection to the ESXi host(s) and getting other strange errors. As a matter of fact... it seems like I am losing connections from other computers trying to get to internet.

I believe we may have a networking issue of some sort when transferring a large amount of data. As soon as I cancelled the copy all of the computers that could not get to internet could get back to the internet and I could then connect back to esxi hosts via vSphere.

I think at this point I need to isolate the physical network switch because the copy should NOT saturate the network causing other computers to not be able to communicate and should not cause vSphere to lose connection to hosts.

This is a small business so I'll replace the GB switch with a known good one to see if that resolves the issues... if so then I have my culprit. I know the wiring is good since I have tested all if it prior to this with high-end network testing equipment and it all past.

Thanks for the help and I'll update what I find.

Greg

[Gostev]

Might be a better idea to install Veeam into a VM. This way, VeeamZIP will leverage hot add processing mode for all VMs on the same host, which will provide significantly better performance than network processing mode (the only possible mode when you are not using shared storage, and Veeam is installed on a physical server - which, if I understand correctly, is your case).

[gsaunders]

Interesting... while the transfer is running the vSphere client is losing connection to the ESXi host(s) and getting other strange errors. As a matter of fact... it seems like I am losing connections from other computers trying to get to internet.

I believe we may have a networking issue of some sort when transferring a large amount of data. As soon as I cancelled the copy all of the computers that could not get to internet could get back to the internet and I could then connect back to esxi hosts via vSphere.

I think at this point I need to isolate the physical network switch because the copy should NOT saturate the network causing other computers to not be able to communicate and should not cause vSphere to lose connection to hosts.

This is a small business so I'll replace the GB switch with a known good one to see if that resolves the issues... if so then I have my culprit. I know the wiring is good since I have tested all if it prior to this with high-end network testing equipment and it all past.

Ok... I put in a temp switch and now the file copy transfer hit 80MB/s vs 4MB/s in the current configuration of Veeam on a physical desktop. So now I know I have a switch issue I will go ahead and get a new switch.

So now that leads me to some questions. I have read and as stated by Gostev and others that it may be better to Veeam into a VM to leverage hot add processing mode. This leads to an additional question...

I have two hosts. 1st host will run primary servers. 1nd host will run secondary things like 2nd DC and will also run whatever software backup solution I go with. So let's say it is a 2008R2 server with Veeam. Does simply housing Veeam in VM on second host allow for maximum throughput from 1st host for backup or does something need to be also installed on 1st host to get maximum benefit. I have a lot of reading to do!

[Gostev]

With local storage and free edition, to achieve hot add processing, you will need one Veeam install per host, and process VMs on each host with the corresponding install.

However, full version of our product allows for deploying backup proxy servers on your existing Windows VMs instead of relying on the default proxy only, so you will not have to manage two separate installs - just one backup server (which can be left physical in this case), and one backup proxy per host with local storage.

[linux4guru]

Just curious, what type of network switch are you using? Does it have the buffer size to handle large data. We saw this in our VM environment when we rolled out VMware on Cisco 2960. In the end, we ended up going with the Cisco 6506 and Nexus 5k and Nexus 2k switches. That fixed our network issue with VM dropping off the network.

Ok... I put in a temp switch and now the file copy transfer hit 80MB/s vs 4MB/s in the current configuration of Veeam on a physical desktop. So now I know I have a switch issue I will go ahead and get a new switch.

So now that leads me to some questions. I have read and as stated by Gostev and others that it may be better to Veeam into a VM to leverage hot add processing mode. This leads to an additional question...

I have two hosts. 1st host will run primary servers. 1nd host will run secondary things like 2nd DC and will also run whatever software backup solution I go with. So let's say it is a 2008R2 server with Veeam. Does simply housing Veeam in VM on second host allow for maximum throughput from 1st host for backup or does something need to be also installed on 1st host to get maximum benefit. I have a lot of reading to do!

[gsaunders]

So now that leads me to some questions. I have read and as stated by Gostev and others that it may be better to Veeam into a VM to leverage hot add processing mode. This leads to an additional question...

I have two hosts. 1st host will run primary servers. 1nd host will run secondary things like 2nd DC and will also run whatever software backup solution I go with. So let's say it is a 2008R2 server with Veeam. Does simply housing Veeam in VM on second host allow for maximum throughput from 1st host for backup or does something need to be also installed on 1st host to get maximum benefit. I have a lot of reading to do!

Linux4guru,

The old switch was a DLINK DGS-1248T. It is a basic lower end switch, but it should have been able to handle things... at least you would have thought. Here was the specs on the unit:
ftp://ftp10.dlink.com/pdfs/products/DGS ... _ds_ca.pdf

It doesn't indicated if it is non-blocking, or wirespeed, but the buffer size does seem small.

Either way we are not going to use it any more... I put in a temp switch (Netgear Pro JGS524E) and it does have non-blocking wirespeed ports:
http://netgear.com/business/products/sw ... S524E.aspx#

Thanks,

Greg

[linux4guru]

After switching to the net gear switch, are you still seeing issue?

[gsaunders]

After switching to the net gear switch, are you still seeing issue?

I think I posted it earlier... after putting in test switch all is well. We are getting a permanent switch, but putting in the temporary switch solved everything.

Greg