ESXi4.1U3 - B&R6.5 - Network performance decreasing

[rascho]

Hello there

I am facing alot of trouble getting my backup to work.

I have infrastructure as following:
ESXi 4.1 U3 Host with latest patches form VMware on a HP ProLiant ML350 G6, Intel Xeon E5620, 16 GB RAM, Internal SAS Storage of 540 GB (RAID 5)
-> 3 VMs running
--> Windows Server 2008 R2 as vCenter Server and VEEAM B&R Server
--> Windows Small Business Server 2008 as DC, FS, Exchange
--> Windows Server 2008 R2 as Terminalserver

The backup job is backing up all three VMs to a WD NAS connected over 1 Gbps LAN.
The order of VMs starts with the vCenter Server, SBS and finally Terminalserver. The backup of the vCenter succeeds after about 20 min at a speed of 32 MB/s.
The job continues with the SBS which has two virtual HDs. The first with a size of 70 GB ends after 22 min at a speed of 50 MB/s. The second HD with a size of 160 GB (100 GB of data) isn't able to finish as the backup speed is decreasing down to 2 MB/s. I cancelled the backup after 4 hours and 30 mins with 40 GB read.

What I notice is that during is speed is slow, also all network connection to the host and all the VMs are very slow aswell. Per example it takes 5 minute to download a 400 MB ISO from the hosts datastore. Doing a ping -t -l 65000 from the VEEAM Server to the ESXi hosts results in several ping losses. As soon I cancel the job the network connection is fine again. Even when the job is still cancelling and removing the created snapshots.

I found this thread: http://forums.veeam.com/viewtopic.php?f ... 4&start=30 which is kind of the same issue I am facing. But there it is said that it is an bug in 6.0. As I am using 6.5 I have no idea what could cause the problem. It seems like as soon as the job is running at that low speed, all network connection to the host and its VMs is reduced to 2 MB/s. The cpu usage of host, VEEAM and NAS isnt high at all times.

After cancelling the job I face this bottlenecks: Source 31% > Proxy 33% > Network 11% > Target 37% which doesnt seem pretty bad for me.

Hope somebody has some ideas for me :S

Thanks in advance!

[Vitaliy S.]

Hi Ramon,

I don't believe you have a similar issue as you say that your network connection gets fully utilized during backup job while in the topic you've referenced there was an issue with Windows system cache behavior and we have implemented a workaround to resolve that.

Do you have an opportunity to deploy a virtual proxy server to leverage HotAdd processing mode? Btw, do you also see network performance degradation while backing up the vCenter Server VM?

Thanks!

[rascho]

Hi Vitaliy

unfortunatley I dont have this opportunity as the ESXi Hosts resources are almost used and I'm afraid this will create more performance problems.

But I have intresting news. Once I run a backup the first VMs is backed up pretty quick. Although speed is decreasing even with the first VM.

BUT: When I start a backup the speed is decreasing until it comes down to zero and I have to cancel the job. Now all backups will be at that slow speed UNTIL I restart the vCenter/VEEAM server.
So the source of the issue has to be either the Windows OS, vCenter or VEEAM.

Hope this helps for further investigation.

Cheers
Ramon

[foggy]

After cancelling the job I face this bottlenecks: Source 31% > Proxy 33% > Network 11% > Target 37% which doesnt seem pretty bad for me.

It is much more interesting to look at the bottleneck stats numbers at the different stages of the job execution. Hover over the Bottleneck label in the job statistics window with the mouse to see instant bottleneck stats.

Hope this helps for further investigation.

Ramon, I would advise to perform this further investigation with our technical support involved. It's pretty obvious that issue is not on the surface and debug logs should be reviewed.