SAN Backup Slow - real slow!

[samuk]

Hi All,

I have configured the Backup Proxy (Physical Server) to use SAN method only for transport.

The LUN's assigned to the ESX Hosts are also assigned to the Backup Proxy.

Jobs are running but near 20MB/s - also i can see thye are using the network still. Network utilization of the Backup proxy is utalised near 4% (no other backups are running)

Any idea's - 1:26 into the job...Real Slow!

Code: Select all

06/11/2015 18:58:30 :: Queued for processing at 06/11/2015 18:58:30 
06/11/2015 18:58:31 :: Required backup infrastructure resources have been assigned 
06/11/2015 18:58:37 :: VM processing started at 06/11/2015 18:58:37 
06/11/2015 18:58:37 :: VM size: 180.0 GB 
06/11/2015 18:58:40 :: Getting VM info from vSphere 
06/11/2015 18:59:00 :: VMware Tools are not up to date, some functionality may be affected 
06/11/2015 18:59:11 :: Inventorying guest system 
06/11/2015 18:59:14 :: Preparing guest for hot backup 
06/11/2015 18:59:20 :: Creating snapshot 
06/11/2015 18:59:23 :: Releasing guest 
06/11/2015 18:59:41 :: Saving [3Par_Store03_OS] Con/Con.vmx 
06/11/2015 18:59:44 :: Saving [3Par_Store03_OS] Con/Con.vmxf 
06/11/2015 18:59:47 :: Saving [3Par_Store03_OS] Con/Con.nvram 
06/11/2015 18:59:49 :: Using backup proxy VMware Backup Proxy for disk Hard disk 1 [san] 
06/11/2015 18:59:53 :: Hard disk 1 (30.0 GB) 26.1 GB read at 25 MB/s [CBT]
06/11/2015 19:19:17 :: Using backup proxy VMware Backup Proxy for disk Hard disk 2 [san] 
06/11/2015 19:19:26 :: Hard disk 2 (150.0 GB) 81.6 GB read at 23 MB/s [CBT]

3Par SAN - connected with 8GB Fabric.

[emachabert]

That's really really slow.

First, can you tell us what is the main bottleneck as seen by Veeam

[samuk]

Just odd...as it should be using san and not network

from on of the other jobs

Code: Select all

06/11/2015 23:14:40 :: Truncating transaction logs 
06/11/2015 23:14:59 :: Swap file blocks skipped: 4.0 GB 
06/11/2015 23:15:04 :: Busy: Source 93% > Proxy 12% > Network 23% > Target 37% 
06/11/2015 23:15:04 :: Primary bottleneck: Source 
06/11/2015 23:15:04 :: Network traffic verification detected no corrupted blocks 
06/11/2015 23:15:04 :: Processing finished at 06/11/2015 23:15:04

[emachabert]

Why do you say they are still using network ? The log says [SAN] not [NBD], if it had been failovering to network it should have changed SAN to NBD.

To be sure it does not failover to network, please uncheck the box in the proxy setting "failover to network". Then try again.
What kind of transport are using the other jobs ?

[veremin]

According to the provided information, SAN mode is being utilized, not the network one. Though, the performance seems to be very slow, indeed. I'd recommend opening a ticket with a support team for a closer investigation. Thanks.

[kristal1]

any updates? Veeam is a new implementation in our environment and we experience the same slowness

[Vitaliy S.]

Please let us know a bit more details on your setup as well as provide bottleneck stats for the backup job to start the discussion. Thanks!

[Gostev]

any updates? Veeam is a new implementation in our environment and we experience the same slowness

As long as Veeam reports SAN mode is being utilized, it always comes down to some environment-specific issue, so there is no universal recipe. Suggest opening a support case with Veeam first to have our support review your deployment and do performance tests using industry-standard tools to demonstrate that the performance issue lies with your storage or SAN fabric. After that, you may have to engage your storage and/or fabric vendor.

[kristal1]

We have a virtual veeam master server running Veeam v8 update 3. It has 4 vcpus and 8 GB ram.

For our QA environment, we have 2 physical Veeam Proxy hosts with :
2x4 core xeon E5540 cpus
144 GB ram
2 Qlogic QLE2462 HBAs with 2 ports each, all connected at 4Gbps (4 connections per proxy)
HP NC523SFP 10Gb 2 port network adapter. One server configured with a 20Gb aggregate team, one using a single 10Gb to test.

VM Storage is a VNX2 Model 5600 connected to the fabric at 8Gb per port.

Data Domain target storage is a ddr4200, with a bonded lacp 10Gb connection.

Any suggestions?

[Gostev]

Suggestion is in the previous post