SAN Backup Slow - real slow!

VMware specific discussions

SAN Backup Slow - real slow!

Veeam Logoby samuk » Fri Nov 06, 2015 8:23 pm

Hi All,

I have configured the Backup Proxy (Physical Server) to use SAN method only for transport.

The LUN's assigned to the ESX Hosts are also assigned to the Backup Proxy.

Jobs are running but near 20MB/s - also i can see thye are using the network still. Network utilization of the Backup proxy is utalised near 4% (no other backups are running)

Any idea's - 1:26 into the job...Real Slow!

Code: Select all
06/11/2015 18:58:30 :: Queued for processing at 06/11/2015 18:58:30
06/11/2015 18:58:31 :: Required backup infrastructure resources have been assigned
06/11/2015 18:58:37 :: VM processing started at 06/11/2015 18:58:37
06/11/2015 18:58:37 :: VM size: 180.0 GB
06/11/2015 18:58:40 :: Getting VM info from vSphere
06/11/2015 18:59:00 :: VMware Tools are not up to date, some functionality may be affected
06/11/2015 18:59:11 :: Inventorying guest system
06/11/2015 18:59:14 :: Preparing guest for hot backup
06/11/2015 18:59:20 :: Creating snapshot
06/11/2015 18:59:23 :: Releasing guest
06/11/2015 18:59:41 :: Saving [3Par_Store03_OS] Con/Con.vmx
06/11/2015 18:59:44 :: Saving [3Par_Store03_OS] Con/Con.vmxf
06/11/2015 18:59:47 :: Saving [3Par_Store03_OS] Con/Con.nvram
06/11/2015 18:59:49 :: Using backup proxy VMware Backup Proxy for disk Hard disk 1 [san]
06/11/2015 18:59:53 :: Hard disk 1 (30.0 GB) 26.1 GB read at 25 MB/s [CBT]
06/11/2015 19:19:17 :: Using backup proxy VMware Backup Proxy for disk Hard disk 2 [san]
06/11/2015 19:19:26 :: Hard disk 2 (150.0 GB) 81.6 GB read at 23 MB/s [CBT]


3Par SAN - connected with 8GB Fabric.
samuk
Expert
 
Posts: 127
Liked: never
Joined: Wed Mar 18, 2009 2:15 pm
Full Name: Sam

Re: SAN Backup Slow - real slow!

Veeam Logoby emachabert » Fri Nov 06, 2015 10:41 pm

That's really really slow.

First, can you tell us what is the main bottleneck as seen by Veeam
Veeamizing your IT since 2009/ Vanguard 2015,2016,2017
emachabert
Veeam Vanguard
 
Posts: 354
Liked: 163 times
Joined: Wed Nov 17, 2010 11:42 am
Location: France
Full Name: Eric Machabert

Re: SAN Backup Slow - real slow!

Veeam Logoby samuk » Fri Nov 06, 2015 11:24 pm

Just odd...as it should be using san and not network

from on of the other jobs
Code: Select all
06/11/2015 23:14:40 :: Truncating transaction logs
06/11/2015 23:14:59 :: Swap file blocks skipped: 4.0 GB
06/11/2015 23:15:04 :: Busy: Source 93% > Proxy 12% > Network 23% > Target 37%
06/11/2015 23:15:04 :: Primary bottleneck: Source
06/11/2015 23:15:04 :: Network traffic verification detected no corrupted blocks
06/11/2015 23:15:04 :: Processing finished at 06/11/2015 23:15:04
samuk
Expert
 
Posts: 127
Liked: never
Joined: Wed Mar 18, 2009 2:15 pm
Full Name: Sam

Re: SAN Backup Slow - real slow!

Veeam Logoby emachabert » Sat Nov 07, 2015 10:51 am

Why do you say they are still using network ? The log says [SAN] not [NBD], if it had been failovering to network it should have changed SAN to NBD.

To be sure it does not failover to network, please uncheck the box in the proxy setting "failover to network". Then try again.
What kind of transport are using the other jobs ?
Veeamizing your IT since 2009/ Vanguard 2015,2016,2017
emachabert
Veeam Vanguard
 
Posts: 354
Liked: 163 times
Joined: Wed Nov 17, 2010 11:42 am
Location: France
Full Name: Eric Machabert

Re: SAN Backup Slow - real slow!

Veeam Logoby v.Eremin » Mon Nov 09, 2015 8:12 am

According to the provided information, SAN mode is being utilized, not the network one. Though, the performance seems to be very slow, indeed. I'd recommend opening a ticket with a support team for a closer investigation. Thanks.
v.Eremin
Veeam Software
 
Posts: 13296
Liked: 973 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin

Re: SAN Backup Slow - real slow!

Veeam Logoby kristal1 » Fri Dec 18, 2015 3:16 pm

any updates? Veeam is a new implementation in our environment and we experience the same slowness
kristal1
Novice
 
Posts: 3
Liked: never
Joined: Fri Dec 18, 2015 3:10 pm
Full Name: LA

Re: SAN Backup Slow - real slow!

Veeam Logoby Vitaliy S. » Fri Dec 18, 2015 3:38 pm

Please let us know a bit more details on your setup as well as provide bottleneck stats for the backup job to start the discussion. Thanks!
Vitaliy S.
Veeam Software
 
Posts: 19575
Liked: 1104 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: SAN Backup Slow - real slow!

Veeam Logoby Gostev » Fri Dec 18, 2015 3:42 pm

kristal1 wrote:any updates? Veeam is a new implementation in our environment and we experience the same slowness

As long as Veeam reports SAN mode is being utilized, it always comes down to some environment-specific issue, so there is no universal recipe. Suggest opening a support case with Veeam first to have our support review your deployment and do performance tests using industry-standard tools to demonstrate that the performance issue lies with your storage or SAN fabric. After that, you may have to engage your storage and/or fabric vendor.
Gostev
Veeam Software
 
Posts: 21396
Liked: 2350 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: SAN Backup Slow - real slow!

Veeam Logoby kristal1 » Fri Dec 18, 2015 8:15 pm

We have a virtual veeam master server running Veeam v8 update 3. It has 4 vcpus and 8 GB ram.

For our QA environment, we have 2 physical Veeam Proxy hosts with :
2x4 core xeon E5540 cpus
144 GB ram
2 Qlogic QLE2462 HBAs with 2 ports each, all connected at 4Gbps (4 connections per proxy)
HP NC523SFP 10Gb 2 port network adapter. One server configured with a 20Gb aggregate team, one using a single 10Gb to test.

VM Storage is a VNX2 Model 5600 connected to the fabric at 8Gb per port.

Data Domain target storage is a ddr4200, with a bonded lacp 10Gb connection.

Any suggestions?
kristal1
Novice
 
Posts: 3
Liked: never
Joined: Fri Dec 18, 2015 3:10 pm
Full Name: LA

Re: SAN Backup Slow - real slow!

Veeam Logoby Gostev » Sat Dec 19, 2015 1:30 am

Suggestion is in the previous post ;)
Gostev
Veeam Software
 
Posts: 21396
Liked: 2350 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland


Return to VMware vSphere



Who is online

Users browsing this forum: No registered users and 22 guests