Slow Backup Speed (7-30 MB/s)

VMware specific discussions

Slow Backup Speed (7-30 MB/s)

Veeam Logoby silbro » Tue Jan 19, 2016 4:25 pm

Hi everyone

I've been using Veeam multiple times now and haven't had this problem before. This is the setup

ESXi1
|
|----------SAS---SAS enclosure
|
ESXi2

VeeamBackupServer (physical) connected with 2*1GB backing up to Synology NAS with 2*1GB

So basically we have 2 ESX servers connected to the SAS storage and the SAS storage also has an enclosure with more Disks. These are all attached via 10 or 16GB cables. The Veeam Backup server which is also the vCenter server has Win2012R2 and is connected with 2*1GB LAN cables to a switch. The ESXi have 4*1GB LAN cables to the switch. The speed is very slow when backing up with Veeam (7-30MB/s). With the old setup (same setup but different hardware, using the same NAS for backup) it was much faster! (around 120 - 240MB/s). I tried all options in the proxy. I tried incremental backups and inverted incremental. I tested the network by transferring files from VMs to the VeeamBackupServer and I got the excpected full 1GB/s speed (around 120MB/s). Copying files locally on the VM I get up to 350 MB/s, so the storage seems to be fine.

I have Veeam 8 with the latest available patch. Does anyone have any hints as to where I can search for the error? I've searched using google and in this forum and tried so many solutions, I have no more clue :(

Thanks for any help!
silbro
silbro
Novice
 
Posts: 9
Liked: never
Joined: Wed Feb 11, 2015 8:03 am
Full Name: Silvio Brossi

Re: Slow Backup Speed (7-30 MB/s)

Veeam Logoby PTide » Tue Jan 19, 2016 4:51 pm

Hi,

Please provide some details about your environment:

1. Which host do you use as a proxy server?
2. How is your proxy connected to production storage?
3. What's shown as a bottleneck in session stats?

Thank you.
PTide
Veeam Software
 
Posts: 3019
Liked: 246 times
Joined: Tue May 19, 2015 1:46 pm

Re: Slow Backup Speed (7-30 MB/s)

Veeam Logoby silbro » Wed Jan 20, 2016 10:34 am

Hi

Thank you for your answer. Here the information:

1) I use the Veeam Backup Server as a proxy (Under Backup Infrastructure -> Backup Proxies -> I have, Name: VMware Backup Proxy, Type:VMware, Host:This server)
2) The proxy is connected with 2*1GB/s (LACP) to the switch, both ESXi Hosts with their SAS are also connected via Network (4*1GB/s LACP). Basically the Storage is connected locally with Fiber Cables to the ESXi Hosts.
3) Shown is Bottleneck: Source (Load: 99%)

Is this information sufficient or do I need to provide logfiles etc. as well? Would it be better to open a case?

Sincerely
silbro
silbro
Novice
 
Posts: 9
Liked: never
Joined: Wed Feb 11, 2015 8:03 am
Full Name: Silvio Brossi

Re: Slow Backup Speed (7-30 MB/s)

Veeam Logoby tok-tokkie » Wed Jan 20, 2016 2:43 pm

Hi

We have the same slow performance in our ESX-Cluster, speed arround 10 MB/s.
We use Veeam Backup Server as proxy
Server DELL RX7300 with RAID10, the Server is connected with 2 10GBit NIC link Aggregation

the problem came first after we install ESX Update ESXi550-201512401-BG, ESXi550-201512402-BG
Have anyone the same problem?

thanks
tok-tokkie
tok-tokkie
Novice
 
Posts: 4
Liked: never
Joined: Wed Jan 20, 2016 1:45 pm

Re: Slow Backup Speed (7-30 MB/s)

Veeam Logoby Vitaliy S. » Wed Jan 20, 2016 2:54 pm

tok-tokkie, please tell us what you have as a bottleneck fro the backup job too.
Vitaliy S.
Veeam Software
 
Posts: 19558
Liked: 1102 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: Slow Backup Speed (7-30 MB/s)

Veeam Logoby PTide » Wed Jan 20, 2016 3:15 pm

1) I use the Veeam Backup Server as a proxy (Under Backup Infrastructure -> Backup Proxies -> I have, Name: VMware Backup Proxy, Type:VMware, Host:This server)
Currently your proxy works in nbd-mode, which considered to be the slowest. Have you considered using Hot-Add mode that involves usage of a proxy-VM?

2)both ESXi Hosts with their SAS are also connected via Network (4*1GB/s LACP). Basically the Storage is connected locally with Fiber Cables to the ESXi Hosts.
Not sure that got it right...How the production starge is connected to host? Via FC (maybe FCoE?) or via SAS connector? Please elaborate.

3) Shown is Bottleneck: Source (Load: 99%)
What NIC do you have on your ESXi management interface?

Also please provide more details on this:
same setup but different hardware,
What exactly was different?

Thank you.
PTide
Veeam Software
 
Posts: 3019
Liked: 246 times
Joined: Tue May 19, 2015 1:46 pm

Re: Slow Backup Speed (7-30 MB/s)

Veeam Logoby tok-tokkie » Thu Jan 21, 2016 9:55 am

Hi
the bottleneck, copy form Job Windows_Server 2008R2
20.01.2016 22:02:42 :: Busy: Source 93% > Proxy 4% > Network 7% > Target 0%

the Storage EMC VNX5200 is connected with 10GB we use NFS
the ESXi mgmt interface is 10GbE Server Adapter HP NC522SFP another ESXi Server Intel 82599 10 GbE Dual Port
tok-tokkie
Novice
 
Posts: 4
Liked: never
Joined: Wed Jan 20, 2016 1:45 pm

Re: Slow Backup Speed (7-30 MB/s)

Veeam Logoby tok-tokkie » Thu Jan 21, 2016 10:51 am

I found following error from Restore Job

Code: Select all
21.01.2016 11:34:22          Starting restore job
21.01.2016 11:34:23          Locking required backup files
21.01.2016 11:35:01          Queued for processing at 21.01.2016 11:35:01
21.01.2016 11:35:06          Preparing next VM for processing
21.01.2016 11:35:01          Required backup infrastructure resources have been assigned
21.01.2016 11:35:01          No available proxies are running on ESX(i) management interface subnet. Using proxies from a different subnet, performance may be impacted.
21.01.2016 11:35:06          Using source proxy VMware Backup Proxy [nbd]
21.01.2016 11:35:09          7 files to restore (125,0 GB)
21.01.2016 11:38:32 Error    Restoring [NX4] VCSA55_restored/VCSA55.vmx
                            [b] [color=#FF0000]NFC storage connection is unavailable[/color]. Storage: [stg:192.168.x.y:/NX4_Datastore01,nfchost:ha-host,conn:esx08][/b]. Storage display name: [NX4].
                             Failed to create NFC upload stream. NFC path: [nfc://conn:esx08,nfchost:ha-host,stg:192.168.x.y:/NX4_Datastore01@VCSA55_restored/VCSA55.vmx].
                             
21.01.2016 11:38:32 Error    Restore job failed Error: NFC storage connection is unavailable. Storage: [stg:192.168.x.y:/NX4_Datastore01,nfchost:ha-host,conn:esx08]. Storage display name: [NX4].
                             Failed to create NFC upload stream. NFC path: [nfc://conn:esx08,nfchost:ha-host,stg:192.168.x.y:/NX4_Datastore01@VCSA55_restore /VCSA55.vmx].


thanks
tok-tokkie
Novice
 
Posts: 4
Liked: never
Joined: Wed Jan 20, 2016 1:45 pm

Re: Slow Backup Speed (7-30 MB/s)

Veeam Logoby silbro » Thu Jan 21, 2016 2:05 pm

Hi

I've made a rough overview of how the setup is so it's easier to follow (I didn't draw 4 cables to the switch etc. but I labeled it that way):

Image

I haven't used hotadd yet. If it helps, then I'd consider it. I still think it should be ok with this setup.

The production storage is connected via SAS Cables (wrote that a bit wrong in my previous post)

My mgmt NIC is a normal 1GB/s network card

The difference of the old setup was the hardware. Exactly the same setup as you cann see on my rough overview, except that the servers and storage were a generation before this one. The speed of all network components were the same, hosts only had less RAM and less CPUs ;)

EDIT:
I should mention, I also had those NFC errors until I patched Veeam. I then had another error though :) but eventually I ignored it because I had that error when trying to restore old backups. I didn't need to do that anymore so I forgot about it.

I use vCenter 6.0 with Veeam 8 latest patch and the esxi hosts are also version 6

thanks and sincerely
silbro
silbro
Novice
 
Posts: 9
Liked: never
Joined: Wed Feb 11, 2015 8:03 am
Full Name: Silvio Brossi

Re: Slow Backup Speed (7-30 MB/s)

Veeam Logoby PTide » Thu Jan 21, 2016 3:04 pm

*that picture*
Awesome! Thank you.

I haven't used hotadd yet. If it helps, then I'd consider it. I still think it should be ok with this setup.
I think that you should give it a try, here is a brief explanation of how it works.

My mgmt NIC is a normal 1GB/s network card
With your current configuration Network Mode is utilized, you can check it if you view session logs (see those "[nbd]" letters?) which causes data to flow through 1Gb mgmt interfaces and that can be very slow.

In conclusion - it's not NAS what causes a slowdown, it's your ESXi <--> Backup Proxy link. So I suggest you to try hotadd proxy (just install an extra windows VM to both of your hosts). It might be slow when hotadding disk when the job start, but disk processing speed will be way faster. Another approach would be to place 10Gb at management interfaces.

I also had those NFC errors until I patched Veeam
Please use forum search engine as there a plenty of threads on NFS-related issues.

Thank you.
PTide
Veeam Software
 
Posts: 3019
Liked: 246 times
Joined: Tue May 19, 2015 1:46 pm

Re: Slow Backup Speed (7-30 MB/s)

Veeam Logoby tok-tokkie » Thu Jan 21, 2016 4:15 pm

Hi,

i think we solved the problem, install Update 3 (kb2068) and the perfomance is good.
some things can be so easy.

thanks
tok-tokkie
tok-tokkie
Novice
 
Posts: 4
Liked: never
Joined: Wed Jan 20, 2016 1:45 pm

Re: Slow Backup Speed (7-30 MB/s)

Veeam Logoby silbro » Fri Jan 22, 2016 3:24 pm

I haven't solved the problem yet. Strangely enough the linux VMs backup at a speed of 1GB/s. I don't know why, but at least that is good :)

I won't be able to report back next week. As soon as I get more information about this problem I'll report back.

Until then, thanks a lot for the great help so far!
silbro
silbro
Novice
 
Posts: 9
Liked: never
Joined: Wed Feb 11, 2015 8:03 am
Full Name: Silvio Brossi

Re: Slow Backup Speed (7-30 MB/s)

Veeam Logoby PTide » Fri Jan 22, 2016 3:46 pm

I haven't solved the problem yet. Strangely enough the linux VMs backup at a speed of 1GB/s.
Are you 100% positive that network utilization on ESXi mgmt interface was the same during Linux VM backup session? It might be so that some backups go faster than others due to a lesser load on mgmt interface at the time of backup session, please perform some tests, if possible.

Looking forward to hear news from you.

Thank you.
PTide
Veeam Software
 
Posts: 3019
Liked: 246 times
Joined: Tue May 19, 2015 1:46 pm

[MERGED] Backup Jobs really slow B&R 8.0.0.2084

Veeam Logoby xxstyler20xx » Sat Jan 23, 2016 10:26 am

Hello to everyone and thanks for your help!
First of all I want to say that I'm totally new in the use of veeam and my english is not over the top so please try to keep it simple.

A other company sets up our veeam system and now they want to see money everytime I ask them something so I decided to get into it on my own and with the help of the community .

Our VEEAM Backup and Replication Software runs on an external Server (was one of our old HP ProLiant with Windows Server2012 on it 2 Sokets with 16 cores 2Ghz and 96 GB RAM) the VM's are running on an NetApp System.

The main problem is that the daily incremental backup of our VMs are really slow.. 12 Mbit/s
the whole Job with 3.3 TB of data needs round about 2 days (6VMs, 2 same time) .. Our VEEAM Proxy is set in Network mode (1GB) and I already tested it in Hotadd mode witch gives me a speed of 17 Mbit/s
In the Job statistics it says Bottleneck: Source

Best regards
xxstyler20xx
Novice
 
Posts: 8
Liked: never
Joined: Sat Jan 23, 2016 9:56 am

Re: Slow Backup Speed (7-30 MB/s)

Veeam Logoby Shestakov » Sat Jan 23, 2016 11:14 am

Hello and welcome to the community!
First of all note that the backup speed is measured in MBps, not Mbps.
Once your bottleneck is source, you need to either switch to the Hot-Add mode or improve the link from datastore to proxy server.
Thanks!
Shestakov
Veeam Software
 
Posts: 4856
Liked: 394 times
Joined: Wed May 21, 2014 11:03 am
Location: Saint Petersburg
Full Name: Nikita Shestakov

Next

Return to VMware vSphere



Who is online

Users browsing this forum: No registered users and 19 guests