Slow Backup Speed (7-30 MB/s)

[silbro]

Hi everyone

I've been using Veeam multiple times now and haven't had this problem before. This is the setup

ESXi1
|
|----------SAS---SAS enclosure
|
ESXi2

VeeamBackupServer (physical) connected with 2*1GB backing up to Synology NAS with 2*1GB

So basically we have 2 ESX servers connected to the SAS storage and the SAS storage also has an enclosure with more Disks. These are all attached via 10 or 16GB cables. The Veeam Backup server which is also the vCenter server has Win2012R2 and is connected with 2*1GB LAN cables to a switch. The ESXi have 4*1GB LAN cables to the switch. The speed is very slow when backing up with Veeam (7-30MB/s). With the old setup (same setup but different hardware, using the same NAS for backup) it was much faster! (around 120 - 240MB/s). I tried all options in the proxy. I tried incremental backups and inverted incremental. I tested the network by transferring files from VMs to the VeeamBackupServer and I got the excpected full 1GB/s speed (around 120MB/s). Copying files locally on the VM I get up to 350 MB/s, so the storage seems to be fine.

I have Veeam 8 with the latest available patch. Does anyone have any hints as to where I can search for the error? I've searched using google and in this forum and tried so many solutions, I have no more clue

Thanks for any help!
silbro

[PTide]

Hi,

Please provide some details about your environment:

1. Which host do you use as a proxy server?
2. How is your proxy connected to production storage?
3. What's shown as a bottleneck in session stats?

Thank you.

[silbro]

Hi

Thank you for your answer. Here the information:

1) I use the Veeam Backup Server as a proxy (Under Backup Infrastructure -> Backup Proxies -> I have, Name: VMware Backup Proxy, Type:VMware, Host:This server)
2) The proxy is connected with 2*1GB/s (LACP) to the switch, both ESXi Hosts with their SAS are also connected via Network (4*1GB/s LACP). Basically the Storage is connected locally with Fiber Cables to the ESXi Hosts.
3) Shown is Bottleneck: Source (Load: 99%)

Is this information sufficient or do I need to provide logfiles etc. as well? Would it be better to open a case?

Sincerely
silbro

[tok-tokkie]

Hi

We have the same slow performance in our ESX-Cluster, speed arround 10 MB/s.
We use Veeam Backup Server as proxy
Server DELL RX7300 with RAID10, the Server is connected with 2 10GBit NIC link Aggregation

the problem came first after we install ESX Update ESXi550-201512401-BG, ESXi550-201512402-BG
Have anyone the same problem?

thanks
tok-tokkie

[Vitaliy S.]

tok-tokkie, please tell us what you have as a bottleneck fro the backup job too.

[PTide]

1) I use the Veeam Backup Server as a proxy (Under Backup Infrastructure -> Backup Proxies -> I have, Name: VMware Backup Proxy, Type:VMware, Host:This server)

Currently your proxy works in nbd-mode, which considered to be the slowest. Have you considered using Hot-Add mode that involves usage of a proxy-VM?

2)both ESXi Hosts with their SAS are also connected via Network (4*1GB/s LACP). Basically the Storage is connected locally with Fiber Cables to the ESXi Hosts.

Not sure that got it right...How the production starge is connected to host? Via FC (maybe FCoE?) or via SAS connector? Please elaborate.

3) Shown is Bottleneck: Source (Load: 99%)

What NIC do you have on your ESXi management interface?

Also please provide more details on this:

same setup but different hardware,

What exactly was different?

Thank you.

[tok-tokkie]

Hi
the bottleneck, copy form Job Windows_Server 2008R2
20.01.2016 22:02:42 :: Busy: Source 93% > Proxy 4% > Network 7% > Target 0%

the Storage EMC VNX5200 is connected with 10GB we use NFS
the ESXi mgmt interface is 10GbE Server Adapter HP NC522SFP another ESXi Server Intel 82599 10 GbE Dual Port

[tok-tokkie]

I found following error from Restore Job

Code: Select all

21.01.2016 11:34:22          Starting restore job
21.01.2016 11:34:23          Locking required backup files
21.01.2016 11:35:01          Queued for processing at 21.01.2016 11:35:01
21.01.2016 11:35:06          Preparing next VM for processing
21.01.2016 11:35:01          Required backup infrastructure resources have been assigned
21.01.2016 11:35:01          No available proxies are running on ESX(i) management interface subnet. Using proxies from a different subnet, performance may be impacted.
21.01.2016 11:35:06          Using source proxy VMware Backup Proxy [nbd]
21.01.2016 11:35:09          7 files to restore (125,0 GB)
21.01.2016 11:38:32 Error    Restoring [NX4] VCSA55_restored/VCSA55.vmx
                            [b] [color=#FF0000]NFC storage connection is unavailable[/color]. Storage: [stg:192.168.x.y:/NX4_Datastore01,nfchost:ha-host,conn:esx08][/b]. Storage display name: [NX4].
                             Failed to create NFC upload stream. NFC path: [nfc://conn:esx08,nfchost:ha-host,stg:192.168.x.y:/NX4_Datastore01@VCSA55_restored/VCSA55.vmx].
                             
21.01.2016 11:38:32 Error    Restore job failed Error: NFC storage connection is unavailable. Storage: [stg:192.168.x.y:/NX4_Datastore01,nfchost:ha-host,conn:esx08]. Storage display name: [NX4].
                             Failed to create NFC upload stream. NFC path: [nfc://conn:esx08,nfchost:ha-host,stg:192.168.x.y:/NX4_Datastore01@VCSA55_restore /VCSA55.vmx].

thanks

[silbro]

Hi

I've made a rough overview of how the setup is so it's easier to follow (I didn't draw 4 cables to the switch etc. but I labeled it that way):



I haven't used hotadd yet. If it helps, then I'd consider it. I still think it should be ok with this setup.

The production storage is connected via SAS Cables (wrote that a bit wrong in my previous post)

My mgmt NIC is a normal 1GB/s network card

The difference of the old setup was the hardware. Exactly the same setup as you cann see on my rough overview, except that the servers and storage were a generation before this one. The speed of all network components were the same, hosts only had less RAM and less CPUs

EDIT:
I should mention, I also had those NFC errors until I patched Veeam. I then had another error though but eventually I ignored it because I had that error when trying to restore old backups. I didn't need to do that anymore so I forgot about it.

I use vCenter 6.0 with Veeam 8 latest patch and the esxi hosts are also version 6

thanks and sincerely
silbro

[PTide]

*that picture*

Awesome! Thank you.

I haven't used hotadd yet. If it helps, then I'd consider it. I still think it should be ok with this setup.

I think that you should give it a try, here is a brief explanation of how it works.

My mgmt NIC is a normal 1GB/s network card

With your current configuration Network Mode is utilized, you can check it if you view session logs (see those "[nbd]" letters?) which causes data to flow through 1Gb mgmt interfaces and that can be very slow.

In conclusion - it's not NAS what causes a slowdown, it's your ESXi <--> Backup Proxy link. So I suggest you to try hotadd proxy (just install an extra windows VM to both of your hosts). It might be slow when hotadding disk when the job start, but disk processing speed will be way faster. Another approach would be to place 10Gb at management interfaces.

I also had those NFC errors until I patched Veeam

Please use forum search engine as there a plenty of threads on NFS-related issues.

Thank you.

[tok-tokkie]

Hi,

i think we solved the problem, install Update 3 (kb2068) and the perfomance is good.
some things can be so easy.

thanks
tok-tokkie

[silbro]

I haven't solved the problem yet. Strangely enough the linux VMs backup at a speed of 1GB/s. I don't know why, but at least that is good

I won't be able to report back next week. As soon as I get more information about this problem I'll report back.

Until then, thanks a lot for the great help so far!
silbro

[PTide]

I haven't solved the problem yet. Strangely enough the linux VMs backup at a speed of 1GB/s.

Are you 100% positive that network utilization on ESXi mgmt interface was the same during Linux VM backup session? It might be so that some backups go faster than others due to a lesser load on mgmt interface at the time of backup session, please perform some tests, if possible.

Looking forward to hear news from you.

Thank you.

[xxstyler20xx]

Hello to everyone and thanks for your help!
First of all I want to say that I'm totally new in the use of veeam and my english is not over the top so please try to keep it simple.

A other company sets up our veeam system and now they want to see money everytime I ask them something so I decided to get into it on my own and with the help of the community .

Our VEEAM Backup and Replication Software runs on an external Server (was one of our old HP ProLiant with Windows Server2012 on it 2 Sokets with 16 cores 2Ghz and 96 GB RAM) the VM's are running on an NetApp System.

The main problem is that the daily incremental backup of our VMs are really slow.. 12 Mbit/s
the whole Job with 3.3 TB of data needs round about 2 days (6VMs, 2 same time) .. Our VEEAM Proxy is set in Network mode (1GB) and I already tested it in Hotadd mode witch gives me a speed of 17 Mbit/s
In the Job statistics it says Bottleneck: Source

Best regards

[Shestakov]

Hello and welcome to the community!
First of all note that the backup speed is measured in MBps, not Mbps.
Once your bottleneck is source, you need to either switch to the Hot-Add mode or improve the link from datastore to proxy server.
Thanks!

[xxstyler20xx]

Hot Add mode takes me to 17 MBps
for testing I installed the transport feature on one of our test VM...

Just for understanding.. The VEEAM runs on an physical server and the backup runs from the NetApp to the VEEAM server over Network..
will the backup still be written on the VEEAM in Hot-Add Mode??
for my understanding.. the backup stays on the NetApp and don't pass the network

Just seen that they have disabled CBT too...

[Shestakov]

What backup method do you use? You may try to switch to the fastest, simple incremental mode.
Data path for backup job is always: Disk > Backup proxy > Network > Backup repository > Disk regardless what transport mode is chosen.
What do you use as a backup repository?
Thanks!

[xxstyler20xx]

--What backup method do you use? --
-- You may try to switch to the fastest, simple incremental mode.---
It's a incremental (recommended) in advanced propertys synthetic full is selected one time per week.
Should I enable CBT on the Job? But then I have to delete all snapshots on the VEEAM right?

Also we have a lot of Databases on the VM's can there occour any big problems when I change this settings?

--Data path for backup job is always: Disk > Backup proxy > Network > Backup repository > Disk regardless what transport mode is chosen.--
Ok and can there occour any problems on the VM where the proxy is running.. ? It's a test server.. we don't use it very often but we do..
Now the Proxy is set to default (VEEAM Server) and transport mode is automatic

--What do you use as a backup repository?--
In the Job the "default repository" is selected which is the veeam server

Thanks!

[Shestakov]

It's a incremental (recommended) in advanced propertys synthetic full is selected one time per week.
Should I enable CBT on the Job? But then I have to delete all snapshots on the VEEAM right?

What vSphere version are you at(including patches)? CBT should speed up the backup process, but beware of CBT bug in ESXi 6.
It`s recommended not to keep snapshots. May I ask why you need them?

Ok and can there occour any problems on the VM where the proxy is running.. ? It's a test server.. we don't use it very often but we do..
Now the Proxy is set to default (VEEAM Server) and transport mode is automatic

No problems should arise on that VM. Please provide full bottleneck stats to see the wider picture.
Thanks!

[xxstyler20xx]

What vSphere version are you at(including patches)? CBT should speed up the backup process, but beware of CBT bug in ESXi 6.
It`s recommended not to keep snapshots. May I ask why you need them?

Why I need them. Just to have a backup if some users delete any files..
I am using vsphere 6.0.0 .. Have to take a closer look to it on monday (because patches)

No problems should arise on that VM. Please provide full bottleneck stats to see the wider picture.
Thanks!

Maybe a stupid question but where can I find more Information about the bottle neck ?
In the statistic window of the current job it only says Bottleneck: Source 99% Proxy 5% Network 0%

The wired thing is that I have a incremental job running since 4 am 1TB data which has a speed of 12 MBs and its still running..
Now the secound incremental job started with 3 VM's same time with a speed of 22 MBs - 30MBs
(I splitted the 6 VM's in two jobs because I thought it would be little bit faster then.

[foggy]

the backup stays on the NetApp and don't pass the network

That one confuses me, could you please elaborate on your setup? If I'm getting you right, you have NetApp as a source storage, Veeam B&R as a physical backup server utilizing the default proxy server and storing backups on the Veeam B&R itself, right?

Then, you said you've tried hotadd - have you installed a dedicated proxy VM on the source host for this and checked that it is effectively used during the job? With NetApp you can also try direct SAN access mode, typically the fastest of all the three.

Should I enable CBT on the Job? But then I have to delete all snapshots on the VEEAM right?

CBT does not delete any snapshots, enabling it should improve your overall job performance.

[Shestakov]

Why I need them. Just to have a backup if some users delete any files..

Snapshots are not backups. I`d suggest to make a copy of the primary backup to be more protected.

CBT does not delete any snapshots, enabling it should improve your overall job performance.

That`s correct, but as was mentioned you need to install the latest vSphere 1b update first to avoid problems.

[xxstyler20xx]

The problem is that a other company has set up the system and configuration.. and they want to see money now everytime i need help.. so i try to fix this on my own but I'am totally new with this.

So thanks for your help and your patience!
---Our Configuration----
We have a Cisco UCS with VMWare ESX
a NetApp where the VM's and Snapshots are located. (NFS)
The "dailyjob" incremental backup will saved on the VEEAM with is a phisycal HP ProLiant Server.

--HOTADD---
I just installed the Transport role on one of our VM's (Testserver) and then I selected in the incremental job that is should use the Testserver as proxy and disabled the "default proxy"

CBT gaves me a big boost the backup was finish in 5 hours instead of 38 hours


But for me it seems that the HP Server is the main bottleneck when I copy a file on the server (1.5GB) it takes 3min but it's a raid 5 with 7 discs and 4x1GB Network Team

[foggy]

So what are the job performance ad bottleneck stats after those changes?

[xxstyler20xx]

Bottleneck: Source
Can you tell me where I find the LOGs of the jobs?

[widmerkarl]

Hello

Below are the relative log path lists for Veeam Backup & Replication.

• - Windows 2003, XP – C:\Documents and Settings\All Users\Application Data\Veeam\Backup\
  - Windows Vista, 7 - C:\Users\All Users\Veeam\Backup\
  - Windows 2008/2008 R2/2012 - C:\ProgramData\Veeam\Backup\
  - Linux - /var/log/VeeamBackup/

Typical log files requested for a manual log compilation will be:

• - The task's folder located in the parent directory, zipped. This will contain all relevant job/task/agent data.
  - All logs from the parent directory named svc.*.log, util.*.log, as well as the VeeamBackupManager.log and VeeamShell.log

For gathering the logs you can use the support assistant:
http://www.veeam.com/kb1832

[xxstyler20xx]

Only found this..
Load: Source 99% > Proxy 5% > Network 0% > Target 0%
Primary bottleneck: Source

Source1.log
.2016 03:59:34] < 91036> cli| - 0%, workload src: 99/0/0, ntf: 0/0/0

Success 6 Start time 03:06:25 Total size 3,5 TB Backup size 132,2 GB
Warning 0 End time 07:55:38 Data read 263,1 GB Dedupe 1,0x
Error 0 Duration 4:49:13 Transferred 132,1 GB Compression 1,7x
Details
Name Status Start time End time Size Read Transferred Duration
---- Success 03:10:00 03:59:39 40,0 GB 34,0 GB 774,6 MB 0:49:39
Success 03:10:00 03:19:26 300,0 GB 2,3 GB 751,0 MB 0:09:26
Success 03:10:00 05:32:21 1,1 TB 90,1 GB 69,4 GB 2:22:21
Success 03:19:17 05:16:23 1,1 TB 33,2 GB 17,7 GB 1:57:06
Success 03:59:31 05:25:36 752,1 GB 46,5 GB 9,1 GB 1:26:04
Success 05:19:23 07:55:22 273,4 GB 57,1 GB 34,4 GB 2:35:59
With CDB Enable

[xxstyler20xx]

meant CBT
Can someone explain to me why CBT fails sometimes?

On Saturday I tested CBT for the first time and it fails for VM1 (that was no problem because it is the smallest one)
but yesterday it worked for VM1 but failed for VM2 which is the biggest one so the job is still running...

[Shestakov]

Please check this VMware KB: http://kb.vmware.com/selfservice/micros ... Id=2136854
And this Veeam topic: http://forums.veeam.com/vmware-vsphere- ... 31280.html

[silbro]

Hello everyone

So I'm back and I've invested more time trying to solve this problem. I upgraded to Veeam 9 to test if that makes a difference, then I reinstalled Veeam 8 and tested again. Same results, except! and now this makes the problem even more strange, that I started to notice it's certain VMs having this problem. I have 4 VMs I'm backing up. 3 Linux VMs and 1 Windows VM. 2 out of the 3 Linux VMs have a speed of 95MB/s while the other one drops to 3-9MB/s just like the Windows VM. I even had the backup jobs run at the same time. I've also taken the MGMT of the esxi to the 4Cables in LACP hoping to get more speed. It seems to me, that the VMs must be the problem... I compared them to each other (f.e. if they are on the same storage or have the vm tools missing and compared harddrive settings. They are all identical... I'm using vcenter 6, esxi 6... might this be a problem?