Discussions specific to the VMware vSphere hypervisor
Post Reply
AndreZEUS
Novice
Posts: 7
Liked: never
Joined: May 10, 2015 1:22 pm
Contact:

Flapping backupspeed B&R V8 with ESXi5.0.0

Post by AndreZEUS » May 14, 2015 7:02 pm

Hello,

i have some trouble with my Veeam Backup (newest Version).
Support case ID: 00918380 (just opend 1min ago)

We use Veeam B&R 8 P2.
My issue: Flapping Backup Speed.
Backup starts with 90-100 MB/s till 7gb transferred (within 3min). Then speed breaks to 3 MB/s for about 20minutes. Then speed goes to 10-15MB/s till the end. In the period when the speed is between 0-4% i sometimes cant take an action on the Backup Reopsitory Server. i just can click in the Task Manager, but not in the explorer. Is it an indication of an issue on that server? But this is not always. Most time i can click in the explorer.
Get the same issue with all VMs.
Tried storage optimization local, lan and wan target. no difference.
Proxy Mode is Hot Add, 15 TCP connections allowed (dont make a different if 5-15, tested...).

My infrastruktur:
Veeam B&R as a VM (6 Cores, 8GB RAM) on a ESXi5.0.0 Host
Proxy on a VM (3 Cores, 4GB RAM) on the same ESXi5.0.0 Host (monitored; its enough ram and cpu)
Backup Repository is a Win 2008 R2 physikal Server (4 Cores with HyperThreading, so 8 Cores, 12GB RAM) connectet via 1gb Network. Its configured as a CIFS.
Second issue is: if i configure my Backup Repository as a Windows Type, the backup starts with 90-100mbit till ~7gb, then it breaks to 0 Kb/s. No error is shown.
The Backup just work on CIFS for me.

Third issue i solved by myself (research on internet):
At the end i got a Red X on "Hard disk 1 (40,0 GB) 36,0 GB read at 34 MB/s [CBT]"
Then:"Error: Der angegebene Netzwerkname ist nicht mehr verfügbar. Error code: 64
Failed to flush file buffers. File: [\\172.21.48.68\E$\AD-DFS\Fileserver\Fileserver2015-05-14T164628.vbk].
Failed to download disk.End of file.Failed to upload disk. Agent failed to process method {DataTransfer.SyncDisk}.
"

My changes so far:
- disabled IPv4 Checksum Offload and TCP Checksum Offload an the NICs from my Proxy, Backup Repository and Veeam B&R Server.
- Created the SESSTIMEOUT (DWORD) 60000 on my Proxy, Backup Repository and Veeam B&R Server
- ExtendedSessTimeout to 3600 seconds
- SServersWithExtendedSessTimeout
- Create the NetUseShareAccess (DWORD) registry value under HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication\ and set the value in to 1

What you suggest to do next?
Thanks in advance for your help.

foggy
Veeam Software
Posts: 18295
Liked: 1569 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Flapping backupspeed B&R V8 with ESXi5.0.0

Post by foggy » May 15, 2015 4:09 pm

Have you checked the amount of cached memory during the job run? Windows cache memory management might be the problem.

emachabert
Veeam Vanguard
Posts: 373
Liked: 165 times
Joined: Nov 17, 2010 11:42 am
Full Name: Eric Machabert
Location: France
Contact:

Re: Flapping backupspeed B&R V8 with ESXi5.0.0

Post by emachabert » May 15, 2015 8:55 pm

Are the crenels regular ? It could be the result of flow control being triggered in the network switches.
Veeamizing your IT since 2009/ Vanguard 2015,2016,2017,2018,2019

AndreZEUS
Novice
Posts: 7
Liked: never
Joined: May 10, 2015 1:22 pm
Contact:

Re: Flapping backupspeed B&R V8 with ESXi5.0.0

Post by AndreZEUS » May 16, 2015 10:55 am

Hello again,

i tested and tested and tested.


My Backup Repository in details:
Intel Server, 4 Cores with HT (so 8 Cores), 12GB RAM
RAID Controller: MegaRaid SAS 8708EM2 with newest Firmware, newest drivers on the Server.
Raid on Detail:
System Raid; 2 HDDs in RAID 1 with 250Gb for the System (2 HDDs, each with 250GB)
Storage Raid; 4 HDDs in RAID 5 with 4090Gb (4 HDDs, each with 1,5TB)

i made a chkdsk on system raid. Windows found some "not in use index entrys" and corrected them. Think this is no big deal.
Then i made a chkdsk on storage raid. Windows found no failures ot "not in use index entrys".

Now i make a consistenc check on Storage Raid with a duration of round about 30h, (startet 2h ago). So i will post the Result tomorrow.

I made a new backup repository to the system raid and backuped a small VM. It worked with fullspeed (read ~100MB/s, write 45-50MB/s).
So i guess there is something with the storage raid.

I even was able to temporarly make a new backup repository aiming on my external HDD of my notebook. I backuped our Fileserver without problems ( transferred 637Gb in 3h45min).

@emachabert
The crenels are regular when i try my Storage Backup Repository. I updated my onboard intel network chip drivers to the newest version. didnt help. i build a teaming between the managment networkport and the regular networkport, also didnt help. i dont know if i should investigate the posibility of the flow control thing on the network swtiches, cause on other backup repositorys it work. What do you mean?

@foggy
i dont see that the server ran out of memory. there is always an amount of 6-8GB free RAM shown on windows task manager (if you mean this). we dont make parallel jobs.

Any idears what i could do if the consisty check of the Storage raid ends without failures?

Thanks

emachabert
Veeam Vanguard
Posts: 373
Liked: 165 times
Joined: Nov 17, 2010 11:42 am
Full Name: Eric Machabert
Location: France
Contact:

Re: Flapping backupspeed B&R V8 with ESXi5.0.0

Post by emachabert » May 16, 2015 12:33 pm

Looks like you are having issues with the RAID5 array that is not able to sustain a consistent speed. Do you have cache memory installed on the controler ? Is write cache enabled for the RAID5 array ?It should.

Forget the flow control stuff for the moment, if you are a le to backup with normal speed to the system RAID, then the network is working fine.
Veeamizing your IT since 2009/ Vanguard 2015,2016,2017,2018,2019

AndreZEUS
Novice
Posts: 7
Liked: never
Joined: May 10, 2015 1:22 pm
Contact:

Re: Flapping backupspeed B&R V8 with ESXi5.0.0

Post by AndreZEUS » May 16, 2015 2:16 pm

Mhh, wouldn't be the System Raid array infected with this slow speed too, if the cache wont work?

The System array and Storage array is configured equal.
You just can configure the Controller for all arrays on our Controller. "Native command Queuing" is enabled
Flash Size: 4MB
Memory Size: 256MB
Error Count: 0

Maybe the consistency check shows us more information... still 14h

AndreZEUS
Novice
Posts: 7
Liked: never
Joined: May 10, 2015 1:22 pm
Contact:

Re: Flapping backupspeed B&R V8 with ESXi5.0.0

Post by AndreZEUS » May 17, 2015 9:52 am

Hello again,

consistency check worked so far!
I am happy now. transferspeed constant at 45-50MB/s
Bottleneck is not Network 19%, target 0%

i just have another question about vPower NFS.
I deactivated this option on my new config because i never see that working.
If i activate this, there should be data in the NfsDatastore Folder, right?
But there never was any data. Not at beginning of the backup, during or at the end. Thats not normal or?

Gostev
SVP, Product Management
Posts: 24826
Liked: 3579 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Flapping backupspeed B&R V8 with ESXi5.0.0

Post by Gostev » May 17, 2015 6:54 pm

Yes, this is normal.

alanbolte
Expert
Posts: 635
Liked: 172 times
Joined: Jun 18, 2012 8:58 pm
Full Name: Alan Bolte
Contact:

Re: Flapping backupspeed B&R V8 with ESXi5.0.0

Post by alanbolte » May 17, 2015 9:39 pm

vPower NFS is used for Instant Recovery, Surebackup, and Other-OS File Level Recovery. It isn't used during backup.

Post Reply

Who is online

Users browsing this forum: Google [Bot], SLennon and 15 guests