Flapping backupspeed B&R V8 with ESXi5.0.0

VMware specific discussions

Flapping backupspeed B&R V8 with ESXi5.0.0

Veeam Logoby AndreZEUS » Thu May 14, 2015 7:02 pm

Hello,

i have some trouble with my Veeam Backup (newest Version).
Support case ID: 00918380 (just opend 1min ago)

We use Veeam B&R 8 P2.
My issue: Flapping Backup Speed.
Backup starts with 90-100 MB/s till 7gb transferred (within 3min). Then speed breaks to 3 MB/s for about 20minutes. Then speed goes to 10-15MB/s till the end. In the period when the speed is between 0-4% i sometimes cant take an action on the Backup Reopsitory Server. i just can click in the Task Manager, but not in the explorer. Is it an indication of an issue on that server? But this is not always. Most time i can click in the explorer.
Get the same issue with all VMs.
Tried storage optimization local, lan and wan target. no difference.
Proxy Mode is Hot Add, 15 TCP connections allowed (dont make a different if 5-15, tested...).

My infrastruktur:
Veeam B&R as a VM (6 Cores, 8GB RAM) on a ESXi5.0.0 Host
Proxy on a VM (3 Cores, 4GB RAM) on the same ESXi5.0.0 Host (monitored; its enough ram and cpu)
Backup Repository is a Win 2008 R2 physikal Server (4 Cores with HyperThreading, so 8 Cores, 12GB RAM) connectet via 1gb Network. Its configured as a CIFS.
Second issue is: if i configure my Backup Repository as a Windows Type, the backup starts with 90-100mbit till ~7gb, then it breaks to 0 Kb/s. No error is shown.
The Backup just work on CIFS for me.

Third issue i solved by myself (research on internet):
At the end i got a Red X on "Hard disk 1 (40,0 GB) 36,0 GB read at 34 MB/s [CBT]"
Then:"Error: Der angegebene Netzwerkname ist nicht mehr verf├╝gbar. Error code: 64
Failed to flush file buffers. File: [\\172.21.48.68\E$\AD-DFS\Fileserver\Fileserver2015-05-14T164628.vbk].
Failed to download disk.End of file.Failed to upload disk. Agent failed to process method {DataTransfer.SyncDisk}.
"

My changes so far:
- disabled IPv4 Checksum Offload and TCP Checksum Offload an the NICs from my Proxy, Backup Repository and Veeam B&R Server.
- Created the SESSTIMEOUT (DWORD) 60000 on my Proxy, Backup Repository and Veeam B&R Server
- ExtendedSessTimeout to 3600 seconds
- SServersWithExtendedSessTimeout
- Create the NetUseShareAccess (DWORD) registry value under HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication\ and set the value in to 1

What you suggest to do next?
Thanks in advance for your help.
AndreZEUS
Novice
 
Posts: 7
Liked: never
Joined: Sun May 10, 2015 1:22 pm

Re: Flapping backupspeed B&R V8 with ESXi5.0.0

Veeam Logoby foggy » Fri May 15, 2015 4:09 pm

Have you checked the amount of cached memory during the job run? Windows cache memory management might be the problem.
foggy
Veeam Software
 
Posts: 14752
Liked: 1083 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Flapping backupspeed B&R V8 with ESXi5.0.0

Veeam Logoby emachabert » Fri May 15, 2015 8:55 pm

Are the crenels regular ? It could be the result of flow control being triggered in the network switches.
Veeamizing your IT since 2009/ Vanguard 2015,2016,2017
emachabert
Veeam Vanguard
 
Posts: 354
Liked: 163 times
Joined: Wed Nov 17, 2010 11:42 am
Location: France
Full Name: Eric Machabert

Re: Flapping backupspeed B&R V8 with ESXi5.0.0

Veeam Logoby AndreZEUS » Sat May 16, 2015 10:55 am

Hello again,

i tested and tested and tested.


My Backup Repository in details:
Intel Server, 4 Cores with HT (so 8 Cores), 12GB RAM
RAID Controller: MegaRaid SAS 8708EM2 with newest Firmware, newest drivers on the Server.
Raid on Detail:
System Raid; 2 HDDs in RAID 1 with 250Gb for the System (2 HDDs, each with 250GB)
Storage Raid; 4 HDDs in RAID 5 with 4090Gb (4 HDDs, each with 1,5TB)

i made a chkdsk on system raid. Windows found some "not in use index entrys" and corrected them. Think this is no big deal.
Then i made a chkdsk on storage raid. Windows found no failures ot "not in use index entrys".

Now i make a consistenc check on Storage Raid with a duration of round about 30h, (startet 2h ago). So i will post the Result tomorrow.

I made a new backup repository to the system raid and backuped a small VM. It worked with fullspeed (read ~100MB/s, write 45-50MB/s).
So i guess there is something with the storage raid.

I even was able to temporarly make a new backup repository aiming on my external HDD of my notebook. I backuped our Fileserver without problems ( transferred 637Gb in 3h45min).

@emachabert
The crenels are regular when i try my Storage Backup Repository. I updated my onboard intel network chip drivers to the newest version. didnt help. i build a teaming between the managment networkport and the regular networkport, also didnt help. i dont know if i should investigate the posibility of the flow control thing on the network swtiches, cause on other backup repositorys it work. What do you mean?

@foggy
i dont see that the server ran out of memory. there is always an amount of 6-8GB free RAM shown on windows task manager (if you mean this). we dont make parallel jobs.

Any idears what i could do if the consisty check of the Storage raid ends without failures?

Thanks
AndreZEUS
Novice
 
Posts: 7
Liked: never
Joined: Sun May 10, 2015 1:22 pm

Re: Flapping backupspeed B&R V8 with ESXi5.0.0

Veeam Logoby emachabert » Sat May 16, 2015 12:33 pm

Looks like you are having issues with the RAID5 array that is not able to sustain a consistent speed. Do you have cache memory installed on the controler ? Is write cache enabled for the RAID5 array ?It should.

Forget the flow control stuff for the moment, if you are a le to backup with normal speed to the system RAID, then the network is working fine.
Veeamizing your IT since 2009/ Vanguard 2015,2016,2017
emachabert
Veeam Vanguard
 
Posts: 354
Liked: 163 times
Joined: Wed Nov 17, 2010 11:42 am
Location: France
Full Name: Eric Machabert

Re: Flapping backupspeed B&R V8 with ESXi5.0.0

Veeam Logoby AndreZEUS » Sat May 16, 2015 2:16 pm

Mhh, wouldn't be the System Raid array infected with this slow speed too, if the cache wont work?

The System array and Storage array is configured equal.
You just can configure the Controller for all arrays on our Controller. "Native command Queuing" is enabled
Flash Size: 4MB
Memory Size: 256MB
Error Count: 0

Maybe the consistency check shows us more information... still 14h
AndreZEUS
Novice
 
Posts: 7
Liked: never
Joined: Sun May 10, 2015 1:22 pm

Re: Flapping backupspeed B&R V8 with ESXi5.0.0

Veeam Logoby AndreZEUS » Sun May 17, 2015 9:52 am

Hello again,

consistency check worked so far!
I am happy now. transferspeed constant at 45-50MB/s
Bottleneck is not Network 19%, target 0%

i just have another question about vPower NFS.
I deactivated this option on my new config because i never see that working.
If i activate this, there should be data in the NfsDatastore Folder, right?
But there never was any data. Not at beginning of the backup, during or at the end. Thats not normal or?
AndreZEUS
Novice
 
Posts: 7
Liked: never
Joined: Sun May 10, 2015 1:22 pm

Re: Flapping backupspeed B&R V8 with ESXi5.0.0

Veeam Logoby Gostev » Sun May 17, 2015 6:54 pm

Yes, this is normal.
Gostev
Veeam Software
 
Posts: 21396
Liked: 2350 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Flapping backupspeed B&R V8 with ESXi5.0.0

Veeam Logoby alanbolte » Sun May 17, 2015 9:39 pm

vPower NFS is used for Instant Recovery, Surebackup, and Other-OS File Level Recovery. It isn't used during backup.
alanbolte
Expert
 
Posts: 635
Liked: 170 times
Joined: Mon Jun 18, 2012 8:58 pm
Full Name: Alan Bolte


Return to VMware vSphere



Who is online

Users browsing this forum: foggy, iwik and 32 guests