Flapping backupspeed B&R V8 with ESXi5.0.0

[AndreZEUS]

Hello,

i have some trouble with my Veeam Backup (newest Version).
Support case ID: 00918380 (just opend 1min ago)

We use Veeam B&R 8 P2.
My issue: Flapping Backup Speed.
Backup starts with 90-100 MB/s till 7gb transferred (within 3min). Then speed breaks to 3 MB/s for about 20minutes. Then speed goes to 10-15MB/s till the end. In the period when the speed is between 0-4% i sometimes cant take an action on the Backup Reopsitory Server. i just can click in the Task Manager, but not in the explorer. Is it an indication of an issue on that server? But this is not always. Most time i can click in the explorer.
Get the same issue with all VMs.
Tried storage optimization local, lan and wan target. no difference.
Proxy Mode is Hot Add, 15 TCP connections allowed (dont make a different if 5-15, tested...).

My infrastruktur:
Veeam B&R as a VM (6 Cores, 8GB RAM) on a ESXi5.0.0 Host
Proxy on a VM (3 Cores, 4GB RAM) on the same ESXi5.0.0 Host (monitored; its enough ram and cpu)
Backup Repository is a Win 2008 R2 physikal Server (4 Cores with HyperThreading, so 8 Cores, 12GB RAM) connectet via 1gb Network. Its configured as a CIFS.
Second issue is: if i configure my Backup Repository as a Windows Type, the backup starts with 90-100mbit till ~7gb, then it breaks to 0 Kb/s. No error is shown.
The Backup just work on CIFS for me.

Third issue i solved by myself (research on internet):
At the end i got a Red X on "Hard disk 1 (40,0 GB) 36,0 GB read at 34 MB/s [CBT]"
Then:"Error: Der angegebene Netzwerkname ist nicht mehr verfügbar. Error code: 64
Failed to flush file buffers. File: [\\172.21.48.68\E$\AD-DFS\Fileserver\Fileserver2015-05-14T164628.vbk].
Failed to download disk.End of file.Failed to upload disk. Agent failed to process method {DataTransfer.SyncDisk}.
"

My changes so far:
- disabled IPv4 Checksum Offload and TCP Checksum Offload an the NICs from my Proxy, Backup Repository and Veeam B&R Server.
- Created the SESSTIMEOUT (DWORD) 60000 on my Proxy, Backup Repository and Veeam B&R Server
- ExtendedSessTimeout to 3600 seconds
- SServersWithExtendedSessTimeout
- Create the NetUseShareAccess (DWORD) registry value under HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication\ and set the value in to 1

What you suggest to do next?
Thanks in advance for your help.

[foggy]

Have you checked the amount of cached memory during the job run? Windows cache memory management might be the problem.

[emachabert]

Are the crenels regular ? It could be the result of flow control being triggered in the network switches.

[AndreZEUS]

Hello again,

i tested and tested and tested.


My Backup Repository in details:
Intel Server, 4 Cores with HT (so 8 Cores), 12GB RAM
RAID Controller: MegaRaid SAS 8708EM2 with newest Firmware, newest drivers on the Server.
Raid on Detail:
System Raid; 2 HDDs in RAID 1 with 250Gb for the System (2 HDDs, each with 250GB)
Storage Raid; 4 HDDs in RAID 5 with 4090Gb (4 HDDs, each with 1,5TB)

i made a chkdsk on system raid. Windows found some "not in use index entrys" and corrected them. Think this is no big deal.
Then i made a chkdsk on storage raid. Windows found no failures ot "not in use index entrys".

Now i make a consistenc check on Storage Raid with a duration of round about 30h, (startet 2h ago). So i will post the Result tomorrow.

I made a new backup repository to the system raid and backuped a small VM. It worked with fullspeed (read ~100MB/s, write 45-50MB/s).
So i guess there is something with the storage raid.

I even was able to temporarly make a new backup repository aiming on my external HDD of my notebook. I backuped our Fileserver without problems ( transferred 637Gb in 3h45min).

@emachabert
The crenels are regular when i try my Storage Backup Repository. I updated my onboard intel network chip drivers to the newest version. didnt help. i build a teaming between the managment networkport and the regular networkport, also didnt help. i dont know if i should investigate the posibility of the flow control thing on the network swtiches, cause on other backup repositorys it work. What do you mean?

@foggy
i dont see that the server ran out of memory. there is always an amount of 6-8GB free RAM shown on windows task manager (if you mean this). we dont make parallel jobs.

Any idears what i could do if the consisty check of the Storage raid ends without failures?

Thanks

[emachabert]

Looks like you are having issues with the RAID5 array that is not able to sustain a consistent speed. Do you have cache memory installed on the controler ? Is write cache enabled for the RAID5 array ?It should.

Forget the flow control stuff for the moment, if you are a le to backup with normal speed to the system RAID, then the network is working fine.

[AndreZEUS]

Mhh, wouldn't be the System Raid array infected with this slow speed too, if the cache wont work?

The System array and Storage array is configured equal.
You just can configure the Controller for all arrays on our Controller. "Native command Queuing" is enabled
Flash Size: 4MB
Memory Size: 256MB
Error Count: 0

Maybe the consistency check shows us more information... still 14h

[AndreZEUS]

Hello again,

consistency check worked so far!
I am happy now. transferspeed constant at 45-50MB/s
Bottleneck is not Network 19%, target 0%

i just have another question about vPower NFS.
I deactivated this option on my new config because i never see that working.
If i activate this, there should be data in the NfsDatastore Folder, right?
But there never was any data. Not at beginning of the backup, during or at the end. Thats not normal or?

[Gostev]

Yes, this is normal.

[alanbolte]

vPower NFS is used for Instant Recovery, Surebackup, and Other-OS File Level Recovery. It isn't used during backup.