in February we moved our Backup and Replication server from version 8.0 on an HP DL360G7 with Windows Server 2008 R2 to version 9.5 on an HP DL360Gen9 and Windows Server 2016. We are backing up machines from VMware 6.0 U2 using Direct SAN access and Reserved Incrementals and are writing the backup files as per-vm backup files to a local ReFS formatted drive directly attached to the Veeam server via Fibre Channel. The ReFS formatted drive is based on a volume in an HP MSA2040 storage array that is dedicated to the Veeam server only.
To stay safe, we are running Active Fulls for every job each Saturday. Our problem is that the Active Fulls sometimes (and it seems to get worse from week to week now) runs very slowy so meanwhile we have to cancel the backup. At first we thought that the slow backup speed is solved by a reboot of the server and therefore might be caused by a yet unknown driver problem in Windows Server 2016 i.e., however, last Saturday we rebooted the server before the start of the Active Full backup and it sill ran very slowly.
The total size of our primary backup job is 18.5TB and actually contains about 11TB data. When the Active Full is running as expected, the backup takes about 8 hours to complete:
As mentioned above, it is getting worse and now we have backup times from over 60 hours and have to cancel the job before completion:
Some technical specifications of the environment:
- 1 Veeam Backup and Replication 9.5 U1 server based on Windows Server 2016
- Server has 16 cores, 32GB RAM and one Backup Proxy with 12 concurrent tasks
- 3 Backup Repositores which all together are using 10 concurrent tasks
- Backup Repositories are all per-vm based and reside on one local drive
- The local drive resides on a HP MSA2040 FC array dedicated for Veeam
- FC speed is 8gb/s, the mentioned drive consists of 48x 1.2TB SAS 10k drives
- Data source is a HP 3PAR 8200 with VMs residing on 64x 1.2TB SAS 10k drives
In case the Active Full for the above mentioned job with 11TB is completed within 8 hours we are experiencing a speed of roughly 385MB/s which is what we expect. In the case of 60 hours, we are down to roughly 50 MB/s which is way to slow and even slower than it was with Veeam 8.0 on the old server which wrote the backups to an even slower HP P2000 G3 with only 12 4TB 7.2K SATA drives!
It is also unlikely that the problem comes from the 3PAR since this would mean that the load on 3PAR would exist for the 60 hours as well which is not the case. There is also not that much load on the SAN switches for 60 hours. When running a Veeam Zip right now, speed is just fine so currently we cannot reproduce the behaviour when we want to. We however have not tested this during the backup window when the backup is actually running.
In addition to that I'm wondering why the restore assistant shows the full backups with a timestamp from Friday and not Saturday. The VBK files also have a time stamp from Fridays where however no full backup is run at all:
We have also opened a support case on this with ticket ID 02146161 but it currently seems that the logs do not show any indication of a problem related to Veeam. Therefore I wanted to share it on the forums as well. Any help is highly appreciated!
Thanks
Michael
I've seen some other Forum members also benefiting much from more RAM with these kinds of issues.
Going to have a look at the information you provided, thanks a lot!
