Backup too slow, problem started from nothing

[ivanildo.galvao]

I have a customer using Veeam B&R 8.0, which I installed and configured, at least 4 years ago. It is on a physical server, Dell R640, with 32GB of RAM, 02 Xeon processors and 4TB of SATA disks, where backups of at least 10 virtual machines in a vSphere 6.0 cluster are stored. This Veeam server is also the vCenter Server, Windows Server 2012 operating system.
Backup and replication have always had a good speed, despite traffic passing on the same LAN network as other servers and services.
But, two weeks ago the backup was extremely slow, there are times when the transfer rate reaches 10MB / s, it is quite ridiculous, the last backup took 22 hours to finish.
In an attempt to solve this problem, I created a new management vmkernel adapter on each ESXi, enabled jumbo frame with MTU 9000, using separate switches (they are the same switches on the iSCSI network that connects the hosts to Dell's MD 3200i storage, which is where the datastore of the Vmware cluster). On the Veeam server, I configured two network cards in NIC Teaming (1 + 1 = 2Gb) and also with MTU 9000, it didn't work, the backup is still extremely slow, with very low rates.
What can it be ?
I don't understand why this problem started all of a sudden, I even installed Update 3 for Veeam 8, but it didn't work.

I await ideas and suggestions, thank you.

[HannesK]

Hello,
the only serious suggestion I can give is bringing the whole environment into a supported state (Veeam & VMware and probably also Windows). I guess that there was an update somewhere that broke something. Changing MTU is no silver bullet. It's just fine tuning.

This Veeam server is also the vCenter Server

that's the first thing I would fix. And upgrade / new installation to V10. Keep in mind that there is no direct upgrade path from V8.

Best regards,
Hannes

[ivanildo.galvao]

Well, I can try to run Windows Update to see if any bugs are fixed and download a more current version of ESXi 6.0 and apply it to the hosts. Veeam already has the latest update, there is no way to install Veeam version 10, as the customer does not have an active support contract for this and does not want to spend on it now.

[HannesK]

ESXi 6.0 is also already out of general support...

https://www.vmware.com/content/dam/digi ... matrix.pdf

[ivanildo.galvao]

In another attempt to solve the problem, I updated the firmware and driver for the Veeam server's network cards, they were well out of date, I updated to versions released by Dell this year.

But even so the problem continues, I'm running out of options.

[Regnor]

I wouldn't touch that environment to much as it sounds like everything is out of date; you can easily introduce new issues when updating/upgrading only parts of it.
My only guest would be some kind of hardware problem on the VMware or backup storage; perhaps a faulty disk or raid cache/battery.

[PetrM]

Hello,

I would also recommend to examine backup job statistics and to check:
1) The data processing stage which is marked as Bottleneck
2) If there are some warnings related to the failed attempts to use the selected proxy or transport mode or maybe even CBT
3) You could compare statistics corresponding to job runs before the problem occured with the recent job runs: probably you would find there some differences related to the issue

Thanks!

[ivanildo.galvao]

I wouldn't touch that environment to much as it sounds like everything is out of date; you can easily introduce new issues when updating/upgrading only parts of it.
My only guest would be some kind of hardware problem on the VMware or backup storage; perhaps a faulty disk or raid cache/battery.

I don't think it's a hardware problem.
VMware servers, Veeam server and Storage do not have alerts on the front panel, also without error logs.

[ivanildo.galvao]

Hello,

I would also recommend to examine backup job statistics and to check:
1) The data processing stage which is marked as Bottleneck
2) If there are some warnings related to the failed attempts to use the selected proxy or transport mode or maybe even CBT
3) You could compare statistics corresponding to job runs before the problem occured with the recent job runs: probably you would find there some differences related to the issue

Thanks!

VM01
8/20/2020 01:44:56 :: Busy: Source 51%> Proxy 18%> Network 62%> Target 0%
8/20/2020 01:44:56 :: Primary bottleneck: Network

VM02
8/20/2020 10:44:57 :: Busy: Source 24%> Proxy 11%> Network 88%> Target 24%
8/20/2020 10:44:57 :: Primary bottleneck: Network

VM03

8/20/2020 07:15:18 :: Busy: Source 31%> Proxy 11%> Network 56%> Target 92%
8/20/2020 07:15:18 :: Primary bottleneck: Target

VM04

8/20/2020 10:32:45 :: Busy: Source 20%> Proxy 13%> Network 70%> Target 91%
8/20/2020 10:32:45 :: Primary bottleneck: Target

[PetrM]

Hello,

Based on this statistics I'd recommend to run additional performance tests against the repository disk sub-system, for example this KB can give a better idea how to do it.
The network channel between proxy and repository is worth to be tested as well, for instance by iPerf.

Anyway, the most important step is to bring the environment into a supported state as Hannes has suggested.

Thanks!

[n0creativity]

I don't think it's a hardware problem.
VMware servers, Veeam server and Storage do not have alerts on the front panel, also without error logs.

I don't know Dell servers as well (I prefer HPE ProLiant), but I am quite sure that there is a better way to determine whether hardware failure/fault is the cause than the lights. The whole scenario sounds very much like a hardware issue.

• It started suddenly
• The server clearly has not been kept up with updates, so updates are likely not the cause
• Poor disk performance is a common result of a dead RAID cache battery, a failed/failing drive, a RAID rebuild as a result of a detected failure, etc etc

I would definitely stay away from making changes to the software unless the client is willing to do it correctly and get it up to date. You'll likely just introduce more issues or make it more difficult to determine the root cause.
Other things to check would be...

• Available disk space
• Other running processes
• Is storage dedup enabled? It can seriously impact performance while doing certain tasks
• Is vCenter writing excessive logs?
• What changed right around the time the issue appeared?

[mweissen13]

My long shot guess would also be the RAID battery. AFAIK Dell servers display shows no error when the battery goes bad, but they disable the write cache which in turn affects the performance badly. To verify this, you can change the cache mode to "Force Write Back". When the performance goes back to normal the Battery is bad. DANGER: do not leave this setting, as data loss is guaranteed on the next power loss. Change it back to "Write Back".

[fosterc]

I am going to assume by your comment at you're possibly not familiar with Dell/EMC Server Administrator package that monitors the server hardware for errors and/or their monitoring solution?

I'd suggest that you possibly install this. It should highlight any hardware issues clearly to you.

Problem being is that should you contact Dell, their default answer will be to update to the latest firmware and drivers. You could do this ahead of time using the SUU (Server Update Utility) ISO which will scan the machine and advise which components need attention.

At least that way you could make an educated decision as to the risk/impact vs reward prior to progressing.

As has already been said, bringing the environment up to a supported level would personally be my priority. If that really isn't an option to take a look at the update history and investigate if in the wild there are many others experiencing similar issues, then advise your client. At the end of the day, the client has created this issue by not maintaining their environment and as good as us IT types are we aren't miracle workers... most of the time.

[vmtech123]

Incompatible Firmware / Software versions is a nightmare with VMware and weird issues. Be very careful about updating to something unsupported as well.
In any type of business you shouldn't be running software/hardware if you can't afford to have maintenance and support. For Security patches alone this is a HUGE red flag if they are unwilling to pay. They may Pay down the road, but in bitcoin to some one else.

On that note, failed cache battery cripples storage and have seen it many times. Just because there isn't a blinking orange light and things are working doesn't mean there isn't an issue either. That you can buy on ebay if support is gone on the host

[ivanildo.galvao]

I fully understand all the points mentioned here, unfortunately the customer does not want to renew support at Veeam, as the dollar here in Brazil is very high, it makes the purchase of hardware and software very expensive, of course I do all the work of raising customer awareness, informing the importance of using modern software and equipment, with warranty and support, sometimes works and sometimes not.

Well, as another attempt, I did the following: I completely uninstalled Veeam and installed it again, same version 8.0, then I applied Update 3, restored the configuration backup and it seems that it worked, at least voted the previous time average, the transfer rate was 120Mb / s, it is little yet, but it was worse before, between 10 and 30 / Mbs, the backup was taking 24 hours, this time it took 12 hours.

I could later, at another time, try to update the firmware of the hosts, storage and switches, but I don't take any chances, as these are also out of warranty and without being able to also evolve to Veeam 10, I see no sense, it is an unnecessary risk.

Thank you all !

[vmtech123]

Trust me.

If you update a network driver or storage adapter driver to an unsupported level just ONCE on a production machine, you won't do it again

happy you got it going