Slow VM backups after Veeam (proxy) server reboots

[acatic]

After monthly MS patch reboots, my VM backups get slow. No changes to infrastructure. This started happening some months ago, but it was also around the time that I introduced iSCSI.

Veeam 10.0.0.4461 P2 (Enterprise)
Veeam server is HP DL 360 G9, 6 CPU, 64GB RAM, running Windows Server 2019 (This is also the proxy server.)
Storage is HPE 3PAR 8200
Veeam-storage connection is a single 10Gb iSCSI
Veeam-VMware connection is a teamed 2x1Gb
VMware vVols are exposed to the backup server
In Veeam, 'storage latency control' is off; Transport mode was originally 'Automatic', but I've set it to 'Direct storage access' as well to test.

A job that normally takes 30 mins takes 8-15 hours with nothing else running on the Veeam server. In this job, some disks go at 5MB/s, some at 100KB/s, while being on same storage volume (for example).

Doing a simple sequential file transfer from Veeam server to storage goes at a healthy 170MB/s.

I tried with and without MS MPIO, and it didn't appear to matter. (I guess I don't even need it with just a single 10Gb iSCSI link)

iSCSI is set to Least Queue Depth. I've also tried Round Robin.

In the past, I've fixed it but was never sure what truly fixed it - restarting this and that, changing this and that setting, but nothing seems to work now.

Thank you in advance for all wisdom on this!

Case 04566239.

[acatic]

I had also suspected if dirty shutdowns messed up the DB or something, so the last patch update I made sure that all jobs and services were successful and stopped before the reboot.

[acatic]

I was able to download an 8GB ISO just now from my VMware environment to my backup repository using the backup server at a smooth 26 MB/s. I realize a VM backup is different than a sequential file copy; this is just to show what is possible.

[Andreas Neufert]

My guess is that the reboot messed with the detection logic of the volumes.
Can you please check that the volumes are still visible at the Proxy (for direct SAN) => Don´t enable/initialize them !!!! They need to be in offline state!!!.
Do a Rescan on the Managed Server level for the Proxy.

Then check the Backup Job Statistics - clieck a single VM and then look at the right statistics for the transport mode if it is DirectSAN or NBD?

[acatic]

I should also mention that the backup repository and the production VM storage are in the same 3PAR array. They are separate volumes though. (I have daily Backup Copy and Tape jobs configured as well.)

All infrastructure is in one rack too, so no firewalls/distance between. The iSCSI connections are in their own layer-2 network on a shared switch - same as always.

[acatic]

My guess is that the reboot messed with the detection logic of the volumes.
Can you please check that the volumes are still visible at the Proxy (for direct SAN) => Don´t enable/initialize them !!!! They need to be in offline state!!!.
Do a Rescan on the Managed Server level for the Proxy.

Then check the Backup Job Statistics - clieck a single VM and then look at the right statistics for the transport mode if it is DirectSAN or NBD?

Andreas:
That was my thinking too along the way. The volumes are still visible though (in the offline state, yes! haha)
I hadn't rescanned at the Managed Server level, so I've done that now - thanks!
I had already rescanned Backup Repositories, as well as Storage Infrastructure.

[acatic]

As for san/nbd, it showed san for each of them, even while it was set to 'automatic'.

[acatic]

Repository is also NTFS.

[acatic]

The VM backups are reverse-incremental; compression 'Optimal'; Storage optimization 'Local target'; encryption enabled; CBT enabled.

[acatic]

Also, why I initially had MPIO was due to the 3PAR itself having four iSCSI ports, but the backup server just has one. Would it serve a purpose?

[Andreas Neufert]

MPIO will only help here. No negative side effects even with one connection.
What does the Job Statistics bottleneck analyse say. What does the bottleneck analyse of an individual slow processed VM say?

Do you see any messages about encryption forced enabled?
Any messages that CBT can not be used?
Any messages about job throttling?

What does the storage monitoring say ? Specifically the latency?

[acatic]

Source is consistently the highest bottleneck for jobs and individual VMs, but the others aren't tiny - 15-50%
CBT shows ok on each run.
No messages about job throttling. (I never tried it anyway.)
Where do I check storage monitoring? You mean in Veeam or in general on my infrastructure?

One thing I was going to check after my current job finishes is to manually rescan attached storage in the server in Windows.

Thank you so far Andreas!!! It feels good to not be alone.

[acatic]

I am focusing on the repository's disk latency right now; stay tuned.

[acatic]

No disk latency anomalies. When the slow VM backups are running, the related datastores (source and destination) are pretty idle.

[acatic]

After reading that 'Least Queue Depth' is not ideal for 3PAR, I reverted back to Round Robin, rebooted, rescanned storage / managed servers; still no luck.

[acatic]

Is it not possible to add an image here?...

[Andreas Neufert]

When you click "Full editor & preview" you can add attachement and add it to the text as well.

Overall I think you should continue within the support case as it is a lot of guess work otherwise.

[acatic]

Hmm, I see no option for adding an attachment. The 'image' option offers me to add an http: path for an image.

[PetrM]

Hello,

As far as I understand from this post, Direct SAN mode is in use instead of reading from storage snapshots. I would ask our support team to check it, from my point view it does not make any sense to troubleshoot repository performance as long as Source is constantly shown as the "bottleneck". Also, I would pay attention to possible issues with iSCSI, for example recovering from dropped network packets on a switch can result in large performance degradation.

Anyway as Andreas has said, the best option is to continue working with our support team as the issue is not at all straight forward. And please keep in mind that you have an option to escalate the support request if you feel that a deeper technical analysis is required.

Thanks!

[acatic]

Petr: Thank you for the feedback.

The bottlenecks seem to alternate between source and target. However, both of them are within my 3PAR array.

It's perplexing, because if I sign into vSphere on my backup/proxy server, I can download a whole VM very quickly to the backup repository - basically the same path that a backup would take.

I currently have a VM backup going at 372KB/s - yes, KB. No other jobs running. Latency on source and target disks is near zero. Backup/proxy server hardly burdened. Why is it processing at *KB* rates??? And what does a reboot have to do with it? So weird.

I will look at switch config too, but I am hesitant to look too far if the issue surrounds the rebooting of the backup server.

I will keep poking the support tech on this.

Thank you for at least confirming that it's "not at all straightforward".

[Matts N]

Any chance you have Windows Defender turned on? If so, try with it disabled.

[acatic]

Matts: No, it's been off, but thanks for the suggestion.

[acatic]

After two tests, my solution appears to be: change the proxy transport mode from DSA to network then back to DSA.

ISCSI mode (RR/LQD) did not matter. Also, while I read that LQD is not supported, it gives me 50% better throughput.