Host-based backup of VMware vSphere VMs.
Post Reply
acatic
Enthusiast
Posts: 39
Liked: 3 times
Joined: Oct 07, 2013 3:08 pm
Contact:

Slow VM backups after Veeam (proxy) server reboots

Post by acatic »

After monthly MS patch reboots, my VM backups get slow. No changes to infrastructure. This started happening some months ago, but it was also around the time that I introduced iSCSI.

Veeam 10.0.0.4461 P2 (Enterprise)
Veeam server is HP DL 360 G9, 6 CPU, 64GB RAM, running Windows Server 2019 (This is also the proxy server.)
Storage is HPE 3PAR 8200
Veeam-storage connection is a single 10Gb iSCSI
Veeam-VMware connection is a teamed 2x1Gb
VMware vVols are exposed to the backup server
In Veeam, 'storage latency control' is off; Transport mode was originally 'Automatic', but I've set it to 'Direct storage access' as well to test.

A job that normally takes 30 mins takes 8-15 hours with nothing else running on the Veeam server. In this job, some disks go at 5MB/s, some at 100KB/s, while being on same storage volume (for example).

Doing a simple sequential file transfer from Veeam server to storage goes at a healthy 170MB/s.

I tried with and without MS MPIO, and it didn't appear to matter. (I guess I don't even need it with just a single 10Gb iSCSI link)

iSCSI is set to Least Queue Depth. I've also tried Round Robin.

In the past, I've fixed it but was never sure what truly fixed it - restarting this and that, changing this and that setting, but nothing seems to work now.

Thank you in advance for all wisdom on this!

Case 04566239.
acatic
Enthusiast
Posts: 39
Liked: 3 times
Joined: Oct 07, 2013 3:08 pm
Contact:

Re: Slow VM backups after Veeam (proxy) server reboots

Post by acatic »

I had also suspected if dirty shutdowns messed up the DB or something, so the last patch update I made sure that all jobs and services were successful and stopped before the reboot.
acatic
Enthusiast
Posts: 39
Liked: 3 times
Joined: Oct 07, 2013 3:08 pm
Contact:

Re: Slow VM backups after Veeam (proxy) server reboots

Post by acatic »

I was able to download an 8GB ISO just now from my VMware environment to my backup repository using the backup server at a smooth 26 MB/s. I realize a VM backup is different than a sequential file copy; this is just to show what is possible.
Andreas Neufert
VP, Product Management
Posts: 6707
Liked: 1401 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: Slow VM backups after Veeam (proxy) server reboots

Post by Andreas Neufert »

My guess is that the reboot messed with the detection logic of the volumes.
Can you please check that the volumes are still visible at the Proxy (for direct SAN) => Don´t enable/initialize them !!!! They need to be in offline state!!!.
Do a Rescan on the Managed Server level for the Proxy.

Then check the Backup Job Statistics - clieck a single VM and then look at the right statistics for the transport mode if it is DirectSAN or NBD?
acatic
Enthusiast
Posts: 39
Liked: 3 times
Joined: Oct 07, 2013 3:08 pm
Contact:

Re: Slow VM backups after Veeam (proxy) server reboots

Post by acatic »

I should also mention that the backup repository and the production VM storage are in the same 3PAR array. They are separate volumes though. (I have daily Backup Copy and Tape jobs configured as well.)

All infrastructure is in one rack too, so no firewalls/distance between. The iSCSI connections are in their own layer-2 network on a shared switch - same as always.
acatic
Enthusiast
Posts: 39
Liked: 3 times
Joined: Oct 07, 2013 3:08 pm
Contact:

Re: Slow VM backups after Veeam (proxy) server reboots

Post by acatic »

Andreas Neufert wrote: Jan 07, 2021 3:31 pm My guess is that the reboot messed with the detection logic of the volumes.
Can you please check that the volumes are still visible at the Proxy (for direct SAN) => Don´t enable/initialize them !!!! They need to be in offline state!!!.
Do a Rescan on the Managed Server level for the Proxy.

Then check the Backup Job Statistics - clieck a single VM and then look at the right statistics for the transport mode if it is DirectSAN or NBD?
Andreas:
That was my thinking too along the way. The volumes are still visible though (in the offline state, yes! haha)
I hadn't rescanned at the Managed Server level, so I've done that now - thanks!
I had already rescanned Backup Repositories, as well as Storage Infrastructure.
acatic
Enthusiast
Posts: 39
Liked: 3 times
Joined: Oct 07, 2013 3:08 pm
Contact:

Re: Slow VM backups after Veeam (proxy) server reboots

Post by acatic »

As for san/nbd, it showed san for each of them, even while it was set to 'automatic'.
acatic
Enthusiast
Posts: 39
Liked: 3 times
Joined: Oct 07, 2013 3:08 pm
Contact:

Re: Slow VM backups after Veeam (proxy) server reboots

Post by acatic »

Repository is also NTFS.
acatic
Enthusiast
Posts: 39
Liked: 3 times
Joined: Oct 07, 2013 3:08 pm
Contact:

Re: Slow VM backups after Veeam (proxy) server reboots

Post by acatic »

The VM backups are reverse-incremental; compression 'Optimal'; Storage optimization 'Local target'; encryption enabled; CBT enabled.
acatic
Enthusiast
Posts: 39
Liked: 3 times
Joined: Oct 07, 2013 3:08 pm
Contact:

Re: Slow VM backups after Veeam (proxy) server reboots

Post by acatic »

Also, why I initially had MPIO was due to the 3PAR itself having four iSCSI ports, but the backup server just has one. Would it serve a purpose?
Andreas Neufert
VP, Product Management
Posts: 6707
Liked: 1401 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: Slow VM backups after Veeam (proxy) server reboots

Post by Andreas Neufert » 1 person likes this post

MPIO will only help here. No negative side effects even with one connection.
What does the Job Statistics bottleneck analyse say. What does the bottleneck analyse of an individual slow processed VM say?

Do you see any messages about encryption forced enabled?
Any messages that CBT can not be used?
Any messages about job throttling?

What does the storage monitoring say ? Specifically the latency?
acatic
Enthusiast
Posts: 39
Liked: 3 times
Joined: Oct 07, 2013 3:08 pm
Contact:

Re: Slow VM backups after Veeam (proxy) server reboots

Post by acatic »

Source is consistently the highest bottleneck for jobs and individual VMs, but the others aren't tiny - 15-50%
CBT shows ok on each run.
No messages about job throttling. (I never tried it anyway.)
Where do I check storage monitoring? You mean in Veeam or in general on my infrastructure?

One thing I was going to check after my current job finishes is to manually rescan attached storage in the server in Windows.

Thank you so far Andreas!!! It feels good to not be alone.
acatic
Enthusiast
Posts: 39
Liked: 3 times
Joined: Oct 07, 2013 3:08 pm
Contact:

Re: Slow VM backups after Veeam (proxy) server reboots

Post by acatic »

I am focusing on the repository's disk latency right now; stay tuned.
acatic
Enthusiast
Posts: 39
Liked: 3 times
Joined: Oct 07, 2013 3:08 pm
Contact:

Re: Slow VM backups after Veeam (proxy) server reboots

Post by acatic »

No disk latency anomalies. When the slow VM backups are running, the related datastores (source and destination) are pretty idle.
acatic
Enthusiast
Posts: 39
Liked: 3 times
Joined: Oct 07, 2013 3:08 pm
Contact:

Re: Slow VM backups after Veeam (proxy) server reboots

Post by acatic »

After reading that 'Least Queue Depth' is not ideal for 3PAR, I reverted back to Round Robin, rebooted, rescanned storage / managed servers; still no luck.
acatic
Enthusiast
Posts: 39
Liked: 3 times
Joined: Oct 07, 2013 3:08 pm
Contact:

Re: Slow VM backups after Veeam (proxy) server reboots

Post by acatic »

Is it not possible to add an image here?...
Andreas Neufert
VP, Product Management
Posts: 6707
Liked: 1401 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: Slow VM backups after Veeam (proxy) server reboots

Post by Andreas Neufert »

When you click "Full editor & preview" you can add attachement and add it to the text as well.

Overall I think you should continue within the support case as it is a lot of guess work otherwise.
acatic
Enthusiast
Posts: 39
Liked: 3 times
Joined: Oct 07, 2013 3:08 pm
Contact:

Re: Slow VM backups after Veeam (proxy) server reboots

Post by acatic »

Hmm, I see no option for adding an attachment. The 'image' option offers me to add an http: path for an image.
PetrM
Veeam Software
Posts: 3229
Liked: 520 times
Joined: Aug 28, 2013 8:23 am
Full Name: Petr Makarov
Location: Prague, Czech Republic
Contact:

Re: Slow VM backups after Veeam (proxy) server reboots

Post by PetrM »

Hello,

As far as I understand from this post, Direct SAN mode is in use instead of reading from storage snapshots. I would ask our support team to check it, from my point view it does not make any sense to troubleshoot repository performance as long as Source is constantly shown as the "bottleneck". Also, I would pay attention to possible issues with iSCSI, for example recovering from dropped network packets on a switch can result in large performance degradation.

Anyway as Andreas has said, the best option is to continue working with our support team as the issue is not at all straight forward. And please keep in mind that you have an option to escalate the support request if you feel that a deeper technical analysis is required.

Thanks!
acatic
Enthusiast
Posts: 39
Liked: 3 times
Joined: Oct 07, 2013 3:08 pm
Contact:

Re: Slow VM backups after Veeam (proxy) server reboots

Post by acatic » 1 person likes this post

Petr: Thank you for the feedback.

The bottlenecks seem to alternate between source and target. However, both of them are within my 3PAR array.

It's perplexing, because if I sign into vSphere on my backup/proxy server, I can download a whole VM very quickly to the backup repository - basically the same path that a backup would take.

I currently have a VM backup going at 372KB/s - yes, KB. No other jobs running. Latency on source and target disks is near zero. Backup/proxy server hardly burdened. Why is it processing at *KB* rates??? And what does a reboot have to do with it? So weird.

I will look at switch config too, but I am hesitant to look too far if the issue surrounds the rebooting of the backup server.

I will keep poking the support tech on this.

Thank you for at least confirming that it's "not at all straightforward".
Matts N
Enthusiast
Posts: 59
Liked: 10 times
Joined: Dec 27, 2010 10:41 am
Full Name: Matts Nilsson
Contact:

Re: Slow VM backups after Veeam (proxy) server reboots

Post by Matts N »

Any chance you have Windows Defender turned on? If so, try with it disabled.
acatic
Enthusiast
Posts: 39
Liked: 3 times
Joined: Oct 07, 2013 3:08 pm
Contact:

Re: Slow VM backups after Veeam (proxy) server reboots

Post by acatic »

Matts: No, it's been off, but thanks for the suggestion.
acatic
Enthusiast
Posts: 39
Liked: 3 times
Joined: Oct 07, 2013 3:08 pm
Contact:

Re: Slow VM backups after Veeam (proxy) server reboots

Post by acatic »

After two tests, my solution appears to be: change the proxy transport mode from DSA to network then back to DSA.

ISCSI mode (RR/LQD) did not matter. Also, while I read that LQD is not supported, it gives me 50% better throughput.
Post Reply

Who is online

Users browsing this forum: No registered users and 65 guests