On-line backups have slowed

[mwatson536]

Besides doing local backups we also backup to a Veeam cloud provider, everything was working fine until 9-5-22 when all of a sudden the backups were taking hours to complete. Looking at the job logs I can see the difference, the job for our exchange server went from 30-40 min to 3hrs. Nothing changed on our end except windows updates on the backup server. I have opened a ticket with the provider but it is starting to look like they are going to point to me as the issue. So my question is where can I start looking to see what may be going on?

Running Veeam B/R 11 11.0.1.1261 P20220302

Thanks

[RomanK]

Hello Mitch,

I have opened a ticket with the provider

Do you know if they opened a case with our support? The case number would help to check the current state.

If there is no case number you may open a support case with Veeam on your behalf. Then please provide case ID in this thread so we could collect all the additional details and investigate it accordingly.

Thanks

[mwatson536]

The provider has stated they would get Veeam involved but have not yet given me any ticket info. My question here is more based upon where I can look on my side to make sure that in fact my setup is not causing the issue while the provider works out their side. If they come back and say it's not them I will open a ticket with Veeam for sure. Just trying to trouble shoot.

Thanks

[MarkBoothmaa]

Do your jobs run straight away or are they waiting for backup infrastructure resource availability if you compare it to a before / after session.

[mwatson536]

All jobs local or cloud run straight away one after the other, no competing for resources.

[MarkBoothmaa]

Have you checked exclusions in Windows Defender? Just wondering if the updates have changed something there.

[mwatson536]

Valid point and I did put Veeam in the exclusions but it does not explain why local backups run fine, but again valid thought.

Thanks

[Regnor]

When you look at the job and select the Exchange VM, what task I now requiring more time than before?

[mwatson536]

The whole backup to the cloud. It use to take 40 min not its running for 3-4 hours.

[Regnor]

So in the task overview, it's the one reading the disk which takes longer now? And if you compare it to one before 9/5, the performance decreased?

[mwatson536]

So yes it seem that the delay is in reading some of the Hard disks during the cloud backup see below:
9/23/2022 7:20:45 PM :: Hard disk 2 (100 GB) 40.5 MB read at 21 MB/s [CBT] 00:32
9/23/2022 7:20:49 PM :: Hard disk 1 (170 GB) 22.4 GB read at 2 MB/s [CBT] 03:05:22



But during the local NAS backup :
9/23/2022 6:33:30 PM :: Hard disk 2 (100 GB) 87 MB read at 38 MB/s [CBT] 00:06
9/23/2022 6:33:31 PM :: Hard disk 1 (170 GB) 41.9 GB read at 20 MB/s [CBT] 35:17

Thoughts?

[MarkBoothmaa]

We’ve seen this behaviour with clients before but have found that doing a full backup resolves this. Is that an option I.e. do you have sufficient free space in your quota to do this?

[Regnor]

Interesting that it's only a single VM/disk. Same proxy, compression settings, etc for both jobs?
If so, I would work together with the VCSP and see if they can find anything on their side.

[mwatson536]

OK lets be clear here because everyone is focusing on one backup. All my backups to the cloud provider are now taking longer whether it is my DC or my SQL or my Exchange server all backups to the cloud are taking 2-3x longer than before 9-6-22. Also I did not say this before but I have a second server out in CA that does our backups there and I am seeing the same thing. The VCSP decided it is Sentinel One and closed the ticket so I am moving to have it escalated.

Thanks

[HTownVinny]

Mitch,
Have you considered setting up a trial with a different VCSP to see if the same scenario occurs with another provider?
1. This might give you data to take to your current provider. Could be helpful…
2. If you do this, duplicate an existing Backup Copy Job and change ONLY the target.
-HTownVinny

[mwatson536]

Valid option I will consider.

Thanks

[LickABrick]

Hi, we've had issues where our firewall's AV/IPS interfered with cloud backups making them terribly slow or sometimes making them fail.

[mwatson536]

Yea I thought about that and have exceptions in both for it.

Thanks