Comprehensive data protection for all workloads
Post Reply
mwatson536
Influencer
Posts: 14
Liked: 1 time
Joined: Jul 10, 2015 8:59 pm
Full Name: Mitch Watson
Contact:

On-line backups have slowed

Post by mwatson536 »

Besides doing local backups we also backup to a Veeam cloud provider, everything was working fine until 9-5-22 when all of a sudden the backups were taking hours to complete. Looking at the job logs I can see the difference, the job for our exchange server went from 30-40 min to 3hrs. Nothing changed on our end except windows updates on the backup server. I have opened a ticket with the provider but it is starting to look like they are going to point to me as the issue. So my question is where can I start looking to see what may be going on?

Running Veeam B/R 11 11.0.1.1261 P20220302

Thanks
RomanK
Veeam Software
Posts: 628
Liked: 164 times
Joined: Nov 01, 2016 11:26 am
Contact:

Re: On-line backups have slowed

Post by RomanK »

Hello Mitch,
I have opened a ticket with the provider
Do you know if they opened a case with our support? The case number would help to check the current state.

If there is no case number you may open a support case with Veeam on your behalf. Then please provide case ID in this thread so we could collect all the additional details and investigate it accordingly.

Thanks
mwatson536
Influencer
Posts: 14
Liked: 1 time
Joined: Jul 10, 2015 8:59 pm
Full Name: Mitch Watson
Contact:

Re: On-line backups have slowed

Post by mwatson536 »

The provider has stated they would get Veeam involved but have not yet given me any ticket info. My question here is more based upon where I can look on my side to make sure that in fact my setup is not causing the issue while the provider works out their side. If they come back and say it's not them I will open a ticket with Veeam for sure. Just trying to trouble shoot.

Thanks
MarkBoothmaa
Veeam Legend
Posts: 179
Liked: 49 times
Joined: Mar 22, 2017 11:10 am
Full Name: Mark Boothman
Location: Darlington, United Kingdom
Contact:

Re: On-line backups have slowed

Post by MarkBoothmaa »

Do your jobs run straight away or are they waiting for backup infrastructure resource availability if you compare it to a before / after session.
mwatson536
Influencer
Posts: 14
Liked: 1 time
Joined: Jul 10, 2015 8:59 pm
Full Name: Mitch Watson
Contact:

Re: On-line backups have slowed

Post by mwatson536 »

All jobs local or cloud run straight away one after the other, no competing for resources.
MarkBoothmaa
Veeam Legend
Posts: 179
Liked: 49 times
Joined: Mar 22, 2017 11:10 am
Full Name: Mark Boothman
Location: Darlington, United Kingdom
Contact:

Re: On-line backups have slowed

Post by MarkBoothmaa »

Have you checked exclusions in Windows Defender? Just wondering if the updates have changed something there.
mwatson536
Influencer
Posts: 14
Liked: 1 time
Joined: Jul 10, 2015 8:59 pm
Full Name: Mitch Watson
Contact:

Re: On-line backups have slowed

Post by mwatson536 »

Valid point and I did put Veeam in the exclusions but it does not explain why local backups run fine, but again valid thought.

Thanks
Regnor
Veeam Software
Posts: 934
Liked: 287 times
Joined: Jan 31, 2011 11:17 am
Full Name: Max
Contact:

Re: On-line backups have slowed

Post by Regnor »

When you look at the job and select the Exchange VM, what task I now requiring more time than before?
mwatson536
Influencer
Posts: 14
Liked: 1 time
Joined: Jul 10, 2015 8:59 pm
Full Name: Mitch Watson
Contact:

Re: On-line backups have slowed

Post by mwatson536 »

The whole backup to the cloud. It use to take 40 min not its running for 3-4 hours.
Regnor
Veeam Software
Posts: 934
Liked: 287 times
Joined: Jan 31, 2011 11:17 am
Full Name: Max
Contact:

Re: On-line backups have slowed

Post by Regnor »

So in the task overview, it's the one reading the disk which takes longer now? And if you compare it to one before 9/5, the performance decreased?
mwatson536
Influencer
Posts: 14
Liked: 1 time
Joined: Jul 10, 2015 8:59 pm
Full Name: Mitch Watson
Contact:

Re: On-line backups have slowed

Post by mwatson536 »

So yes it seem that the delay is in reading some of the Hard disks during the cloud backup see below:
9/23/2022 7:20:45 PM :: Hard disk 2 (100 GB) 40.5 MB read at 21 MB/s [CBT] 00:32
9/23/2022 7:20:49 PM :: Hard disk 1 (170 GB) 22.4 GB read at 2 MB/s [CBT] 03:05:22



But during the local NAS backup :
9/23/2022 6:33:30 PM :: Hard disk 2 (100 GB) 87 MB read at 38 MB/s [CBT] 00:06
9/23/2022 6:33:31 PM :: Hard disk 1 (170 GB) 41.9 GB read at 20 MB/s [CBT] 35:17

Thoughts?
MarkBoothmaa
Veeam Legend
Posts: 179
Liked: 49 times
Joined: Mar 22, 2017 11:10 am
Full Name: Mark Boothman
Location: Darlington, United Kingdom
Contact:

Re: On-line backups have slowed

Post by MarkBoothmaa »

We’ve seen this behaviour with clients before but have found that doing a full backup resolves this. Is that an option I.e. do you have sufficient free space in your quota to do this?
Regnor
Veeam Software
Posts: 934
Liked: 287 times
Joined: Jan 31, 2011 11:17 am
Full Name: Max
Contact:

Re: On-line backups have slowed

Post by Regnor »

Interesting that it's only a single VM/disk. Same proxy, compression settings, etc for both jobs?
If so, I would work together with the VCSP and see if they can find anything on their side.
mwatson536
Influencer
Posts: 14
Liked: 1 time
Joined: Jul 10, 2015 8:59 pm
Full Name: Mitch Watson
Contact:

Re: On-line backups have slowed

Post by mwatson536 » 1 person likes this post

OK lets be clear here because everyone is focusing on one backup. All my backups to the cloud provider are now taking longer whether it is my DC or my SQL or my Exchange server all backups to the cloud are taking 2-3x longer than before 9-6-22. Also I did not say this before but I have a second server out in CA that does our backups there and I am seeing the same thing. The VCSP decided it is Sentinel One and closed the ticket so I am moving to have it escalated.

Thanks
HTownVinny
Novice
Posts: 4
Liked: never
Joined: Dec 15, 2021 5:53 pm
Full Name: Vince Wood
Location: Houston, TX USA
Contact:

Re: On-line backups have slowed

Post by HTownVinny »

Mitch,
Have you considered setting up a trial with a different VCSP to see if the same scenario occurs with another provider?
1. This might give you data to take to your current provider. Could be helpful…
2. If you do this, duplicate an existing Backup Copy Job and change ONLY the target.
-HTownVinny
mwatson536
Influencer
Posts: 14
Liked: 1 time
Joined: Jul 10, 2015 8:59 pm
Full Name: Mitch Watson
Contact:

Re: On-line backups have slowed

Post by mwatson536 »

Valid option I will consider.

Thanks
LickABrick
Enthusiast
Posts: 58
Liked: 30 times
Joined: Dec 23, 2019 7:26 pm
Full Name: Lick A Brick
Contact:

Re: On-line backups have slowed

Post by LickABrick »

Hi, we've had issues where our firewall's AV/IPS interfered with cloud backups making them terribly slow or sometimes making them fail.
mwatson536
Influencer
Posts: 14
Liked: 1 time
Joined: Jul 10, 2015 8:59 pm
Full Name: Mitch Watson
Contact:

Re: On-line backups have slowed

Post by mwatson536 »

Yea I thought about that and have exceptions in both for it.

Thanks
Post Reply

Who is online

Users browsing this forum: Bing [Bot], Google [Bot] and 134 guests