Sudden drop in speed

[sssleo]

Hi.

As described in the subject, my backup recently experienced a huge drop off in speed in the middle of a backup, and speed has not restored to normal levels since then. Below is the set up

VMWare host
1x 6TB production VM
1x Veeam backup VM
backup to 2x Synology NAS (scale out repository)

Prior to last week backup speeds were around 100MB/s, so a full backup takes around 24 hours to complete. However, starting last Tuesday, 10 hours into the backup, the speed dropped from 100MB/s to around 20-25MB/s, and has not gone back to the usual, meaning one full backup now takes up to 4 days to complete. Not exactly a very ideal situation for us. My original suspicion was a faulty switch as all devices were connected to a switch that coincidentally failed 2 days later, however we have replaced the switch with no change in results. Flipping through the event logs of the backup VM and the NAS did not yield any result of note as well. Anyone have any idea on what could cause a sudden decrease in speed in the middle of a backup?

Bottleneck stats as of current job
Source 19% -> Proxy 26% -> Network 21% -> Target 24%
Not sure where to get bottleneck stats of previously successful jobs.

[bdufour]

bottleneck stats have a normal spread.
you can search through your job history to pull stats from when the job was at 100mb/s.

what transport mode are you using? (hotadd, network, ect)
how many disk are being processed at this time? (if multiple disk are being processed that speed could be normal)
are there any throttling rules set up in veeam?
is the switch identical to the other, with identical configs?

have you tried restarting the veeam server and production vm?

[sssleo]

Hi, thanks for the reply

I managed to find my previous job stats, bottleneck stats then pretty much look like this
Source 93% > Proxy 58% > Network 41% > Target 8%
which is completely different from what I'm experiencing now

Transport mode is automatic, we have not touched this setting since the day we set up the backup. Most of the settings in there are pretty much the default settings.
Just 1 disk is being processed at the moment, and the switch is a like-for-like replacement, just a simple plug and play with no extra configuration

I've tried restarting the backup VM with the same results. I've not had the chance to restart the production yet, and although technically I can do it while the backup is running I'd prefer to do it after the backup is done, as I've not been able to successfully backup for various reasons since a week ago and I'd like to avoid any action that could potentially cause the backup to fail once again.
Backup of the Veeam VM seems fine as I managed to get one backup complete with the same amount of time taken as previous jobs, so it's just my production VM having issues.

[bdufour]

seems the issue is isolated to the production VM, as other jobs are completing within the expected time. when you can, id restart that vm, also have a look at your resource monitor on the production vm and see if anything is out of the ordinary with the cpu, memory, and also network.

[foggy]

Since the bottleneck has moved from source to a more even distribution, I'd assume some issues with the target.