Sudden performance issue on backups

[trevora]

We have 4 reverse incremental backup jobs than run each night on two different Veeam VM's. Two of the backup jobs (one on each Veeam VM) appear to have suddenly started taking so long to run they don't finish by the time it gets around to the next scheduled backup. This takes a specific Veeam box out of the list of possible issues. All jobs backup to the same target and the jobs that work successfully start after the slow jobs start and finish before them in a time that looks comparable to before this issue started - this presumably takes the target out of the list of possible issues.

Both failing backup jobs are our two largest VM's (file server and email archive, both around 1 Tb). I have noticed that one of the VM's takes over 3 hours on the 'Preparing guest for hot backup' whereas all the others (including the other 'slow to backup' VM) take around 10 seconds. What would the cause of this be and could it be affecting the other backup? I've rebooted the target NAS device and both the Veeam servers, but haven't yet rebooted the VM's that are begin backed up themselves.

I'll log a call with Veeam when i get into work, but thought i'd post this quick question just in case anybody has any quick ideas on what may be going on.

[trevora]

A bit more information: my colleague rebooted one of the servers that seemed to be taking too long. As soon as he rebooted it the backup job for that VM immediately finished successfully and then carried on backing up the other VM's in the job. At the same time several of our VM's started migrating around the hosts, so I assume the VM was using a lot of the resource during the backup job.

The second slow backup job however is still running slowly however.

[ksauer]

A few quick questions that may help isolate this issue:

1.) What is the full build version of your Veeam console?
2.) What is the bottleneck of the job via statistics? (Source %, Proxy %, Network %, Target %)
3.) Is this backup job local or off-site?
4.) Resources of the proxy being utilized? (Cores, Mem)
5.) How is the repository configured? (Windows Server, CIFS, Linux)
6.) What is the storage device?

It may be best to open a ticket with support with these answers and your issue to help further isolate this issue more in-depth.

[Vitaliy S.]

Also I would check Windows Event log at the time our backup job runs, that would allow you to see what preparation steps that long, I assume it could be VSS-related problem.

[DavidReimers]

I have come across the same thing. I have one backup job (to local Windows repository) and 2 replication jobs (to remote ESX host over 1Gbps link).

The backup job runs fine. The first replication job runs. The second replication job DID run fine for several months, but all of a sudden has slowed down dramatically.
It is only for specific VMs within the second replication, where the bottleneck is the target (99%). This happens whether the 2 replication jobs run simultaneously or sequentially.

The odd thing is that compared to the first replication job, the CBT read speed is around 2-4MB/min versus 15-100 for the other VMs in the other replication job.

Source VMs are on different ESX servers (so its unlikely to be a host-specific issue) and also have different OSs.

Veeam version is 6.5 with latest patches. Destination host is ESX 5.1 Update 1 with 9TB RAID6 array.

[foggy]

The backup job runs fine. The first replication job runs. The second replication job DID run fine for several months, but all of a sudden has slowed down dramatically.
It is only for specific VMs within the second replication, where the bottleneck is the target (99%). This happens whether the 2 replication jobs run simultaneously or sequentially.

What are the full bottleneck stats for this job? What were the bottleneck stats numbers prior it started to run slow? What about target proxies - is the same proxy used for all the VMs in the job? What transport mode is used to populate target datastore?

[trevora]

It may be best to open a ticket with support with these answers and your issue to help further isolate this issue more in-depth.

I'll log a call when this re-occurs. I've been waiting since this post and it hasn't occurred since!