Guest timeout issues caused by target performance?

[trevora]

I haven't raised a support call as this is in relation to an ongoing 'niggle', as opposed to a specific error or problem and am just after some thoughts at this stage.

We have around 25 VM's, running on 3 Dell Poweredge hosts with Equalogic SAN datastores. Every VM is backed up Mon-Fri as a reverse incremental backup to a QNAP 16 Tb NAS device. We have two virtual Veeam servers, each running two jobs to do this (i.e. 4 jobs in total with around 6-7 VM's in each job). The start times of these jobs are staggered to prevent them all starting at once, but there is some inevitable overlap of backups through the night.

Sometimes every backup is successful. However, more often than not, a few VM's fail with guest time-out errors. Occasionally, a lot of the backups fail with guest time-out errors. There is no pattern in which VM's fail - it can happen to any at any-time. There is also very little going on on the VM's when the backup takes place, so I'm not sure the problem is with the guests. Is it possible our target is being over-worked and causing guest time-out issues?

Also, we perform a full backup of all the servers in a single job each weekend, when some of the VM's will be being used more than they are overnight. This backup works every time with no issues (although it does take 24+ hours).

Please feel free to suggest what may be causing these issues.

[Shestakov]

Hello trevora,
Thank you for the detailed description.

We have two virtual Veeam servers, each running two jobs to do this (i.e. 4 jobs in total with around 6-7 VM's in each job). The start times of these jobs are staggered to prevent them all starting at once, but there is some inevitable overlap of backups through the night.

Out of curiosity, why do you need 4 jobs for 25VMs, why not to go for 2 jobs?

There is no pattern in which VM's fail - it can happen to any at any-time. There is also very little going on on the VM's when the backup takes place, so I'm not sure the problem is with the guests. Is it possible our target is being over-worked and causing guest time-out issues?

Target storage is not related to VM`s guest errors.

If you experience one of these errors: "VSSControl: Failed to freeze guest, wait timeout" or "VSSFreezer: Failed to prepare guest, wait timeout 900 sec", please review this Knowledge Base article.
P.S. Whatever type of error you have, you can always contact Veeam Support team.

Thank you.

[trevora]

Thanks for the reply.

The reason we have 4 jobs is purely for admin purposes - all SQL servers are backed up together in one job, file servers in another, simple client OS's are backed up in another etc. A lot of the VM's change very little on a daily basis (RDS servers, firewalls, XP clients that we use for remote workers etc) so we have been thinking about not bothering to back up everything every night so I guess that may help.

Target storage not affecting guest errors makes sense - I was just clucthing at straws trying to work out what's going on!

I've had a look at the KB article and that only seems to be related to SQL or vCenter VM's and we have it happening on a wide variety of VM's. For example, this morning the following VM's have failed to back up - Win2008 print server, Win2008 TFS server, a couple of XP clients, an Exchange server and a SQL server.

I think I'll log a support call as it's happened again today - I'll update this post appropriately with any solutions.

[Shestakov]

trevora,

Having 4 jobs should not cause any problems, including the one with timeout.
Let`s see what will be a result of logs investigation.
Could you please provide support case number once you open it, so I can follow your situation?

Thank you.

[trevora]

Case logged as # 00629221.

Currently being pointed to kb1337 which suggests moving our VM's to a quicker datastore. Gives us a bit of a problem as we only have the one datastore! I'm guessing we'll therefore need to look at our backup routine to reduce the load on the datastore.

[Shestakov]

Thanks for updating the topic, trevora!

I`ll keep an eye on your situation.

[trevora]

Quick update on this one. After I logged the call it started working correctly, backing up all VM's properly without me making any changes, so I left it.

However, I was off for a week last week and all during that week the problem re-occurred, so I had a look yesterday when i got back to work. I had a look at the performance monitors in vCenter but couldn't see anything obvious there, but I did notice that DRS had moved both our Veeam VM's onto the same host. I manually migrated one of them to another host and last night the backup worked perfectly for the first time in over a week, so it looks like it could be a resource issue when running both Veeam VM's on a single one of our hosts. If this proves to work over the next couple of nights then I'll create a DRS rule to keep them on separate hosts (as we do with our DC's to prevent them all ending up on the same host).

[Vitaliy S.]

If both Veeam proxy and backed up VMs cause issues when they are located on the same host, then I would also check what backup more you're using and the performance of the host when all jobs start.