Per-VM backup files problems still not fixed in U2!

[mkretzer]

Hello,

case 01752760 and 01880021.

in Veeam 9 update 1 we had massive issues when we used per-VM backup files with about ~600 VMs splitted in multiple copy jobs. The main issues were database deadlocks.
According to support that should be fixed.

Sadly, it only got worse in update2. After only two days we get errors like "SqlTransaction wurde beendet und kann nicht mehr verwendet werden." ("SqlTransaction was canceled and cannot be used") and "Failed to merge full backup file Error: Unable to create attacher: already exists." which leads to "Processing XXXX Error: File does not exist. File: [W:\Backups\XXXX\XXXX.vm-4086332016-08-12T220734.vbk]. Failed to download disk. Reconnectable protocol device was closed. Failed to upload disk. Agent failed to process method {DataTransfer.SyncDisk}."

Veeam support thinks our backup files are corrupted on the copy target. We never had one corruption in years and this only happens with per-VM backup files. It it 100 % reproducable - yesterday we had this for another backup copy job, did and active full which fixed it and and now it happens on a different job.

Per-VM backup files is just not stable for production in slightly bigger environments it seems... Sadly we could really use the additional performance right now...

Markus

[veremin]

Veeam support thinks our backup files are corrupted on the copy target.

I tend to disagree with this statement, as the issue looks more like product DB not handling well such number of per-VM sources in the backup copy job.

Speaking about improvements for this matter, even though small number of those have been introduced in Update 2, most of them are still scheduled for the upcoming version 9.5.

Thanks.

[mkretzer]

Indeed this seems to be the case... To be honest it feels like the errors are the same as before and only the SQL messages have been supressed...

[Gostev]

While I realize it does not help you now, 9.5 is truly a game changer in terms of scalability. At least our enterprise Solution Architects are pretty excited after having run its beta through paces - and these guys don't even talk to customers with less than 1000 VMs to protect

And as far as I remember, the issue above is actually not due to per-VM chains mostly, but primarily due to the amount of Backup Copy jobs, and the fact that these jobs are creating heavy load on our database due to a few non-optimal procedures.

[mkretzer]

Gostev,

i asked support if it would help if we consolidate our backups all to one job and i was told that this is not the issue and that the problem is the amount of VMs.

It is not such a problem with non-per-VM and the same amount of jobs (btw we are talking about 10 jobs or so).

Can you please check if what you say is really the case, if so i will consolidate all backups into one copy job.

Markus

[veremin]

i asked support if it would help if we consolidate our backups all to one job and i was told that this is not the issue and that the problem is the amount of VMs.

This seems to be true. I've had a quick chat with our main SQL dev and he's confirmed that.

So, the only option you seem to have at the moment is to disable per-VM chains, since consolidating all VMs in one job is unlikely to help here.

Thanks.

[mkretzer]

Support asked me to "test if it works fine if you disable multiple upload streams?"

Do you think that can help in any way? We are testing right now but we have to do active fulls after each error which is bad over a slow link...

Markus

[veremin]

Support asked me to "test if it works fine if you disable multiple upload streams?"

Nope. As I've said, disabling per-VM chains appears to be the only way to go.

We've reached your support engineer to clarify the matter. I think she will update the ticket soon.

Thanks.

[gesiefer]

I agree with this statement, the backups did not improve and failed all over the place for me, it did fix some issues but the performance of the sql calls is
not very good. Every upgrade seems to wreck things and I have to go into each job and fix which is very time consuming.
I have over 500 backup and replication jobs it is not fun with failures.
The mapping of a vm to the replica is awful. It needs to map to the name of the vm and not the ID buried inside.
I had a host issue and failed over to another host and all of the maps got cross linked. This is terrible situation.
I had to delete lots of jobs and rebuild them and the replicas to fix this issue.
The upgrade to 9 wrecked all my tape jobs and bade it a disaster to get backups to tape and reloads as the system was pulling the wrong tapes
in.
I think there needs to be some serious testing in the veeam arena before 9.5 comes out.
This is the 3rd upgrade and made a mess every time.
Also, too many jobs are getting corrupted and ghost images left for backups and replicas. I know it has to do with the pipes being very busy
but the software needs to handle this, I get up every morning wondering how many jobs are broken today.

[erossman]

Hi Guys,

my customer is fighting with the same issue since a long time.
We running only one backup copy job and get this issue after applying Update 2 again. Support told as for few month this will fix the issue.
And yes, we recreated the backup copy job and the hole chain anew. We use a physiscal, standalone backup server with SQL 2012 Standard. There isn't any heavy load.
Is there no solution for this than to disable per-VM backup repository or upgrade to 9.5 ?



Regards,
Eric

[foggy]

Hi Eric, there doesn't seem to be other solutions, so I'd recommend upgrading to v9.5 and check whether it helps. Thanks.

[Gostev]

I noted that OP is talking about Backup Copy jobs specifically, and optimizations for those were actually postponed until 9.5 U1.

[erossman]

Thanks Gostev. So we should wait for 9.5 U1. In the meantime we will create new backup copy job chains each time after the max. number of restore point is reached.