Per-VM backup files problems still not fixed in U2!

Availability for the Always-On Enterprise

Per-VM backup files problems still not fixed in U2!

Veeam Logoby mkretzer » Tue Aug 16, 2016 7:46 am

Hello,

case 01752760 and 01880021.

in Veeam 9 update 1 we had massive issues when we used per-VM backup files with about ~600 VMs splitted in multiple copy jobs. The main issues were database deadlocks.
According to support that should be fixed.

Sadly, it only got worse in update2. After only two days we get errors like "SqlTransaction wurde beendet und kann nicht mehr verwendet werden." ("SqlTransaction was canceled and cannot be used") and "Failed to merge full backup file Error: Unable to create attacher: already exists." which leads to "Processing XXXX Error: File does not exist. File: [W:\Backups\XXXX\XXXX.vm-4086332016-08-12T220734.vbk]. Failed to download disk. Reconnectable protocol device was closed. Failed to upload disk. Agent failed to process method {DataTransfer.SyncDisk}."

Veeam support thinks our backup files are corrupted on the copy target. We never had one corruption in years and this only happens with per-VM backup files. It it 100 % reproducable - yesterday we had this for another backup copy job, did and active full which fixed it and and now it happens on a different job.

Per-VM backup files is just not stable for production in slightly bigger environments it seems... Sadly we could really use the additional performance right now...

Markus
mkretzer
Expert
 
Posts: 310
Liked: 70 times
Joined: Thu Dec 17, 2015 7:17 am

Re: Per-VM backup files problems still not fixed in U2!

Veeam Logoby v.Eremin » Tue Aug 16, 2016 12:47 pm

Veeam support thinks our backup files are corrupted on the copy target.

I tend to disagree with this statement, as the issue looks more like product DB not handling well such number of per-VM sources in the backup copy job.

Speaking about improvements for this matter, even though small number of those have been introduced in Update 2, most of them are still scheduled for the upcoming version 9.5.

Thanks.
v.Eremin
Veeam Software
 
Posts: 13291
Liked: 973 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin

Re: Per-VM backup files problems still not fixed in U2!

Veeam Logoby mkretzer » Tue Aug 16, 2016 5:47 pm

Indeed this seems to be the case... To be honest it feels like the errors are the same as before and only the SQL messages have been supressed...
mkretzer
Expert
 
Posts: 310
Liked: 70 times
Joined: Thu Dec 17, 2015 7:17 am

Re: Per-VM backup files problems still not fixed in U2!

Veeam Logoby Gostev » Tue Aug 16, 2016 9:17 pm

While I realize it does not help you now, 9.5 is truly a game changer in terms of scalability. At least our enterprise Solution Architects are pretty excited after having run its beta through paces - and these guys don't even talk to customers with less than 1000 VMs to protect :D

And as far as I remember, the issue above is actually not due to per-VM chains mostly, but primarily due to the amount of Backup Copy jobs, and the fact that these jobs are creating heavy load on our database due to a few non-optimal procedures.
Gostev
Veeam Software
 
Posts: 21396
Liked: 2350 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Per-VM backup files problems still not fixed in U2!

Veeam Logoby mkretzer » Wed Aug 17, 2016 5:42 am

Gostev,

i asked support if it would help if we consolidate our backups all to one job and i was told that this is not the issue and that the problem is the amount of VMs.

It is not such a problem with non-per-VM and the same amount of jobs (btw we are talking about 10 jobs or so).

Can you please check if what you say is really the case, if so i will consolidate all backups into one copy job.

Markus
mkretzer
Expert
 
Posts: 310
Liked: 70 times
Joined: Thu Dec 17, 2015 7:17 am

Re: Per-VM backup files problems still not fixed in U2!

Veeam Logoby v.Eremin » Wed Aug 17, 2016 11:49 am

i asked support if it would help if we consolidate our backups all to one job and i was told that this is not the issue and that the problem is the amount of VMs.

This seems to be true. I've had a quick chat with our main SQL dev and he's confirmed that.

So, the only option you seem to have at the moment is to disable per-VM chains, since consolidating all VMs in one job is unlikely to help here.

Thanks.
v.Eremin
Veeam Software
 
Posts: 13291
Liked: 973 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin

Re: Per-VM backup files problems still not fixed in U2!

Veeam Logoby mkretzer » Wed Aug 17, 2016 11:53 am

Support asked me to "test if it works fine if you disable multiple upload streams?"

Do you think that can help in any way? We are testing right now but we have to do active fulls after each error which is bad over a slow link...

Markus
mkretzer
Expert
 
Posts: 310
Liked: 70 times
Joined: Thu Dec 17, 2015 7:17 am

Re: Per-VM backup files problems still not fixed in U2!

Veeam Logoby v.Eremin » Wed Aug 17, 2016 12:47 pm 1 person likes this post

Support asked me to "test if it works fine if you disable multiple upload streams?"

Nope. As I've said, disabling per-VM chains appears to be the only way to go.

We've reached your support engineer to clarify the matter. I think she will update the ticket soon.

Thanks.
v.Eremin
Veeam Software
 
Posts: 13291
Liked: 973 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin

Re: Per-VM backup files problems still not fixed in U2!

Veeam Logoby gesiefer » Mon Aug 22, 2016 12:39 pm

I agree with this statement, the backups did not improve and failed all over the place for me, it did fix some issues but the performance of the sql calls is
not very good. Every upgrade seems to wreck things and I have to go into each job and fix which is very time consuming.
I have over 500 backup and replication jobs it is not fun with failures.
The mapping of a vm to the replica is awful. It needs to map to the name of the vm and not the ID buried inside.
I had a host issue and failed over to another host and all of the maps got cross linked. This is terrible situation.
I had to delete lots of jobs and rebuild them and the replicas to fix this issue.
The upgrade to 9 wrecked all my tape jobs and bade it a disaster to get backups to tape and reloads as the system was pulling the wrong tapes
in.
I think there needs to be some serious testing in the veeam arena before 9.5 comes out.
This is the 3rd upgrade and made a mess every time.
Also, too many jobs are getting corrupted and ghost images left for backups and replicas. I know it has to do with the pipes being very busy
but the software needs to handle this, I get up every morning wondering how many jobs are broken today.
gesiefer
Lurker
 
Posts: 1
Liked: never
Joined: Mon Sep 21, 2015 3:04 pm
Full Name: Gerald Siefer

Re: Per-VM backup files problems still not fixed in U2!

Veeam Logoby erossman » Tue Nov 29, 2016 9:11 am

Hi Guys,

my customer is fighting with the same issue since a long time.
We running only one backup copy job and get this issue after applying Update 2 again. Support told as for few month this will fix the issue.
And yes, we recreated the backup copy job and the hole chain anew. We use a physiscal, standalone backup server with SQL 2012 Standard. There isn't any heavy load.
Is there no solution for this than to disable per-VM backup repository or upgrade to 9.5 ?

Image

Regards,
Eric
erossman
Service Provider
 
Posts: 6
Liked: 1 time
Joined: Sun Feb 12, 2012 7:16 pm
Location: Frankfurt
Full Name: Eric Rossmanith

Re: Per-VM backup files problems still not fixed in U2!

Veeam Logoby foggy » Tue Nov 29, 2016 11:07 am

Hi Eric, there doesn't seem to be other solutions, so I'd recommend upgrading to v9.5 and check whether it helps. Thanks.
foggy
Veeam Software
 
Posts: 14752
Liked: 1083 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Per-VM backup files problems still not fixed in U2!

Veeam Logoby Gostev » Tue Nov 29, 2016 11:59 pm

I noted that OP is talking about Backup Copy jobs specifically, and optimizations for those were actually postponed until 9.5 U1.
Gostev
Veeam Software
 
Posts: 21396
Liked: 2350 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Per-VM backup files problems still not fixed in U2!

Veeam Logoby erossman » Wed Nov 30, 2016 5:29 pm

Thanks Gostev. So we should wait for 9.5 U1. In the meantime we will create new backup copy job chains each time after the max. number of restore point is reached.
erossman
Service Provider
 
Posts: 6
Liked: 1 time
Joined: Sun Feb 12, 2012 7:16 pm
Location: Frankfurt
Full Name: Eric Rossmanith


Return to Veeam Backup & Replication



Who is online

Users browsing this forum: Bing [Bot] and 56 guests