Comprehensive data protection for all workloads
Post Reply
mkretzer
Veeam Legend
Posts: 1145
Liked: 388 times
Joined: Dec 17, 2015 7:17 am
Contact:

Still merge issues with 9.5

Post by mkretzer »

Case 01897612 and now 01987409

In 9.5 we still have the issue that backup merges fail with messages like "Failed to process method {Transform.Patch}: Application is shutting down."

The problem is that it happens every month or so. Registry settings, upgrade to 9.5 did not help. Not using per-VM seems to help a little bit...

We start to grow desperate because with every new feature we want to use we worry that it will get worse.

Markus
Gostev
Chief Product Officer
Posts: 31527
Liked: 6700 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Still merge issues with 9.5

Post by Gostev »

I looked up the support cases. First case looks to be closed due to no responses from you, and the second one was just opened a few days ago. Please keep working with our support on this new one to get the issue resolved fully, otherwise even version 19.5 will still have it ;) The issue is not exactly wide spread, so we may never be able to catch it in our QC labs either (and it might not be even a Veeam issue to start with). Thanks!
mkretzer
Veeam Legend
Posts: 1145
Liked: 388 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Still merge issues with 9.5

Post by mkretzer »

We had the reg setting implemented and waited for the issue to occour. It takes some time...
Gostev
Chief Product Officer
Posts: 31527
Liked: 6700 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Still merge issues with 9.5

Post by Gostev »

That reg setting was nothing special, basically a timeout to kill data mover after in case of inactivity. Default is 1 hour, but as I understand the issue still happened when you bumped it to 8 hours. The inactivity is caused by the fact that data mover is waiting for the OS to acknowledge write to disk - most basic WriteFile() API call gets stuck writing a block of data to disk (never hears back from the OS that the data was written). It is very unlikely to be a Veeam bug - so far, all signs point to either some hardware issue (backup server or storage), or a Windows OS bug.

What Windows version are you using for backup server, any chance to upgrade one? If not, it might make sense for our support to open a support case with Microsoft on your behalf. They have debug versions of DLLs they normally use to analyze these sort of issues.
mkretzer
Veeam Legend
Posts: 1145
Liked: 388 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Still merge issues with 9.5

Post by mkretzer »

The remote Repo server is running 2012R2. It happens on two totally different storages both attached to that server, one uses a Dell MD3600f and the other a Fujitsu DX60. No FC switching. Different HBAs.

The issue happens on both storages at the same time so it is most likely a Windows or Veeam issue. Would you recommend we upgrade to windows 2016 (we could do a reinstall, there are no other services on that system)?

The interesting thing is that the next copy always seems to fix this...
Gostev
Chief Product Officer
Posts: 31527
Liked: 6700 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Still merge issues with 9.5

Post by Gostev »

Sure, it is worth a try - even though I am more inclined to think the issue is in hardware or drivers.
mkretzer
Veeam Legend
Posts: 1145
Liked: 388 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Still merge issues with 9.5

Post by mkretzer »

Gostev,

so you really think we should replace the server? Even if the storages are FC connected? With two different HBAs?
Whouldn't that cause NTFS or Disk errors which would show up in event log?

Markus
JoshuaPostSAMC
Expert
Posts: 124
Liked: 22 times
Joined: Jul 30, 2015 7:32 pm
Contact:

Re: Still merge issues with 9.5

Post by JoshuaPostSAMC »

How much RAM does your backup server have and how large are the incrementals you are merging? I had lots of problems with SQL Transaction logs and merge/compact and support was having a hard time nailing it down. Finally I upgraded the server from 16 GB to 32 GB, although it usually looked like it had plenty free, and most of my problems have gone away.
mkretzer
Veeam Legend
Posts: 1145
Liked: 388 times
Joined: Dec 17, 2015 7:17 am
Contact:

Re: Still merge issues with 9.5

Post by mkretzer »

Backup server has 128 GB, remote repo has 384 GB. I do not think this is the issue :-)
dgallego
Lurker
Posts: 1
Liked: never
Joined: Jun 28, 2017 8:41 am
Full Name: Daniel Gallego
Contact:

Re: Still merge issues with 9.5

Post by dgallego »

Hi, ¿Did you find a solution?

Regards
Post Reply

Who is online

Users browsing this forum: Semrush [Bot] and 154 guests