System.OutOfMemoryException

[cfizz34]

Since upgrading to V10 I have been running into issues backing up to my Data Domain. The more resources I throw at the proxy the more it consumes and keeps failing when trying to create the synthetic full.

Code: Select all

3/22/2020 4:30:30 AM :: Error: Exception of type 'System.OutOfMemoryException' was thrown.  

3/22/2020 4:16:00 AM :: Agent: Failed to process method {Transform.CompileFIB}: Unable to allocate memory for storage metadata bank.
Failed to open storage for read access. Storage: [xxx.com:veeam-su01@/Backup-Job-All-VM-Guests _NEW_/xxx.vm-149D2020-03-15T210119_F811.vib].

Agent: Failed to process method {Transform.CompileFIB}: Unable to allocate memory for storage metadata bank.
Failed to open storage for read access. Storage: [xxx.com:veeam-su01@/Backup-Job-All-VM-Guests _NEW_/xxx.vm-  

Case # 04071034

[HannesK]

Hello,
please continue working with support. I assume that your RAM configuration is too low.

If you combine multiple roles (proxy, repository, backup server), then you have to sum up the required hardware for each role.

Best regards,
Hannes

[cfizz34]

They suggested I run a active full job to see if that fixes the issue. Again, all I did was upgrade to V10 and since then I've given it an additional 16GB and it's still is not happy. I'll update the results tomorrow after the active full job runs.

[cfizz34]

Active job still running after almost 12 hours with 50% still to go. Joy!

[cfizz34]

So the active job ran successfully but I am still having issue where the regular vm backup fails during the end of the synthetic full.

Case Number: [ID# 04071034]

[HannesK]

Hello,
except one of the data domain customer have a "magic guess", I can only recommend to continue with support.

There are other customers having the same exception, but the solution and whole situation is different for everyone vmware-vsphere-f24/error-exception-of-t ... yException

Best regards,
Hannes

[cfizz34]

Support had me change it from 135 concurrent tasks to 10 tasks and at least it completed now. The setting was under the repository section for load control. It ran successfully at 10 and at 15 but at 15 it took longer so I am backing it back down to 10 to test once more.

[Gostev]

Wow. 135 concurrent tasks require at least 256GB on the gateway server (as per our System Requirements), which you probably did not have?

[cfizz34]

That was the default when I added it and I didn't know any better to change it. And correct, I was just a tad under that 256GB mark

[HannesK]

That was the default when I added it

hmm, I'm adding repos since V8 and has always been 4 per default. Actually I ask myself how it got to 135 as the UI only allows 128 (ok, powershell maybe)

[Gostev]

I believe it's a Data Domain specific feature: Veeam queries storage capabilities, and sets the default limit accordingly. Dedupe storage devices love multiple streams, 4 streams is just a very poor default for them...