Error: Exception of type 'System.OutOfMemoryException' was thrown.

[lobo519]

Case# 03825745

its been over a week and no resolution. I asked for escalation yesterday and never heard back - on hold now trying to get some one.

I am getting this error on only a single VM. It's larger - has two 16tb disks. started after trying to expand the drive but realized we were at the block size limit and couldn't.

Put in a new test job - same failure

Disabled storage snapshots (Nimble) - works without storagesnap but this isn't really acceptable

Latest suggestion was to vmotion the VM and disable exclude swap files. I did it just to appease and obviously that did not fix it..

[foggy]

As far as I can see, you've already had a webex session with our engineer. Let's see what he will be able to find out based on that. I'd suggest looking at what process chews up the memory during backup and compare job runs with and without storage integration. Chances are disk size is the reason.

[bdunington]

Hi Foggy

Could be the same issue where having the Veeam Installer service is 32bit and runs out of memory.(We need to reboot every two days so we don't get 5001 out of memory (Data Domain but same result) ) I know your engineering team is working on a hotfix. We're still waiting

[foggy]

In that case, storage integration wouldn't make the difference. But let's see.

[MLindsey360]

Not to pile on here, and maybe this requires its own thread, but I have a very similar issue. We have one VM that is having the same issue its the only machine in our entire vCenter that is getting the System.OutOfMemoryException failure. Its one of our more sizable VMs as well attached to back end Pure storage. Storage snapshots are a must and thus cannot be disabled. I have tried all methods mentioned in this thread as well as some other off the wall issues. I cant see how this could be an issue with the VBR as its the only VM that is throwing this error and its happening every night. @Foggy, should I open a new case with support for this or is this a known issue without resolution?

Thanks,
Mike

[wishr]

Hi Michael,

Welcome to Veeam Community Forums!

Yes please, open a support case and let us know your case ID.

Thanks

[MLindsey360]

I have opened case #03866158. Still early in the info gathering stage. I would be pleasantly surprised at an obvious fix in light of the other issues on the board that are tied to this error...although I remain hopeful!

Thanks,
Mike

[MLindsey360]

Just an FYI, this issue has been resolved. The machine in question had a large snapshot and support was able to determine it was causing the out of memory error. I was able to remove the snap and rerun the job and it ran without error. Thanks

[Gostev]

Great, and thanks for coming back to share the resolution!

[mtellier]

I am experiencing the same "System.OutOfMemoryException failure" issue and have been working with Technical Support now for more than 1 1/2 weeks (Case # 03861739) with no resolution or progress.
This is a single Windows 2019 VM (file server) with a total of 12.3TB of data to backup, spread across 4 separate VMDK files each on their own datastore. We are using Nimble Storage All Flash array on the back end.

We have no snapshots on the VM
All systems have been rebooted
Storage and vCenter have been rescanned
Tried dedicating 1 physical proxy server with 1 thread
Placed the VM in it's own backup job
Storage vMotion OS to dedicated datastore
No evidence of running low on physical memory, disk space or CPU
Job runs for 1.5 hours before failure

Any suggestions would be most welcome.

Thanks,
Mark

[wishr]

Hi Mark,

I've asked support management to review the case. Our support engineers should get in touch with you shortly.

Thanks

[mtellier]

The problem has been resolved, support found large number of blockmapping errors in logs. I was asked to storage vMotion the Virtual Machine to re-align the blocks on disk.

I'm not quite certain that the blocks were mis-aligned, as this is vCenter and ESXi 6.5 with Nimble drivers installed on Hosts. All datastores were recently created along with the VM in vCenter.
However, I did notice something interesting with the VMDK file names, they were all on dedicated datastores but with the same name: SERVER.VMDK. After the storage migration, they all had different names: SERVER_4.VMDK.

[lobo519]

Sorry for not updating - Support suggested there was some minor VMDK corruption and that I storage motion to another datastore - which did resolve the issue.

But now I am having this issues on another large VM - nimble storage snaps.

I feel like there is more going on here. Can we get some attention on this??

[wishr]

Hi Lobo,

Thanks for updating. Regarding the new issue please raise an independent support case with our engineers or re-open the existing one to get it fully sorted.

Thanks

[amarsaudon]

Just a heads up - support was unable to assist me with this exact issue. I ended up having to create a new Nimble Volume / VMware Datastore and doing a storage vmotion over on to it. Unfortunately this VM was 22TB, so this was a cumbersome exercise and I was fortunate to have had the surplus capacity to be able to do so.
Case# 04011801