Host-based backup of VMware vSphere VMs.
Post Reply
lobo519
Veteran
Posts: 315
Liked: 38 times
Joined: Sep 29, 2010 3:37 pm
Contact:

Error: Exception of type 'System.OutOfMemoryException' was thrown.

Post by lobo519 »

Case# 03825745

its been over a week and no resolution. I asked for escalation yesterday and never heard back - on hold now trying to get some one.

I am getting this error on only a single VM. It's larger - has two 16tb disks. started after trying to expand the drive but realized we were at the block size limit and couldn't.

Put in a new test job - same failure

Disabled storage snapshots (Nimble) - works without storagesnap but this isn't really acceptable

Latest suggestion was to vmotion the VM and disable exclude swap files. I did it just to appease and obviously that did not fix it..
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Error: Exception of type 'System.OutOfMemoryException' was thrown.

Post by foggy »

As far as I can see, you've already had a webex session with our engineer. Let's see what he will be able to find out based on that. I'd suggest looking at what process chews up the memory during backup and compare job runs with and without storage integration. Chances are disk size is the reason.
bdunington
Influencer
Posts: 22
Liked: 3 times
Joined: Jul 15, 2014 3:41 pm
Contact:

Re: Error: Exception of type 'System.OutOfMemoryException' was thrown.

Post by bdunington »

Hi Foggy

Could be the same issue where having the Veeam Installer service is 32bit and runs out of memory.(We need to reboot every two days so we don't get 5001 out of memory (Data Domain but same result) ) I know your engineering team is working on a hotfix. We're still waiting :(
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Error: Exception of type 'System.OutOfMemoryException' was thrown.

Post by foggy »

In that case, storage integration wouldn't make the difference. But let's see.
MLindsey360
Novice
Posts: 3
Liked: 4 times
Joined: Jul 19, 2013 1:16 am
Full Name: Michael Lindsey
Contact:

Re: Error: Exception of type 'System.OutOfMemoryException' was thrown.

Post by MLindsey360 »

Not to pile on here, and maybe this requires its own thread, but I have a very similar issue. We have one VM that is having the same issue its the only machine in our entire vCenter that is getting the System.OutOfMemoryException failure. Its one of our more sizable VMs as well attached to back end Pure storage. Storage snapshots are a must and thus cannot be disabled. I have tried all methods mentioned in this thread as well as some other off the wall issues. I cant see how this could be an issue with the VBR as its the only VM that is throwing this error and its happening every night. @Foggy, should I open a new case with support for this or is this a known issue without resolution?

Thanks,
Mike
wishr
Veteran
Posts: 3077
Liked: 453 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Error: Exception of type 'System.OutOfMemoryException' was thrown.

Post by wishr »

Hi Michael,

Welcome to Veeam Community Forums!

Yes please, open a support case and let us know your case ID.

Thanks
MLindsey360
Novice
Posts: 3
Liked: 4 times
Joined: Jul 19, 2013 1:16 am
Full Name: Michael Lindsey
Contact:

Re: Error: Exception of type 'System.OutOfMemoryException' was thrown.

Post by MLindsey360 » 1 person likes this post

I have opened case #03866158. Still early in the info gathering stage. I would be pleasantly surprised at an obvious fix in light of the other issues on the board that are tied to this error...although I remain hopeful!

Thanks,
Mike
MLindsey360
Novice
Posts: 3
Liked: 4 times
Joined: Jul 19, 2013 1:16 am
Full Name: Michael Lindsey
Contact:

Re: Error: Exception of type 'System.OutOfMemoryException' was thrown.

Post by MLindsey360 » 3 people like this post

Just an FYI, this issue has been resolved. The machine in question had a large snapshot and support was able to determine it was causing the out of memory error. I was able to remove the snap and rerun the job and it ran without error. Thanks
Gostev
Chief Product Officer
Posts: 31455
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Error: Exception of type 'System.OutOfMemoryException' was thrown.

Post by Gostev »

Great, and thanks for coming back to share the resolution!
mtellier
Influencer
Posts: 15
Liked: 22 times
Joined: Feb 18, 2016 3:56 pm
Full Name: Mark Tellier
Contact:

Re: Error: Exception of type 'System.OutOfMemoryException' was thrown.

Post by mtellier »

I am experiencing the same "System.OutOfMemoryException failure" issue and have been working with Technical Support now for more than 1 1/2 weeks (Case # 03861739) with no resolution or progress.
This is a single Windows 2019 VM (file server) with a total of 12.3TB of data to backup, spread across 4 separate VMDK files each on their own datastore. We are using Nimble Storage All Flash array on the back end.

We have no snapshots on the VM
All systems have been rebooted
Storage and vCenter have been rescanned
Tried dedicating 1 physical proxy server with 1 thread
Placed the VM in it's own backup job
Storage vMotion OS to dedicated datastore
No evidence of running low on physical memory, disk space or CPU
Job runs for 1.5 hours before failure

Any suggestions would be most welcome.

Thanks,
Mark
wishr
Veteran
Posts: 3077
Liked: 453 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Error: Exception of type 'System.OutOfMemoryException' was thrown.

Post by wishr »

Hi Mark,

I've asked support management to review the case. Our support engineers should get in touch with you shortly.

Thanks
mtellier
Influencer
Posts: 15
Liked: 22 times
Joined: Feb 18, 2016 3:56 pm
Full Name: Mark Tellier
Contact:

Re: Error: Exception of type 'System.OutOfMemoryException' was thrown.

Post by mtellier » 1 person likes this post

The problem has been resolved, support found large number of blockmapping errors in logs. I was asked to storage vMotion the Virtual Machine to re-align the blocks on disk.

I'm not quite certain that the blocks were mis-aligned, as this is vCenter and ESXi 6.5 with Nimble drivers installed on Hosts. All datastores were recently created along with the VM in vCenter.
However, I did notice something interesting with the VMDK file names, they were all on dedicated datastores but with the same name: SERVER.VMDK. After the storage migration, they all had different names: SERVER_4.VMDK.
lobo519
Veteran
Posts: 315
Liked: 38 times
Joined: Sep 29, 2010 3:37 pm
Contact:

Re: Error: Exception of type 'System.OutOfMemoryException' was thrown.

Post by lobo519 » 1 person likes this post

Sorry for not updating - Support suggested there was some minor VMDK corruption and that I storage motion to another datastore - which did resolve the issue.

But now I am having this issues on another large VM - nimble storage snaps.

I feel like there is more going on here. Can we get some attention on this??
wishr
Veteran
Posts: 3077
Liked: 453 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: Error: Exception of type 'System.OutOfMemoryException' was thrown.

Post by wishr » 1 person likes this post

Hi Lobo,

Thanks for updating. Regarding the new issue please raise an independent support case with our engineers or re-open the existing one to get it fully sorted.

Thanks
amarsaudon
Lurker
Posts: 2
Liked: never
Joined: Mar 10, 2020 5:05 pm
Full Name: Alex Marsaudon
Contact:

Re: Error: Exception of type 'System.OutOfMemoryException' was thrown.

Post by amarsaudon »

Just a heads up - support was unable to assist me with this exact issue. I ended up having to create a new Nimble Volume / VMware Datastore and doing a storage vmotion over on to it. Unfortunately this VM was 22TB, so this was a cumbersome exercise and I was fortunate to have had the surplus capacity to be able to do so.
Case# 04011801
Post Reply

Who is online

Users browsing this forum: Lewpy and 84 guests