Failed to mount - bad allocation?

[atmarx]

I'm still testing out B&R community edition -- I got a full backup from our file server, but when I attempted to mount it for FLR recovery, it sat there for about 20 minutes before giving me an error "bad allocation". Here's what I see in the logs. The backup showed successful. Any ideas? Did I mess something up during the backup process? I'm trying to run the restore from the B&R server which is also the repo (attached via DAS). Windows Dedup is in use on the VMs being backed up (running Windows 2016), and is also installed on the B&R server (running Windows 2019). All servers are patched up to April 2019.


[15.05.2019 14:33:27] <04> Error Failed to mount remote disks for the file level restore
[15.05.2019 14:33:27] <04> Error bad allocation (Veeam.Backup.Common.CCppComponentException)
[15.05.2019 14:33:27] <04> Error in c++: Failed to load grain tables. Offset: "1347430912". Number of tables: "655360".
[15.05.2019 14:33:27] <04> Error in c++: Failed to load monosparse metadata.
[15.05.2019 14:33:27] <04> Error in c++: Failed to open monosparse disk "C:\Windows\system32\config\systemprofile\AppData\Local\mount.4a1db78e-d0ed-4df3-8b60-12234e842ddc.cache".
[15.05.2019 14:33:27] <04> Error in c++: Failed to create new disk cache "C:\Windows\system32\config\systemprofile\AppData\Local\mount.4a1db78e-d0ed-4df3-8b60-12234e842ddc.cache".
[15.05.2019 14:33:27] <04> Error in c++: Failed mount filesystem for the FSR restore.
[15.05.2019 14:33:27] <04> Error Agent failed to process method {Mount.GenericMount}. (Veeam.Backup.Common.CCppComponentException)
[15.05.2019 14:33:27] <04> Error at Veeam.Backup.AgentProvider.CClientAgentProtocol.Invoke(String command, CVcpCommandArgs inArgs, Boolean noLog, Boolean agentNoLog)
[15.05.2019 14:33:27] <04> Error at Veeam.Backup.AgentProvider.CBackupClient.Invoke(String command, CVcpCommandArgs inArgs, Boolean noLog)
[15.05.2019 14:33:27] <04> Error --- End of stack trace from previous location where exception was thrown ---
[15.05.2019 14:33:27] <04> Error at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
[15.05.2019 14:33:27] <04> Error at Veeam.Backup.Common.ExceptionFactory.ThrowNecessaryAggregateException(IEnumerable`1 exceptionsCollection)
[15.05.2019 14:33:27] <04> Error at Veeam.Backup.AgentProvider.CBackupClientImpl.ConstructException(Exception exception, String error)
[15.05.2019 14:33:27] <04> Error at Veeam.Backup.AgentProvider.CBackupClient.ConstructException(Exception ex, String error, Object[] args)
[15.05.2019 14:33:27] <04> Error at Veeam.Backup.AgentProvider.CBackupClient.OnInvokeError(Exception e, String command, CVcpCommandArgs inArgs)
[15.05.2019 14:33:27] <04> Error at Veeam.Backup.AgentProvider.CBackupClient.Invoke(String command, CVcpCommandArgs inArgs, Boolean noLog)
[15.05.2019 14:33:27] <04> Error at Veeam.Backup.AgentProvider.CBackupClient.MountGenericMount(IFsrMountParams mountParams, IFsrDataSourceParams sourceParams, IEnumerable`1 processingParams)
[15.05.2019 14:33:27] <04> Error at Veeam.Backup.Core.CVeeamFctMounter.MountAsFlat(IBackupClient mountClient, String flatMountPath, IFsrDataSourceParams dataSource, IEnumerable`1 processingParams, CAsyncMountModeSelector mountMode)
[15.05.2019 14:33:27] <04> Error at Veeam.Backup.Core.CVeeamFctMounter.MountDisks(IBackupClient mountClient, IFsrDataSourceParams dataSource, IEnumerable`1 processingParams, CAsyncMountModeSelector mountMode)
[15.05.2019 14:33:27] <04> Error at Veeam.Backup.Core.CVeeamFctMounter.Mount(IMountSource source, IEnumerable`1 processingParams, CAsyncMountModeSelector mountMode)
[15.05.2019 14:33:27] <04> Error at Veeam.Backup.Core.CFlrDisksMounter.Mount(Boolean isShieldedVm, Boolean isKdsUsed, CAsyncMountModeSelector mountMode)
[15.05.2019 14:33:27] <04> Error Mount was failed
[15.05.2019 14:33:27] <04> Error bad allocation (Veeam.Backup.Common.CCppComponentException)
[15.05.2019 14:33:27] <04> Error in c++: Failed to load grain tables. Offset: "1347430912". Number of tables: "655360".
[15.05.2019 14:33:27] <04> Error in c++: Failed to load monosparse metadata.
[15.05.2019 14:33:27] <04> Error in c++: Failed to open monosparse disk "C:\Windows\system32\config\systemprofile\AppData\Local\mount.4a1db78e-d0ed-4df3-8b60-12234e842ddc.cache".
[15.05.2019 14:33:27] <04> Error in c++: Failed to create new disk cache "C:\Windows\system32\config\systemprofile\AppData\Local\mount.4a1db78e-d0ed-4df3-8b60-12234e842ddc.cache".
[15.05.2019 14:33:27] <04> Error in c++: Failed mount filesystem for the FSR restore.
[15.05.2019 14:33:27] <04> Error Agent failed to process method {Mount.GenericMount}. (Veeam.Backup.Common.CCppComponentException)
[15.05.2019 14:33:27] <04> Error at Veeam.Backup.AgentProvider.CClientAgentProtocol.Invoke(String command, CVcpCommandArgs inArgs, Boolean noLog, Boolean agentNoLog)
[15.05.2019 14:33:27] <04> Error at Veeam.Backup.AgentProvider.CBackupClient.Invoke(String command, CVcpCommandArgs inArgs, Boolean noLog)
[15.05.2019 14:33:27] <04> Error --- End of stack trace from previous location where exception was thrown ---
[15.05.2019 14:33:27] <04> Error at System.Runtime.ExceptionServices.ExceptionDispatchInfo.Throw()
[15.05.2019 14:33:27] <04> Error at Veeam.Backup.Common.ExceptionFactory.ThrowNecessaryAggregateException(IEnumerable`1 exceptionsCollection)
[15.05.2019 14:33:27] <04> Error at Veeam.Backup.AgentProvider.CBackupClientImpl.ConstructException(Exception exception, String error)
[15.05.2019 14:33:27] <04> Error at Veeam.Backup.AgentProvider.CBackupClient.ConstructException(Exception ex, String error, Object[] args)
[15.05.2019 14:33:27] <04> Error at Veeam.Backup.AgentProvider.CBackupClient.OnInvokeError(Exception e, String command, CVcpCommandArgs inArgs)
[15.05.2019 14:33:27] <04> Error at Veeam.Backup.AgentProvider.CBackupClient.Invoke(String command, CVcpCommandArgs inArgs, Boolean noLog)
[15.05.2019 14:33:27] <04> Error at Veeam.Backup.AgentProvider.CBackupClient.MountGenericMount(IFsrMountParams mountParams, IFsrDataSourceParams sourceParams, IEnumerable`1 processingParams)
[15.05.2019 14:33:27] <04> Error at Veeam.Backup.Core.CVeeamFctMounter.MountAsFlat(IBackupClient mountClient, String flatMountPath, IFsrDataSourceParams dataSource, IEnumerable`1 processingParams, CAsyncMountModeSelector mountMode)
[15.05.2019 14:33:27] <04> Error at Veeam.Backup.Core.CVeeamFctMounter.MountDisks(IBackupClient mountClient, IFsrDataSourceParams dataSource, IEnumerable`1 processingParams, CAsyncMountModeSelector mountMode)
[15.05.2019 14:33:27] <04> Error at Veeam.Backup.Core.CVeeamFctMounter.Mount(IMountSource source, IEnumerable`1 processingParams, CAsyncMountModeSelector mountMode)
[15.05.2019 14:33:27] <04> Error at Veeam.Backup.Core.CFlrDisksMounter.Mount(Boolean isShieldedVm, Boolean isKdsUsed, CAsyncMountModeSelector mountMode)
[15.05.2019 14:33:27] <04> Error at Veeam.Backup.RestoreServiceLib.CWinOibMountSession.Mount()
[15.05.2019 14:33:27] <04> Error at Veeam.Backup.RestoreServiceLib.COibMounter.DoMount(IDbMethodsInterceptor methodsInterceptor)
[15.05.2019 14:33:27] <40> Error Failed to save MountResult to lease's auxdata.
[15.05.2019 14:33:27] <40> Error bad allocation (Veeam.Backup.Common.CCppComponentException)
[15.05.2019 14:33:27] <40> Error in c++: Failed to load grain tables. Offset: "1347430912". Number of tables: "655360".
[15.05.2019 14:33:27] <40> Error in c++: Failed to load monosparse metadata.
[15.05.2019 14:33:27] <40> Error in c++: Failed to open monosparse disk "C:\Windows\system32\config\systemprofile\AppData\Local\mount.4a1db78e-d0ed-4df3-8b60-12234e842ddc.cache".
[15.05.2019 14:33:27] <40> Error in c++: Failed to create new disk cache "C:\Windows\system32\config\systemprofile\AppData\Local\mount.4a1db78e-d0ed-4df3-8b60-12234e842ddc.cache".
[15.05.2019 14:33:27] <40> Error in c++: Failed mount filesystem for the FSR restore.
[15.05.2019 14:33:27] <40> Error Agent failed to process method {Mount.GenericMount}. (Veeam.Backup.Common.CCppComponentException)
[15.05.2019 14:33:27] <40> Error at Veeam.Backup.RestoreServiceLib.COibMounter.GetMountResult()
[15.05.2019 14:33:27] <40> Error at Veeam.Backup.RestoreServiceLib.COibMount.GetMountResult()
[15.05.2019 14:33:27] <40> Error at Veeam.Backup.RestoreServiceLib.COibMountManager.GetMountResult(CItemRestoreSession session, COibMountUid mountId)
[15.05.2019 14:33:27] <40> Error at Veeam.Backup.RestoreServiceLib.CItemRestoreManager.SaveMountResultToLeaseSafe(CRestoreSessionUid sessionId, COibMountUid mountId)
[15.05.2019 14:33:27] <39> Error Failed to get OIB mount result. SessionId: [8031a2a6d54242a8ac573548f1202c7f], MountId: [888d0a25cac7477f8cbaab5d8c9007da]
[15.05.2019 14:33:27] <39> Error bad allocation (Veeam.Backup.Common.CCppComponentException)
[15.05.2019 14:33:27] <39> Error in c++: Failed to load grain tables. Offset: "1347430912". Number of tables: "655360".
[15.05.2019 14:33:27] <39> Error in c++: Failed to load monosparse metadata.
[15.05.2019 14:33:27] <39> Error in c++: Failed to open monosparse disk "C:\Windows\system32\config\systemprofile\AppData\Local\mount.4a1db78e-d0ed-4df3-8b60-12234e842ddc.cache".
[15.05.2019 14:33:27] <39> Error in c++: Failed to create new disk cache "C:\Windows\system32\config\systemprofile\AppData\Local\mount.4a1db78e-d0ed-4df3-8b60-12234e842ddc.cache".
[15.05.2019 14:33:27] <39> Error in c++: Failed mount filesystem for the FSR restore.
[15.05.2019 14:33:27] <39> Error Agent failed to process method {Mount.GenericMount}. (Veeam.Backup.Common.CCppComponentException)
[15.05.2019 14:33:27] <39> Error at Veeam.Backup.RestoreServiceLib.COibMounter.GetMountResult()
[15.05.2019 14:33:27] <39> Error at Veeam.Backup.RestoreServiceLib.COibMount.GetMountResult()
[15.05.2019 14:33:27] <39> Error at Veeam.Backup.RestoreServiceLib.COibMountManager.GetMountResult(CItemRestoreSession session, COibMountUid mountId)
[15.05.2019 14:33:27] <39> Error at Veeam.Backup.RestoreServiceLib.CItemRestoreMgmtServiceStub.Veeam.Backup.Interaction.MountService.IItemRestoreManagementService.GetOibMountResult(CRestoreSessionUid sessionId, COibMountUid mountId)

[Mike Resseler]

Hi Andrew,

From what I can see, I have a feeling that there is something wrong with the permissions and that our service to mount the disks has an access denied or something like that. I would advice you (even with community edition) to create a support call for this. Our engineers will be much faster in figuring out where the actual issue is. (But I do think it is a permission issue though...). Post the support case ID here and let us know the outcome after the investigation

Thanks
Mike

[atmarx]

I blew the backup away and started fresh last night. Since I'm already backing up deduped data, I turned off compression and inline dedup to see if it would increase throughput (it didn't). I submitted a ticket with a copy of the logs I pulled right after I posted yesterday and included that. I'll know in a few hours if the new job suffers the same issues. The ticket # is 03564153. Thanks for taking a look.

[Mike Resseler]

Thanks Andrew. Keep us informed around the outcome when the engineers have looked through the logs

[atmarx]

Finally got a resolution to this -- forcing the use of the 64-bit agent (and providing lots of RAM) was required to let the Windows deduplicated NTFS volumes mount. The one file server with ~60 attached VHDX files spiked the RAM usage from 12GB to 56GB during the mounting process (took around 30 minutes), but it worked!

For anyone else who runs into a similar error, the fix was to add the following registry entry to the mount server (and reboot / restart the Veeam service):

HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication\
Name: Force64BitAgent
Type: REG_DWORD
Value: 1

[Mike Resseler]

Hi Andrew,

Thanks for posting the resolution here. My coffee almost came out of my nose reading that you have about 60 volumes that you need to attach... That will indeed grab quite some resources... Especially since all that dedup metadata needs to be read.

[atmarx]

Yeah, it was an interesting solution to how we parse out storage for our various research groups. Every group gets a 10TB VHDX created by default, and then their various shared directories are created within that volume. That way, any file system issues are confined to a single group, and the groups are able to gain the full benefit of dedup within their volume. I've been amazed at how much duplication there is within these datasets -- we have one group with 25TB of data deduped down to 6TB, but most hover around 50% savings. It's allowed us to do a lot with 2x 200TB storage arrays.

Previously, I was using DPM to (attempt to) back these up, but I spent more time cajoling and babysitting it than I wanted. I keep pinching myself when I check the Veeam backups and see that, for the 20th time in a row, it's backed up completely with no issues. I couldn't have picked a better version of Veeam to get started with -- that BitLooker functionality saves me a ton of space and really extends the full benefit of the existing dedup right into the backups.

[Mike Resseler]

Andrew!

Thanks for your feedback. Really appreciated. Still think that you have a special solution though . As I said above, 60 volumes that needs to get mounted and all the dedupe metadata that needs to be read. Doing restore means for you probably a beefy server that can handle it. However, the upside you mentioned. Lots of savings. And to me the real cool thing... You have free dedupe, and you can actually "attach" them to another server. Which is kind of portable dedupe