Backups Fail with: 0x80070008 Cannot initialize COM runtime

[emilec]

Veeam Case: 01677539 & 01684114

This is a quick summary of this problem and discoveries so far. The TLDR version is: This appears to be an OS problem, so I reloaded Windows.

Windows 2008 R2 with B&R 8 patch 3

Examples of errors from the logs:

Code: Select all

[26.01.2016 06:25:00] <01> Error    Failed to call RPC function 'FcFindBegin'. InParams: '<InputArguments><SearchMask value="E:\Veeam" /></InputArguments>'. 
[26.01.2016 06:25:00] <01> Error    [VEEAM02] Failed to create directory 'E:\Veeam' (System.Exception)
[26.01.2016 06:25:00] <01> Error       at Veeam.Backup.ProxyProvider.CProxyFileCommander.CreateDir(String dirPath)
[26.01.2016 06:25:00] <01> Error       at Veeam.Backup.Core.CBackupRepositoryEx.CreateRootDir()
[26.01.2016 06:25:00] <01> Error     Error code: 0x80070008
[26.01.2016 06:25:00] <01> Error    Cannot initialize COM runtime

I was still able to create directories in that folder from Windows Explorer, so not a file permission problem.

Code: Select all

[31.01.2016 22:28:26] <18> Error    Failed to call RPC function 'BeginStopAgent'. InParams: '<InputArguments><agentId value="a41e3d05-c532-4a87-9bca-7c39dc3b6d18" /></InputArguments>'.
[31.01.2016 22:28:26] <18> Error     Error code: 0x80070008
[31.01.2016 22:28:26] <18> Error    Cannot initialize COM runtime (System.ApplicationException)
[31.01.2016 22:28:26] <18> Error       at VeeamProxyClient.RegenerateCLRException(exception* excBase)
[31.01.2016 22:28:26] <18> Error       at VeeamProxyClient.CCliProxyRpcInvoker.DoRpc(String cmdName, String inXml, String& outXml)
[31.01.2016 22:28:26] <18> Error       at Veeam.Backup.ProxyProvider.CProxyRpcInvoker.Call(String methodName, CProxyInvokeInputArg inputArgs, Boolean secureCall)

A restart of the DCOM services won't help, it simply produces a new error when you try to run a backup.

Code: Select all

2016/01/26 09:55:55 AM :: Getting VM info from vSphere
2016/01/26 10:00:58 AM :: Error: Timeout to start agent
2016/01/26 10:00:59 AM :: Network traffic verification detected no corrupted blocks
2016/01/26 10:00:59 AM :: Processing finished with errors at 2016/01/26 10:00:59 AM

One of the Veeam support steps was to run a system integrity check (sfc /scannow), but that passed 100%

Only thing that does help is a reboot of the server. However often the first run of the backup will abandon the existing backup chain and start a new full backup in a new directory (would love to know the reason). The only way to prevent this from happening is to edit every single job, recalculate the size and relink the job to the same target (it will generate a warning when you do this). You can also use the relink trick to get your job back onto their original backup chain if a new full backup has been made. First move/delete the duplicates, rescan the repository and edit the jobs. It's possible that you only need to do the relink, I did the size recalculation based on another forum post I'd seen. YMMV.

I also discovered after reloading this server that another Windows 2008 R2 was experiencing the same problem, so it's possible a recent Windows update is to blame. In any event I was planning on moving to 2012 R2 and the Veeam migration procedure is pretty straight forward so no point in spending more time trying to solve this annoying issue on an old OS.

[severt]

I have the exact same problem, but with Veeam 7 running the latest patch on Windows 2008 R2.
Unfortunately Veeam support wasn't able to help me and I was instructed to update to Veeam 8.
However, I don't have an active support contract and I don't want my client to pay $1500 for something that shouldn't occur in the first place. (no matter who'se fault, Microsoft or Veeam).
It sounds like a little corruption between Microsoft and Veeam so let's find out how to solve this...

[severt]

Any update on this?

[Gostev]

According to the support case notes, moving backup server to a fresh Windows installation has fixed this error.

[severt]

How is that a solution that I can offer to a client?
Just because an error occurs that is probably caused by a Windows update the solution is to move to a fresh Windows installation?

[PTide]

Hi,

How is that a solution that I can offer to a client?

There is an existing post where an error that is similar to yours has been mentioned, you might want to try rebooting VBR server. Unfortunately that COM error is not very common so there are two possible directions to take if the reboot does not help:

1. Wait for someone to post here a solution
2. Quickly setup another VBR server leveraging configuration backup feature

Considering all circumstances the second approach appears to be the fastest. Please also take a look at this guide and don't hesitate to open a support case for assistance in case any problems occur. If we find out how to fix that error without moving to a new server we will update this topic immediately.

Thank you.

[Gostev]

How is that a solution that I can offer to a client?
Just because an error occurs that is probably caused by a Windows update the solution is to move to a fresh Windows installation?

Well, at least we know for sure that it helped another client.

[severt]

Rebooting the server did help, but only for 1 or 2 days... Then the problem occured again.
And the problem did appear after I updated Windows with the latest "automatic Windows updates".
There were a bunch so I can't really figure out which one is causing the issue.

Installing it to a new Windows Server 2008 R2 might give the same issue if it's related to a Windows update.
Currently Veeam is installed on a VM itself in combination with accounting software Exact.

Would you suggest creating a new clean VM for Veeam only?
I believe the "best practice" is to even install Veeam outside of the VM environment, but is that really necessary?
Because that would mean investing into another "system" to get Veeam running.

Looking forward to your reply.

Thank you.

[Gostev]

No need for a new system, just reinstall Windows on whatever server Veeam is currently running.
And be sure to format the disks, don't just reinstall on top of the existing OS installation.

[severt]

Now that I looked at this server, there isn't room to add a new VM with Windows Server because all hdd space is in use.
Reinstalling the current VM where Veeam is running isn't possible because of some other applications that run on there.

[mriksman]

This may or may not help - but I started getting these errors after I started using a custom Zabbix Powershell script that uses the Powershell Commandlets.

I noticed HUNDREDS of 'conhost.exe' in the Task Manager Services tab and 'Console Window Host' and 'Veeam.Backup.Satellite' in the Processes tab. Something related to the script has caused this system to not close these processes down.

[Gostev]

Michael, your issue is definitely unrelated to this [very old] post. Judging on your error, it is a known issue and simply installing Update 3a should fix it. Thanks!

[DannyMurphy]

Hi,

I know this is old but we're getting the same issue as the OP on 9.5U3a. Does anyone know if this issue is fixed in U4 or is there an alterative fix than a reinstall of Windows?

[foggy]

Hi Danny, judging on the above, I'm not sure there's something to fix on the Veeam side. You can check with our support engineers though.

[mille]

Hi all, yesterday I have the same failure on Windows Server 2016 and Veeam 9.5 update 4, trying to backup two oracle servers in one job.
One of these backup clients (both VMware-VMs) was already successfully backed up and the second was new. After adding the new client in the job I got first an error to backup the oracle database of the new client. Backup of the other clint was successful.
After correcting some user information for the oracle access data I got this 0x80070008 error for the new client and the other client also fails. Reboot of the new client was without any effectiveness.
Then I logged me in per RDP with the backup user which was configured in the job without problems. Next I started the backup job again and the backup was successful.
I have no explanation for this but hope it helps others.

[Bennon Maina]

I am having the same issue on a physical server managed by VBR (9.5 - 4a) and it only started after the update.
One thing I've noticed is that it appears 'Offline' on the Physical & Cloud Infrastructure under Inventory and last seen status is less than a day ago, even after restarting any failed writers.

I know if I restart it, the backups will run okay and fail after a couple of days, it's been a vicious cycle of reboots and backup retries. It's patched (a 2012 R2 Std).
Any help appreciated.

[foggy]

Hi Bennon, please contact technical support for investigation. Thanks!

[isolonas]

i has the same issue, i restart "Veeam Backup Service" and until now its working