Backup failure and warnings

Availability for the Always-On Enterprise

Backup failure and warnings

Veeam Logoby jfargo » Mon Apr 18, 2016 6:53 pm

I have an issue with my Veeam server and was hoping someone might have an idea.

I've opened support call #01723575 but we don't seem to be getting anywhere.

The Veeam backup server is a VM running Windows 2012 R2 and Veeam 9.0.0.1491. All proxies and repositories are also running Windows 2012 R2. I have 40 WAN sites with a repository and proxy sitting locally. I have approximately 52 backup jobs. Of those, 40 jobs backup remote sites to remote repositories once an hour that I retain for 14 days. I also have 2 backup copy jobs per job, one to copy to my tape server and one to copy to my datacenter for 180 day storage that run once a day.

I'm having jobs either failing or run with warnings. The warnings are sending an email but the failed jobs don't. The jobs are not failing or giving warning consistently, one hour is runs fine and then the next time I get the warning. The warning I'm getting is a commit catalog transaction issue. The Veeam server and the Enterprise Manager server both have over 500 GB of free disk space where the VBRCatalog folder is. I had guest indexing enabled on the jobs but I've since disabled it but I still get the warnings. I created a new backup job that has never had guest indexing enabled and I got a warning email for that job.

On the Backup server I'm also seeing Application errors in the event logs. The application error seems to coincide with the backup failures and warning emails. If I restart the backup server, the jobs seem to run fine for 15-24 hours before I start seeing the application errors and warning emails. I've tried building a new VM for the backup server and then reconnecting it to the existing database. It ran fine for 24 hours and then started pitching the same errors and warnings.

This weekend I created a scale out repository that included my tape out repository. In order to do that I had to disable all the backup copy jobs that were using that repository. While the tape out copy jobs were disabled, I got no application errors or warnings. This lasted the whole day, but as soon as the backup evacuation was complete and I re-enabled the copy jobs, I instantly started getting warning emails and application errors.

This is the warning message:
Failed to commit catalog transaction "D:\VBRCatalog\Publications\Upload\c3354745-99b1-4f65-9fd5-65fdb81a342f". Meta-file "D:\VBRCatalog\Publications\Upload\c3354745-99b1-4f65-9fd5-65fdb81a342f\PubCommands.txt" is unavailable. Retrieving the COM class factory for component with CLSID {4EF18917-F9D6-4A72-AEBF-8A4A88E99496} failed due to the following error: 80070583 Class does not exist. (Exception from HRESULT: 0x80070583).

This is the event log entry:
Code: Select all
Faulting application name: Veeam.Backup.Manager.exe, version: 9.0.0.1491, time stamp: 0x56df70e3
Faulting module name: KERNELBASE.dll, version: 6.3.9600.18233, time stamp: 0x56bb4ebb
Exception code: 0xc0000142
Fault offset: 0x00000000000ecdd0
Faulting process id: 0x4698
Faulting application start time: 0x01d19284c750a047
Faulting application path: D:\Program Files\Veeam\Backup and Replication\Backup\Veeam.Backup.Manager.exe
Faulting module path: KERNELBASE.dll
Report Id: 04fd2c91-fe78-11e5-80e0-005056961478
Faulting package full name:
Faulting package-relative application ID:

The next step the engineer wants me to do is create a new VM for backup and create a new database. I'd rather not do that if possible. We were having the same issue before we updated to 1491. Any ideas are appreciated.

Thanks,
John
jfargo
Novice
 
Posts: 6
Liked: never
Joined: Wed Jun 03, 2015 12:47 am
Full Name: John Fargo

Re: Backup failure and warnings

Veeam Logoby PTide » Tue Apr 19, 2016 10:38 am

Hi,

The next step the engineer wants me to do is create a new VM for backup and create a new database. I'd rather not do that if possible. We were having the same issue before we updated to 1491. Any ideas are appreciated.
You might want to backup your VBR sever configuration and use it to deploy another VBR server in a fast manner in order to check if a new VBR server will help. You don't need to delete or disable an existing one for that. If you still don't want to stick with that approach I suggest you to escalate the case for a deeper investigation by pressing "Talk to Manager" button.

Thank you.
PTide
Product Manager
 
Posts: 2740
Liked: 223 times
Joined: Tue May 19, 2015 1:46 pm

Re: Backup failure and warnings

Veeam Logoby jfargo » Wed Apr 20, 2016 4:28 pm

I created a new backup server with a new database. I'm still getting the warnings and application event log errors.

John
jfargo
Novice
 
Posts: 6
Liked: never
Joined: Wed Jun 03, 2015 12:47 am
Full Name: John Fargo

Re: Backup failure and warnings

Veeam Logoby it@nevastar.ru » Fri Apr 22, 2016 6:21 am

Hi.
I have same issue, with same COM class.
Backup jobs seems fine, but warning notification make me feel sad
Configuration:
veeem 9.0.0.902 (updated v7)
win2012r2
only local esxi hosts

I found some missing update 9.0.0.1491, if it will help i'll bee happy

thnx.
it@nevastar.ru
Lurker
 
Posts: 1
Liked: never
Joined: Fri Apr 22, 2016 6:01 am
Full Name: Виктор Ребячьих

Re: Backup failure and warnings

Veeam Logoby PTide » Fri Apr 22, 2016 3:39 pm

Hi,

I have same issue, with same COM class
I suggest you to open a case with support, because such an error is not something what we can troubleshoot via forum. Please don't forget to post your case ID.

Thank you.
PTide
Product Manager
 
Posts: 2740
Liked: 223 times
Joined: Tue May 19, 2015 1:46 pm

Re: Backup failure and warnings

Veeam Logoby jfargo » Fri Apr 22, 2016 11:33 pm

It might be something to do with how many backup copy jobs there are. Do you have a lot of copy jobs? I noticed that when I disabled 40 of my copy jobs, the errors/warnings stopped. We're still testing to see what the issue is.
#01723575

John
jfargo
Novice
 
Posts: 6
Liked: never
Joined: Wed Jun 03, 2015 12:47 am
Full Name: John Fargo

Re: Backup failure and warnings

Veeam Logoby Branimir » Fri Dec 02, 2016 2:22 pm

Case: 01989435

I work for MSP and I myself am seeing many HRESULT 0x80070583 errors. I narrowed them down to servers running Microsoft SQL Server.
Job error is the following: Unable to truncate Microsoft SQL Server transaction logs. Details: Failed to process 'TruncateSQLLog' command. CoInitialize() failed. HRESULT=0x80070583.

Additional information from VeeamGuestHelper on troubled VM:
Code: Select all
12/2/2016 12:19:03 AM  16844  INFO        RPC: explore SQL AlwaysOn.
12/2/2016 12:19:03 AM  16844                      Enumerating Microsoft SQL Server instances by service names.
12/2/2016 12:19:03 AM  16844                          SQL instance found: []
12/2/2016 12:19:03 AM  16844                      Enumerating Microsoft SQL Server instances by service names.. Ok.
12/2/2016 12:19:03 AM  16844                  Using default SQL provider 'sqloledb' to connect to SQL server
12/2/2016 12:19:03 AM  16844  WARN                   Code = 0x80070583
12/2/2016 12:19:03 AM  16844  WARN                   Code meaning = Class does not exist.
12/2/2016 12:19:03 AM  16844  WARN                   Source =
12/2/2016 12:19:03 AM  16844  WARN                   Description =
12/2/2016 12:19:03 AM  16844  WARN                COM error:  Code: 0x80070583
12/2/2016 12:19:03 AM  16844  WARN            Instance '' connection problem:    Code = 0x80070583
12/2/2016 12:19:03 AM  16844  WARN               Code meaning = Class does not exist.
12/2/2016 12:19:03 AM  16844  WARN               Source =
12/2/2016 12:19:03 AM  16844  WARN               Description =
12/2/2016 12:19:03 AM  16844  WARN           
12/2/2016 12:19:03 AM  16844  WARN            COM error:  Code: 0x80070583.
12/2/2016 12:19:03 AM  16844  INFO        RPC: explore SQL AlwaysOn.. Ok.
12/2/2016 12:19:03 AM  16844  INFO        RPC: get sql last backupsetId.
12/2/2016 12:19:03 AM  16844                  Login: X\veeamsvc
12/2/2016 12:19:03 AM  16844                      Enumerating Microsoft SQL Server instances by service names.
12/2/2016 12:19:03 AM  16844                          SQL instance found: []
12/2/2016 12:19:03 AM  16844                      Enumerating Microsoft SQL Server instances by service names.. Ok.
12/2/2016 12:19:03 AM  16844                  Using default SQL provider 'sqloledb' to connect to SQL server
12/2/2016 12:19:03 AM  16844  WARN                   Code = 0x80070583
12/2/2016 12:19:03 AM  16844  WARN                   Code meaning = Class does not exist.
12/2/2016 12:19:03 AM  16844  WARN                   Source =
12/2/2016 12:19:03 AM  16844  WARN                   Description =
12/2/2016 12:19:03 AM  16844  WARN                COM error:  Code: 0x80070583
12/2/2016 12:19:03 AM  16844  WARN            Instance '' connection problem:    Code = 0x80070583
12/2/2016 12:19:03 AM  16844  WARN               Code meaning = Class does not exist.
12/2/2016 12:19:03 AM  16844  WARN               Source =
12/2/2016 12:19:03 AM  16844  WARN               Description =
12/2/2016 12:19:03 AM  16844  WARN           
12/2/2016 12:19:03 AM  16844  WARN            COM error:  Code: 0x80070583, skipping...
12/2/2016 12:19:03 AM  16844                  Output:
12/2/2016 12:19:03 AM  16844                  <BackupsetIdContainer/>
12/2/2016 12:19:03 AM  16844  INFO        RPC: get sql last backupsetId.. Ok.
12/2/2016 12:19:03 AM  16844  INFO    Serializing HostAddrInfo.
12/2/2016 12:19:03 AM  16844  INFO    <HostAddrInfo Fqdn="NH01VELOCITY.X.X"><IPv4><Address Value="192.168.1.29"/></IPv4><IPv6/></HostAddrInfo>
12/2/2016 12:19:03 AM  16844  INFO        RPC: ExploreOracleInstances
12/2/2016 12:19:03 AM  16844  INFO        RPC: ExploreOracleInstances. Ok.
12/2/2016 12:19:03 AM  16844  INFO    Unregister index job
12/2/2016 12:19:03 AM  16844  INFO    Serializing HostAddrInfo.
12/2/2016 12:19:03 AM  16844  INFO    <HostAddrInfo Fqdn="NH01VELOCITY.X.X"><IPv4><Address Value="192.168.1.29"/></IPv4><IPv6/></HostAddrInfo>
12/2/2016 12:19:03 AM  16844  WARN    Failed to read vCenter database information. Cannot load the specified XML file: [C:\ProgramData\VMware\vCenterServer\cfg\vmware-vpx\vpxd.cfg].
12/2/2016 12:19:03 AM  16844  WARN    COM error: Not enough storage is available to complete this operation.
12/2/2016 12:19:03 AM  16844  WARN     Code: 0x1


Veeam engineer is suggesting that the problem is "storage", however no servers running Microsoft SQL have storage issues. Repository doesn't have storage space issues, and datastores on which VMs are running do not have storage issues. Path: C:\ProgramData\VMware\vCenterServer\cfg\vmware-vpx\vpxd.cfg doesn't even exist on servers which have above mentioned problem.

Reboot of VMs running Microsoft SQL server temporarily fixes this problem. We are currently running 9.5.0.711, but problem was experienced on previous versions of Veeam as well.

Any help is appreciated.

Branimir
Branimir
Lurker
 
Posts: 1
Liked: never
Joined: Mon Mar 28, 2016 4:52 pm
Full Name: Branimir Karajcic

Re: Backup failure and warnings

Veeam Logoby foggy » Fri Dec 02, 2016 3:27 pm

Hi Branimir, I've checked the OP's case, however, it doesn't contain any resolution, so I recommend to continue working on with your engineer and request case escalation, if required.
foggy
Veeam Software
 
Posts: 13899
Liked: 1012 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Backup failure and warnings

Veeam Logoby hoFFy » Fri Mar 03, 2017 2:29 pm

Branimir wrote:Case: 01989435

I work for MSP and I myself am seeing many HRESULT 0x80070583 errors. I narrowed them down to servers running Microsoft SQL Server.
Job error is the following: Unable to truncate Microsoft SQL Server transaction logs. Details: Failed to process 'TruncateSQLLog' command. CoInitialize() failed. HRESULT=0x80070583.
...



Has this ever been resolved? I'm experiencing the same problem / error message. Case #02053671
I've also tried to install a guest interaction proxy on the VM in question and I'm getting the error:
Code: Select all
Failed to call RPC function "Test Compatible": Error code: 0x80070583.
Cannot initialize COM runtime,
Error code: 0x80070583


And the Svc.VeeamInstaller.log on the guest VM says:

Code: Select all
[03.03.2017 11:31:44] <  4676> dpl| RPC: Getting version of the deployment service.
[03.03.2017 11:31:44] <  4676> dpl| RPC: Getting version  of the deployment dll.
[03.03.2017 11:31:44] <  4676> dpl|   Getting version dll...
[03.03.2017 11:31:44] <  4676> dpl|   Deployment dll [C:\Windows\Veeam\Backup\VeeamDeploymentDll.dll] version [9.5.0.823].
[03.03.2017 11:31:44] <  4676> dpl| RPC: Getting version of the deployment service.
[03.03.2017 11:31:44] <  4676> dpl| RPC: Getting platform of the deployment service.
[03.03.2017 11:31:44] <  4676> dpl| ERR |Failed to execute DoRpc command 'TestCompatible'.
[03.03.2017 11:31:44] <  4676> dpl| >>  | Error code: 0x80070583
[03.03.2017 11:31:44] <  4676> dpl| >>  |Cannot initialize COM runtime
[03.03.2017 11:31:44] <  4676> dpl| >>  |--tr:Failed to call DoRpc. CmdName: [TestCompatible].
[03.03.2017 11:31:44] <  4676> dpl| >>  |An exception was thrown from thread [4676].



So I believe the problem doesn't have anything to do with the SQL itself
VMCE 7 / 8 / 9, VCP-DC 5 / 5.5, MCITP:SA
Blog: machinewithoutbrain.de
hoFFy
Veeam ProPartner
 
Posts: 156
Liked: 25 times
Joined: Fri Apr 27, 2012 1:10 pm
Location: Germany / Lohne
Full Name: Sebastian Hoffmann

Re: Backup failure and warnings

Veeam Logoby foggy » Fri Mar 03, 2017 3:20 pm

Branimir's case was closed due to no activity from his side, so please continue working with your engineer on this issue.
foggy
Veeam Software
 
Posts: 13899
Liked: 1012 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson


Return to Veeam Backup & Replication



Who is online

Users browsing this forum: Bing [Bot] and 10 guests