Comprehensive data protection for all workloads
Post Reply
jfargo
Novice
Posts: 9
Liked: never
Joined: Jun 03, 2015 12:47 am
Full Name: John Fargo
Contact:

Backup failure and warnings

Post by jfargo »

I have an issue with my Veeam server and was hoping someone might have an idea.

I've opened support call #01723575 but we don't seem to be getting anywhere.

The Veeam backup server is a VM running Windows 2012 R2 and Veeam 9.0.0.1491. All proxies and repositories are also running Windows 2012 R2. I have 40 WAN sites with a repository and proxy sitting locally. I have approximately 52 backup jobs. Of those, 40 jobs backup remote sites to remote repositories once an hour that I retain for 14 days. I also have 2 backup copy jobs per job, one to copy to my tape server and one to copy to my datacenter for 180 day storage that run once a day.

I'm having jobs either failing or run with warnings. The warnings are sending an email but the failed jobs don't. The jobs are not failing or giving warning consistently, one hour is runs fine and then the next time I get the warning. The warning I'm getting is a commit catalog transaction issue. The Veeam server and the Enterprise Manager server both have over 500 GB of free disk space where the VBRCatalog folder is. I had guest indexing enabled on the jobs but I've since disabled it but I still get the warnings. I created a new backup job that has never had guest indexing enabled and I got a warning email for that job.

On the Backup server I'm also seeing Application errors in the event logs. The application error seems to coincide with the backup failures and warning emails. If I restart the backup server, the jobs seem to run fine for 15-24 hours before I start seeing the application errors and warning emails. I've tried building a new VM for the backup server and then reconnecting it to the existing database. It ran fine for 24 hours and then started pitching the same errors and warnings.

This weekend I created a scale out repository that included my tape out repository. In order to do that I had to disable all the backup copy jobs that were using that repository. While the tape out copy jobs were disabled, I got no application errors or warnings. This lasted the whole day, but as soon as the backup evacuation was complete and I re-enabled the copy jobs, I instantly started getting warning emails and application errors.

This is the warning message:
Failed to commit catalog transaction "D:\VBRCatalog\Publications\Upload\c3354745-99b1-4f65-9fd5-65fdb81a342f". Meta-file "D:\VBRCatalog\Publications\Upload\c3354745-99b1-4f65-9fd5-65fdb81a342f\PubCommands.txt" is unavailable. Retrieving the COM class factory for component with CLSID {4EF18917-F9D6-4A72-AEBF-8A4A88E99496} failed due to the following error: 80070583 Class does not exist. (Exception from HRESULT: 0x80070583).

This is the event log entry:

Code: Select all

Faulting application name: Veeam.Backup.Manager.exe, version: 9.0.0.1491, time stamp: 0x56df70e3
Faulting module name: KERNELBASE.dll, version: 6.3.9600.18233, time stamp: 0x56bb4ebb
Exception code: 0xc0000142
Fault offset: 0x00000000000ecdd0
Faulting process id: 0x4698
Faulting application start time: 0x01d19284c750a047
Faulting application path: D:\Program Files\Veeam\Backup and Replication\Backup\Veeam.Backup.Manager.exe
Faulting module path: KERNELBASE.dll
Report Id: 04fd2c91-fe78-11e5-80e0-005056961478
Faulting package full name: 
Faulting package-relative application ID: 
The next step the engineer wants me to do is create a new VM for backup and create a new database. I'd rather not do that if possible. We were having the same issue before we updated to 1491. Any ideas are appreciated.

Thanks,
John
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Backup failure and warnings

Post by PTide »

Hi,
The next step the engineer wants me to do is create a new VM for backup and create a new database. I'd rather not do that if possible. We were having the same issue before we updated to 1491. Any ideas are appreciated.
You might want to backup your VBR sever configuration and use it to deploy another VBR server in a fast manner in order to check if a new VBR server will help. You don't need to delete or disable an existing one for that. If you still don't want to stick with that approach I suggest you to escalate the case for a deeper investigation by pressing "Talk to Manager" button.

Thank you.
jfargo
Novice
Posts: 9
Liked: never
Joined: Jun 03, 2015 12:47 am
Full Name: John Fargo
Contact:

Re: Backup failure and warnings

Post by jfargo »

I created a new backup server with a new database. I'm still getting the warnings and application event log errors.

John
it@nevastar.ru
Lurker
Posts: 1
Liked: never
Joined: Apr 22, 2016 6:01 am
Full Name: Виктор Ребячьих
Contact:

Re: Backup failure and warnings

Post by it@nevastar.ru »

Hi.
I have same issue, with same COM class.
Backup jobs seems fine, but warning notification make me feel sad
Configuration:
veeem 9.0.0.902 (updated v7)
win2012r2
only local esxi hosts

I found some missing update 9.0.0.1491, if it will help i'll bee happy

thnx.
PTide
Product Manager
Posts: 6408
Liked: 724 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Backup failure and warnings

Post by PTide »

Hi,
I have same issue, with same COM class
I suggest you to open a case with support, because such an error is not something what we can troubleshoot via forum. Please don't forget to post your case ID.

Thank you.
jfargo
Novice
Posts: 9
Liked: never
Joined: Jun 03, 2015 12:47 am
Full Name: John Fargo
Contact:

Re: Backup failure and warnings

Post by jfargo »

It might be something to do with how many backup copy jobs there are. Do you have a lot of copy jobs? I noticed that when I disabled 40 of my copy jobs, the errors/warnings stopped. We're still testing to see what the issue is.
#01723575

John
Branimir
Lurker
Posts: 2
Liked: 1 time
Joined: Mar 28, 2016 4:52 pm
Full Name: Branimir Karajcic
Contact:

Re: Backup failure and warnings

Post by Branimir »

Case: 01989435

I work for MSP and I myself am seeing many HRESULT 0x80070583 errors. I narrowed them down to servers running Microsoft SQL Server.
Job error is the following: Unable to truncate Microsoft SQL Server transaction logs. Details: Failed to process 'TruncateSQLLog' command. CoInitialize() failed. HRESULT=0x80070583.

Additional information from VeeamGuestHelper on troubled VM:

Code: Select all

12/2/2016 12:19:03 AM  16844  INFO        RPC: explore SQL AlwaysOn.
12/2/2016 12:19:03 AM  16844                      Enumerating Microsoft SQL Server instances by service names.
12/2/2016 12:19:03 AM  16844                          SQL instance found: []
12/2/2016 12:19:03 AM  16844                      Enumerating Microsoft SQL Server instances by service names.. Ok.
12/2/2016 12:19:03 AM  16844                  Using default SQL provider 'sqloledb' to connect to SQL server
12/2/2016 12:19:03 AM  16844  WARN                	Code = 0x80070583
12/2/2016 12:19:03 AM  16844  WARN                	Code meaning = Class does not exist.
12/2/2016 12:19:03 AM  16844  WARN                	Source = 
12/2/2016 12:19:03 AM  16844  WARN                	Description = 
12/2/2016 12:19:03 AM  16844  WARN                COM error:  Code: 0x80070583
12/2/2016 12:19:03 AM  16844  WARN            Instance '' connection problem: 	Code = 0x80070583
12/2/2016 12:19:03 AM  16844  WARN            	Code meaning = Class does not exist.
12/2/2016 12:19:03 AM  16844  WARN            	Source = 
12/2/2016 12:19:03 AM  16844  WARN            	Description = 
12/2/2016 12:19:03 AM  16844  WARN            
12/2/2016 12:19:03 AM  16844  WARN            COM error:  Code: 0x80070583.
12/2/2016 12:19:03 AM  16844  INFO        RPC: explore SQL AlwaysOn.. Ok.
12/2/2016 12:19:03 AM  16844  INFO        RPC: get sql last backupsetId.
12/2/2016 12:19:03 AM  16844                  Login: X\veeamsvc
12/2/2016 12:19:03 AM  16844                      Enumerating Microsoft SQL Server instances by service names.
12/2/2016 12:19:03 AM  16844                          SQL instance found: []
12/2/2016 12:19:03 AM  16844                      Enumerating Microsoft SQL Server instances by service names.. Ok.
12/2/2016 12:19:03 AM  16844                  Using default SQL provider 'sqloledb' to connect to SQL server
12/2/2016 12:19:03 AM  16844  WARN                	Code = 0x80070583
12/2/2016 12:19:03 AM  16844  WARN                	Code meaning = Class does not exist.
12/2/2016 12:19:03 AM  16844  WARN                	Source = 
12/2/2016 12:19:03 AM  16844  WARN                	Description = 
12/2/2016 12:19:03 AM  16844  WARN                COM error:  Code: 0x80070583
12/2/2016 12:19:03 AM  16844  WARN            Instance '' connection problem: 	Code = 0x80070583
12/2/2016 12:19:03 AM  16844  WARN            	Code meaning = Class does not exist.
12/2/2016 12:19:03 AM  16844  WARN            	Source = 
12/2/2016 12:19:03 AM  16844  WARN            	Description = 
12/2/2016 12:19:03 AM  16844  WARN            
12/2/2016 12:19:03 AM  16844  WARN            COM error:  Code: 0x80070583, skipping...
12/2/2016 12:19:03 AM  16844                  Output:
12/2/2016 12:19:03 AM  16844                  <BackupsetIdContainer/>
12/2/2016 12:19:03 AM  16844  INFO        RPC: get sql last backupsetId.. Ok.
12/2/2016 12:19:03 AM  16844  INFO    Serializing HostAddrInfo.
12/2/2016 12:19:03 AM  16844  INFO    <HostAddrInfo Fqdn="NH01VELOCITY.X.X"><IPv4><Address Value="192.168.1.29"/></IPv4><IPv6/></HostAddrInfo>
12/2/2016 12:19:03 AM  16844  INFO        RPC: ExploreOracleInstances
12/2/2016 12:19:03 AM  16844  INFO        RPC: ExploreOracleInstances. Ok.
12/2/2016 12:19:03 AM  16844  INFO    Unregister index job
12/2/2016 12:19:03 AM  16844  INFO    Serializing HostAddrInfo.
12/2/2016 12:19:03 AM  16844  INFO    <HostAddrInfo Fqdn="NH01VELOCITY.X.X"><IPv4><Address Value="192.168.1.29"/></IPv4><IPv6/></HostAddrInfo>
12/2/2016 12:19:03 AM  16844  WARN    Failed to read vCenter database information. Cannot load the specified XML file: [C:\ProgramData\VMware\vCenterServer\cfg\vmware-vpx\vpxd.cfg].
12/2/2016 12:19:03 AM  16844  WARN    COM error: Not enough storage is available to complete this operation.
12/2/2016 12:19:03 AM  16844  WARN     Code: 0x1
Veeam engineer is suggesting that the problem is "storage", however no servers running Microsoft SQL have storage issues. Repository doesn't have storage space issues, and datastores on which VMs are running do not have storage issues. Path: C:\ProgramData\VMware\vCenterServer\cfg\vmware-vpx\vpxd.cfg doesn't even exist on servers which have above mentioned problem.

Reboot of VMs running Microsoft SQL server temporarily fixes this problem. We are currently running 9.5.0.711, but problem was experienced on previous versions of Veeam as well.

Any help is appreciated.

Branimir
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Backup failure and warnings

Post by foggy »

Hi Branimir, I've checked the OP's case, however, it doesn't contain any resolution, so I recommend to continue working on with your engineer and request case escalation, if required.
hoFFy
Service Provider
Posts: 183
Liked: 40 times
Joined: Apr 27, 2012 1:10 pm
Full Name: Sebastian Hoffmann
Location: Germany / Lohne
Contact:

Re: Backup failure and warnings

Post by hoFFy »

Branimir wrote:Case: 01989435

I work for MSP and I myself am seeing many HRESULT 0x80070583 errors. I narrowed them down to servers running Microsoft SQL Server.
Job error is the following: Unable to truncate Microsoft SQL Server transaction logs. Details: Failed to process 'TruncateSQLLog' command. CoInitialize() failed. HRESULT=0x80070583.
...

Has this ever been resolved? I'm experiencing the same problem / error message. Case #02053671
I've also tried to install a guest interaction proxy on the VM in question and I'm getting the error:

Code: Select all

Failed to call RPC function "Test Compatible": Error code: 0x80070583.
Cannot initialize COM runtime,
Error code: 0x80070583
And the Svc.VeeamInstaller.log on the guest VM says:

Code: Select all

[03.03.2017 11:31:44] <  4676> dpl| RPC: Getting version of the deployment service.
[03.03.2017 11:31:44] <  4676> dpl| RPC: Getting version  of the deployment dll.
[03.03.2017 11:31:44] <  4676> dpl|   Getting version dll...
[03.03.2017 11:31:44] <  4676> dpl|   Deployment dll [C:\Windows\Veeam\Backup\VeeamDeploymentDll.dll] version [9.5.0.823].
[03.03.2017 11:31:44] <  4676> dpl| RPC: Getting version of the deployment service.
[03.03.2017 11:31:44] <  4676> dpl| RPC: Getting platform of the deployment service.
[03.03.2017 11:31:44] <  4676> dpl| ERR |Failed to execute DoRpc command 'TestCompatible'.
[03.03.2017 11:31:44] <  4676> dpl| >>  | Error code: 0x80070583
[03.03.2017 11:31:44] <  4676> dpl| >>  |Cannot initialize COM runtime
[03.03.2017 11:31:44] <  4676> dpl| >>  |--tr:Failed to call DoRpc. CmdName: [TestCompatible].
[03.03.2017 11:31:44] <  4676> dpl| >>  |An exception was thrown from thread [4676].


So I believe the problem doesn't have anything to do with the SQL itself
VMCE 7 / 8 / 9, VCP-DC 5 / 5.5 / 6, MCITP:SA
Blog: machinewithoutbrain.de
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Backup failure and warnings

Post by foggy »

Branimir's case was closed due to no activity from his side, so please continue working with your engineer on this issue.
Branimir
Lurker
Posts: 2
Liked: 1 time
Joined: Mar 28, 2016 4:52 pm
Full Name: Branimir Karajcic
Contact:

Re: Backup failure and warnings

Post by Branimir » 1 person likes this post

Hi HoFFy,

I decided to abandon the issue as recovering VM with that warning was fine. No production impact. SQL backup job truncates log anyway so this is not a big deal. If you want to get rid of the warning without solving the problem you can disable processing transaction lobs under "Processing Settings" in "Application-Aware Processing Options"

The warning still remains: "Code meaning = Not enough storage is available to complete this operation."
Unless space needed is outrageously high, it can't be storage. Server at the time of the job has 20 Gb free, while all databases combined on the server are less in size than that.

I think the actual problem is related to Virtual Disk service. I can't prove it, but the server that I am working with definitely has issues with Virtual Disk service as when I want to open Disk Management I get the error "Unable to connect to Virtual Disk Service"

I hope that this post at least points someone into right direction of fixing this issue.
aflora
Lurker
Posts: 1
Liked: never
Joined: Aug 06, 2021 11:00 am
Full Name: Alex Flora
Contact:

Re: Backup failure and warnings

Post by aflora »

I think you are on the rights spot with the disk management service. I have the same error and also the diskmanagement issue which resolves when i reboot but eventually returns. I wasn't aware of this as everything seems to be working, but once i resolve the diskmanagent issues i hope it also solves the veeam backup.

It can also explain the error about disk space issues if it can't connect to the virtual disk service although it isn't an disk space problem but more an error when trying to connect, so might be a misleading error.

Will post back if it solves the problems.
Post Reply

Who is online

Users browsing this forum: Dynamic, Google [Bot], Noushad, Semrush [Bot] and 171 guests