Backup Replicate vCenter VM?

VMware specific discussions

Re: Back up vCenter VM?

Veeam Logoby Vitaliy S. » Tue Mar 29, 2011 8:00 am

Yes, you need to add host with vCenter Server to the backup console. Datastores can only be added to the backup/replication job, which wouldn't help here.
Vitaliy S.
Veeam Software
 
Posts: 19539
Liked: 1097 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Can't back up vCenter Server, VSSControl: Failed to freeze g

Veeam Logoby turmoil » Thu Apr 07, 2011 7:02 am

[merged with existing discussion]

Hello,

I haven't been able to back up my vCenter machine for the past 6 days and I would really appreciate your help in resolving my issue.

Our setup:
-5x VMWare 4.1 hosts
-Veeam Backup & Replication 5.0.0.179 (64-bit) running on a Windows Server 2008 SP2 x64 virtual machine
-vCenter running on a Windows Server 2008 SP2 x64 virtual machine
-vCenter server is hosting the vCenter database on a local SQL 2005 Express instance (can't tell which version, but definitely pre-SP4)


My background:
Zero experience with Veeam
Basic experience with vCenter

My simple and short story:
Last Friday we changed the passwords for our domain user account and for the account that Veeam uses to connect to vCenter. All Veeam jobs started failing immediately but I was able to resolve this by recreating the backup jobs. All jobs are successfully backing up except for one: the vCenter Server job. This job backs up the vCenter Server only.

This is the error I get:
Code: Select all
Backup: vCenter Server (Full)
Created by (domain)\(user) at 6/04/2011 6:13:24 PM.
Session Details
Status Error Start time 7/04/2011 1:14:52 PM Details
   
Total VMs 1 End time 7/04/2011 1:17:42 PM   
Processed VMs 1 Duration 0:02:50   
Successful VMs 0 Total size 80.00 GB
Failed VMs 1 Processed size 0.00 KB
VMs in progress 0 Processing rate 0 KB/s
 
Processed Objects
VM name Status Start time End time Total size Processed size Processing rate Duration Details
ic-pra-wsv-08 - vCenter Error   7/04/2011 1:15:45 PM 7/04/2011 1:17:41 PM   80.00 GB 0.00 KB 0 KB/s 0:01:56   
Freezing guest operating system
VSSControl: Failed to freeze guest, wait timeout


As a result of the Veeam job failure, the following issues occur on the vCenter server:
1. SQLWriter, MSSQL$SQLEXP_VIM and SQLVDI errors are logged:
Code: Select all
Level,Date and Time,Source,Event ID,Task Category
Error,7/04/2011 4:41:30 PM,SQLWRITER,24583,None,"Sqllib error: OLEDB Error encountered calling ICommandText::Execute. hr = 0x80040e14. SQLSTATE: 42000, Native Error: 3013
Error state: 1, Severity: 16
Source: Microsoft SQL Native Client
Error message: BACKUP DATABASE is terminating abnormally.
SQLSTATE: 42000, Native Error: 3271
Error state: 1, Severity: 16
Source: Microsoft SQL Native Client
Error message: A nonrecoverable I/O error occurred on file ""{1B33943E-8636-4D54-A20B-D3A9F8606A4D}2:"" 995(error not found).
SQLSTATE: 01000, Native Error: 4035
Error state: 1, Severity: 0
Source: Microsoft SQL Native Client
Error message: Processed 0 pages for database 'msdb', file 'MSDBData' on file 1.
"
Error,7/04/2011 4:41:30 PM,SQLWRITER,24583,None,"Sqllib error: OLEDB Error encountered calling ICommandText::Execute. hr = 0x80040e14. SQLSTATE: 42000, Native Error: 3013
Error state: 1, Severity: 16
Source: Microsoft SQL Native Client
Error message: BACKUP DATABASE is terminating abnormally.
SQLSTATE: 42000, Native Error: 3271
Error state: 1, Severity: 16
Source: Microsoft SQL Native Client
Error message: A nonrecoverable I/O error occurred on file ""{1B33943E-8636-4D54-A20B-D3A9F8606A4D}3:"" 995(error not found).
SQLSTATE: 01000, Native Error: 4035
Error state: 1, Severity: 0
Source: Microsoft SQL Native Client
Error message: Processed 0 pages for database 'VIM_UMDB', file 'VIM_UMDB_dat' on file 1.
"
Error,7/04/2011 4:41:30 PM,SQLWRITER,24583,None,"Sqllib error: OLEDB Error encountered calling ICommandText::Execute. hr = 0x80040e14. SQLSTATE: 42000, Native Error: 3013
Error state: 1, Severity: 16
Source: Microsoft SQL Native Client
Error message: BACKUP DATABASE is terminating abnormally.
SQLSTATE: 42000, Native Error: 3271
Error state: 1, Severity: 16
Source: Microsoft SQL Native Client
Error message: A nonrecoverable I/O error occurred on file ""{1B33943E-8636-4D54-A20B-D3A9F8606A4D}1:"" 995(error not found).
SQLSTATE: 01000, Native Error: 4035
Error state: 1, Severity: 0
Source: Microsoft SQL Native Client
Error message: Processed 0 pages for database 'model', file 'modeldev' on file 1.
"
Error,7/04/2011 4:41:30 PM,SQLWRITER,24583,None,"Sqllib error: OLEDB Error encountered calling ICommandText::Execute. hr = 0x80040e14. SQLSTATE: 42000, Native Error: 3013
Error state: 1, Severity: 16
Source: Microsoft SQL Native Client
Error message: BACKUP DATABASE is terminating abnormally.
SQLSTATE: 42000, Native Error: 3271
Error state: 1, Severity: 16
Source: Microsoft SQL Native Client
Error message: A nonrecoverable I/O error occurred on file ""{1B33943E-8636-4D54-A20B-D3A9F8606A4D}4:"" 995(error not found).
SQLSTATE: 01000, Native Error: 4035
Error state: 1, Severity: 0
Source: Microsoft SQL Native Client
Error message: Processed 0 pages for database 'VIM_VCDB', file 'VIM_VCDB_dat' on file 1.
"
Error,7/04/2011 4:41:30 PM,SQLWRITER,24583,None,"Sqllib error: OLEDB Error encountered calling ICommandText::Execute. hr = 0x80040e14. SQLSTATE: 42000, Native Error: 3013
Error state: 1, Severity: 16
Source: Microsoft SQL Native Client
Error message: BACKUP DATABASE is terminating abnormally.
SQLSTATE: 42000, Native Error: 3271
Error state: 1, Severity: 16
Source: Microsoft SQL Native Client
Error message: A nonrecoverable I/O error occurred on file ""{1B33943E-8636-4D54-A20B-D3A9F8606A4D}5:"" 995(error not found).
SQLSTATE: 01000, Native Error: 4035
Error state: 1, Severity: 0
Source: Microsoft SQL Native Client
Error message: Processed 0 pages for database 'master', file 'master' on file 1.
"
Error,7/04/2011 4:41:30 PM,MSSQL$SQLEXP_VIM,18210,(2),BackupVirtualDeviceFile::SendFileInfoBegin:  failure on backup device '{1B33943E-8636-4D54-A20B-D3A9F8606A4D}2'. Operating system error 995(error not found).
Error,7/04/2011 4:41:30 PM,MSSQL$SQLEXP_VIM,18210,(2),BackupVirtualDeviceFile::SendFileInfoBegin:  failure on backup device '{1B33943E-8636-4D54-A20B-D3A9F8606A4D}3'. Operating system error 995(error not found).
Error,7/04/2011 4:41:30 PM,SQLVDI,1,None,SQLVDI: Loc=TriggerAbort. Desc=invoked. ErrorCode=(0). Process=1248. Thread=292. Server. Instance=SQLEXP_VIM. VD=Global\{1B33943E-8636-4D54-A20B-D3A9F8606A4D}2_SQLVDIMemoryName_0.
Error,7/04/2011 4:41:30 PM,MSSQL$SQLEXP_VIM,3041,(6),BACKUP failed to complete the command BACKUP DATABASE master. Check the backup application log for detailed messages.
Error,7/04/2011 4:41:30 PM,MSSQL$SQLEXP_VIM,3041,(6),BACKUP failed to complete the command BACKUP DATABASE VIM_VCDB. Check the backup application log for detailed messages.
Error,7/04/2011 4:41:30 PM,MSSQL$SQLEXP_VIM,3041,(6),BACKUP failed to complete the command BACKUP DATABASE VIM_UMDB. Check the backup application log for detailed messages.
Error,7/04/2011 4:41:30 PM,MSSQL$SQLEXP_VIM,3041,(6),BACKUP failed to complete the command BACKUP DATABASE msdb. Check the backup application log for detailed messages.
Error,7/04/2011 4:41:30 PM,MSSQL$SQLEXP_VIM,3041,(6),BACKUP failed to complete the command BACKUP DATABASE model. Check the backup application log for detailed messages.
Error,7/04/2011 4:41:30 PM,SQLVDI,1,None,SQLVDI: Loc=TriggerAbort. Desc=invoked. ErrorCode=(0). Process=1248. Thread=2700. Server. Instance=SQLEXP_VIM. VD=Global\{1B33943E-8636-4D54-A20B-D3A9F8606A4D}3_SQLVDIMemoryName_0.
Error,7/04/2011 4:41:30 PM,MSSQL$SQLEXP_VIM,18210,(2),BackupVirtualDeviceFile::SendFileInfoBegin:  failure on backup device '{1B33943E-8636-4D54-A20B-D3A9F8606A4D}4'. Operating system error 995(error not found).
Error,7/04/2011 4:41:30 PM,SQLVDI,1,None,SQLVDI: Loc=TriggerAbort. Desc=invoked. ErrorCode=(0). Process=1248. Thread=2720. Server. Instance=SQLEXP_VIM. VD=Global\{1B33943E-8636-4D54-A20B-D3A9F8606A4D}4_SQLVDIMemoryName_0.
Error,7/04/2011 4:41:30 PM,MSSQL$SQLEXP_VIM,18210,(2),BackupVirtualDeviceFile::SendFileInfoBegin:  failure on backup device '{1B33943E-8636-4D54-A20B-D3A9F8606A4D}1'. Operating system error 995(error not found).
Error,7/04/2011 4:41:30 PM,SQLVDI,1,None,SQLVDI: Loc=TriggerAbort. Desc=invoked. ErrorCode=(0). Process=1248. Thread=2780. Server. Instance=SQLEXP_VIM. VD=Global\{1B33943E-8636-4D54-A20B-D3A9F8606A4D}1_SQLVDIMemoryName_0.
Error,7/04/2011 4:41:30 PM,MSSQL$SQLEXP_VIM,18210,(2),BackupVirtualDeviceFile::SendFileInfoBegin:  failure on backup device '{1B33943E-8636-4D54-A20B-D3A9F8606A4D}5'. Operating system error 995(error not found).
Error,7/04/2011 4:41:30 PM,SQLVDI,1,None,SQLVDI: Loc=TriggerAbort. Desc=invoked. ErrorCode=(0). Process=1248. Thread=2640. Server. Instance=SQLEXP_VIM. VD=Global\{1B33943E-8636-4D54-A20B-D3A9F8606A4D}5_SQLVDIMemoryName_0.
Error,7/04/2011 4:41:30 PM,SQLVDI,1,None,SQLVDI: Loc=SignalAbort. Desc=Client initiates abort. ErrorCode=(0). Process=2412. Thread=3168. Client. Instance=SQLEXP_VIM. VD=Global\{1B33943E-8636-4D54-A20B-D3A9F8606A4D}5_SQLVDIMemoryName_0.
Error,7/04/2011 4:41:30 PM,SQLVDI,1,None,SQLVDI: Loc=SignalAbort. Desc=Client initiates abort. ErrorCode=(0). Process=2412. Thread=3168. Client. Instance=SQLEXP_VIM. VD=Global\{1B33943E-8636-4D54-A20B-D3A9F8606A4D}4_SQLVDIMemoryName_0.
Error,7/04/2011 4:41:30 PM,SQLVDI,1,None,SQLVDI: Loc=SignalAbort. Desc=Client initiates abort. ErrorCode=(0). Process=2412. Thread=3168. Client. Instance=SQLEXP_VIM. VD=Global\{1B33943E-8636-4D54-A20B-D3A9F8606A4D}3_SQLVDIMemoryName_0.
Error,7/04/2011 4:41:30 PM,SQLVDI,1,None,SQLVDI: Loc=SignalAbort. Desc=Client initiates abort. ErrorCode=(0). Process=2412. Thread=3168. Client. Instance=SQLEXP_VIM. VD=Global\{1B33943E-8636-4D54-A20B-D3A9F8606A4D}2_SQLVDIMemoryName_0.
Error,7/04/2011 4:41:30 PM,SQLVDI,1,None,SQLVDI: Loc=SignalAbort. Desc=Client initiates abort. ErrorCode=(0). Process=2412. Thread=3168. Client. Instance=SQLEXP_VIM. VD=Global\{1B33943E-8636-4D54-A20B-D3A9F8606A4D}1_SQLVDIMemoryName_0.


2. VSS breaks forcing a restart to bring VSS back to an operational state:
Code: Select all
C:\Users\user>vssadmin list writers
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2005 Microsoft Corp.

Writer name: 'System Writer'
   Writer Id: {e8132975-6f93-4464-a53e-1050253ae220}
   Writer Instance Id: {5f746417-0701-4387-9e06-9b64da9a1403}
   State: [1] Stable
   Last error: No error

Writer name: 'SqlServerWriter'
   Writer Id: {a65faa63-5ea8-4ebc-9dbd-a0c4db26912a}
   Writer Instance Id: {86b82811-aba1-4eaa-963f-95677202f21c}
   State: [9] Failed
   Last error: Timed out

Writer name: 'ASR Writer'
   Writer Id: {be000cbe-11fe-4426-9c58-531aa6355fc4}
   Writer Instance Id: {32443eb2-3527-4790-ad66-d2089679fa55}
   State: [1] Stable
   Last error: No error

Writer name: 'WMI Writer'
   Writer Id: {a6ad56c2-b509-4e6c-bb19-49d8f43532f0}
   Writer Instance Id: {063c4be6-7b23-41b3-9bb2-9e3537a0c13a}
   State: [9] Failed
   Last error: Timed out

Writer name: 'Registry Writer'
   Writer Id: {afbab4a2-367d-4d15-a586-71dbb18f8485}
   Writer Instance Id: {2b786c56-d1f3-4017-b1a6-2fc330b9b333}
   State: [1] Stable
   Last error: No error

Writer name: 'BITS Writer'
   Writer Id: {4969d978-be47-48b0-b100-f328f07ac1e0}
   Writer Instance Id: {6ef9d58e-b7d7-48de-84d8-6ef2cde5e018}
   State: [1] Stable
   Last error: No error

Writer name: 'COM+ REGDB Writer'
   Writer Id: {542da469-d3e1-473c-9f4f-7847f01fc64f}
   Writer Instance Id: {43723a54-d04f-41e9-b02a-a1b187d89a67}
   State: [1] Stable
   Last error: No error

Writer name: 'Shadow Copy Optimization Writer'
   Writer Id: {4dc3bdd4-ab48-4d07-adb0-3bee2926fd7f}
   Writer Instance Id: {6d484c62-f228-4e8a-949e-4ef1fbb50ae9}
   State: [1] Stable
   Last error: No error

Writer name: 'ADAM (VMwareVCMSDS) Writer'
   Writer Id: {dd846aaa-a1b6-42a8-aaf8-03dcb6114bfd}
   Writer Instance Id: {b06949ce-c507-48ef-83a1-a463c4e5073e}
   State: [9] Failed
   Last error: Timed out


So for some reason, every time I run the vCenter Server backup job, VSS fails to freeze the vCenter SQL database and breaks, forcing a restart of the vCenter server which only fixes VSS until the backup job is run again.

I have tried restarting both the Veeam and vCenter server to try and resolve the issue but I'm really at a loss and not sure how to continue troubleshooting.

Any help will be greatly appreciated!!

Thanks in advance.
turmoil
Lurker
 
Posts: 2
Liked: never
Joined: Thu Apr 07, 2011 1:22 am
Full Name: Erez Carmeli

Re: Back up vCenter VM?

Veeam Logoby turmoil » Fri Apr 08, 2011 12:59 am

Received a reply from Veeam Support and that has immediately resolved my issue:

VSS freezes the I/O of a VM, and in this case that is the vCenter. During the backup, we send a snapshot command to the vCenter, which then sends to the source ESX host, to create a snapshot for the virtual machine we are backing up. For VSS , it is required to have confirmation of the snapshot creation, which is not received because the I/O is frozen on the vCenter which is 'waiting' for the snapshot completion notification from the ESX host. It times out due to this each and every time.

There is a way to workaround this, by adding in the ESX host that is managing this vCenter VM into Veeam Backup by IP address. This allows you to communicate directly with the host, bypassing vCenter communication for operations in the job.


So to put it in layman’s terms:
1. Under Servers, add the ESX host (by IP address) that hosts the vCenter server VM.
2. Edit/create the vCenter backup job. Remove vCenter server if editing existing job.
3. Click to add a server.
4. When browsing for the vCenter server, make sure you add the vCenter server from the ESX host and NOT from vCenter.
turmoil
Lurker
 
Posts: 2
Liked: never
Joined: Thu Apr 07, 2011 1:22 am
Full Name: Erez Carmeli

Re: Back up vCenter VM?

Veeam Logoby Vitaliy S. » Fri Apr 08, 2011 7:50 am

Yes, these are exactly the same steps as explained earlier this topic. :wink:
Vitaliy S.
Veeam Software
 
Posts: 19539
Liked: 1097 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Veeam 5 Backup issues with Virtual Center VM

Veeam Logoby william-pkv » Wed Apr 27, 2011 6:05 am

[merged]

Hi,

we are using Veeam 5.0.1 to backup our vSphere 4.0 environment. I have issues to backup the Virtual Center VM with local installed SQL Database.
I got the the Message: "Freezing guest operating system VSSControl: Failed to freeze guest, wait timeout". I use VSS in this case.

I have no issues to Backup other Servers with local installed SQL Databases. Only the vCenter VM aborts the backup. What could i do?


Thanks,
William
william-pkv
Novice
 
Posts: 4
Liked: never
Joined: Wed Apr 20, 2011 2:29 pm
Full Name: William Sennecke

Re: Veeam 5 Backup issues with Virtual Center VM

Veeam Logoby dybenko » Wed Apr 27, 2011 6:17 am

Hi,

uncheck "Enable application-aware image processing", "Enable guest file system indexing" and "Enable vmware tools quiescence"
dybenko
Lurker
 
Posts: 2
Liked: never
Joined: Thu Dec 30, 2010 9:40 am
Full Name: Yourii Dybenko

Re: Veeam 5 Backup issues with Virtual Center VM

Veeam Logoby william-pkv » Wed Apr 27, 2011 6:54 am

Why i cannot backup the vCenter VM only? Other backup jobs with SQL Servers works with VSS.
william-pkv
Novice
 
Posts: 4
Liked: never
Joined: Wed Apr 20, 2011 2:29 pm
Full Name: William Sennecke

Re: Veeam 5 Backup issues with Virtual Center VM

Veeam Logoby dybenko » Wed Apr 27, 2011 7:16 am

For comments - Trainsignal.com - VMware vSphere Pro Series Training Vol. 2 - DVD2, VeeamBackup, lesson 8 - "Backing Up vCenter with Veeam Backup and VSS". My english is bad, sorry.
dybenko
Lurker
 
Posts: 2
Liked: never
Joined: Thu Dec 30, 2010 9:40 am
Full Name: Yourii Dybenko

Re: Veeam 5 Backup issues with Virtual Center VM

Veeam Logoby Gostev » Wed Apr 27, 2011 9:03 am

william-pkv wrote:Why i cannot backup the vCenter VM only? Other backup jobs with SQL Servers works with VSS.

Because frozen vCenter cannot process API calls which are needed to perform backup. Deadlock.
Gostev
Veeam Software
 
Posts: 21385
Liked: 2348 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Back up vCenter VM?

Veeam Logoby pageda » Wed Apr 27, 2011 5:50 pm

Can someone explain why you can't backup the vcenter vm normally?..i.e why the host has to be directly added.

We use VSphere Essentials and thus don't vmotion our vm's. I've got Veeam backup currently backing up the vcenter server and haven't gotten any errors. Then again, I haven't tried a restore.
pageda
Novice
 
Posts: 7
Liked: 1 time
Joined: Wed Apr 27, 2011 5:18 pm
Full Name: David Page

Re: Back up vCenter VM?

Veeam Logoby Vitaliy S. » Thu Apr 28, 2011 8:00 am

pageda wrote:Can someone explain why you can't backup the vcenter vm normally?..i.e why the host has to be directly added.

Gostev wrote:Because frozen vCenter cannot process API calls which are needed to perform backup. Deadlock.

You can backup your vCenter Server without adding it directly from the corresponding host but with "Application-aware image processing" disabled.
Vitaliy S.
Veeam Software
 
Posts: 19539
Liked: 1097 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

vCenter VM Backup

Veeam Logoby pcrebe » Thu May 19, 2011 12:21 am

[merged]

Hello,
i can't backup my vCenter VM with "Enable application-aware image processing" enabled, i've added the VM trought the Host where it's registred, but it always fails (VSS control: failed to freeze guest, wait timeout).

If i don't use VSS the job goes right.
Could it depend by the vCPU (1) used for my vCenter VM, because sometimes the same failure appends on another VM that have only 1 vCPU too.

Any idea?
Thanks
pcrebe
Enthusiast
 
Posts: 94
Liked: 1 time
Joined: Mon Dec 06, 2010 10:41 pm
Full Name: CARLO

Re: vCenter VM Backup

Veeam Logoby cag » Thu May 19, 2011 1:49 am

you need to backup this VM using the source server as a ESXI host only. Adding the source via vcenter doesn't work because when VSS starts communication to the source is lost via vcenter. This is a simple explanation but other posts have also confirmed to same.
cag
Enthusiast
 
Posts: 74
Liked: never
Joined: Sat Mar 26, 2011 4:02 am
Full Name: Conrad Gotzmann

Re: vCenter VM Backup

Veeam Logoby pcrebe » Thu May 19, 2011 5:31 am

cag wrote:you need to backup this VM using the source server as a ESXI host only.


i did that (like i wrote with my word), but it doesn't work.
pcrebe
Enthusiast
 
Posts: 94
Liked: 1 time
Joined: Mon Dec 06, 2010 10:41 pm
Full Name: CARLO

Re: Back up vCenter VM?

Veeam Logoby Vitaliy S. » Thu May 19, 2011 7:59 am

Carlo, if you have any technical issues, please contact our support team directly as it is described when you click New Topic. Thanks.
Vitaliy S.
Veeam Software
 
Posts: 19539
Liked: 1097 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

PreviousNext

Return to VMware vSphere



Who is online

Users browsing this forum: btmaus and 15 guests