SBS2011/Exchange VSS Issue - 9.5U3

Availability for the Always-On Enterprise

Re: SBS2011/Exchange VSS Issue - 9.5U3

Veeam Logoby Shunx239 » Wed Jan 03, 2018 2:25 pm

I am pessimistic, IMHO the core team must be involved. The support will help documenting issue of course.

Before the update, 6sec freeze had been sufficient on my system.

In the last run (9.5U3) the timescale is as follows:

13:41:05 VSS freeze started
13:41:07 Create virtual machine snapshot - Started
13:41:08 Create virtual machine snapshot - Completed
13:41:25 Shadow copy freeze ended (aborted)

As far as I understand the following step causes the problem (KB1680):

Return of snapshot information via VIM API
These steps should usually be near-instantaneous, but if the vCenter is heavily loaded or has a high latency to the ESXi hosts, the delay may be significant.


The vCenter is the same, I blame Veeam for this issue
Shunx239
Novice
 
Posts: 7
Liked: 1 time
Joined: Sat Feb 11, 2017 9:39 pm
Location: Milan
Full Name: Nikolay

Re: SBS2011/Exchange VSS Issue - 9.5U3

Veeam Logoby bpayne » Wed Jan 03, 2018 3:34 pm

Has anyone been able to nail down the operation systems affected (only affecting SBS2011?)? I have an Exchange 2010 environment running on Windows Server 2008 R2 and wondering if I will be affected by this. I can't risk Exchange transaction logs not truncating. I have been waiting to install Update 3 because I am following this thread. Thanks
bpayne
Enthusiast
 
Posts: 25
Liked: 2 times
Joined: Tue Jan 20, 2015 2:07 pm
Full Name: Brandon Payne

Re: SBS2011/Exchange VSS Issue - 9.5U3

Veeam Logoby Shunx239 » Wed Jan 03, 2018 4:19 pm 1 person likes this post

I believe other Exchange 2010 installations are affected as well by the freeze timeout, just not beeing reported as failures since Veeam fails over to another snapshot technology unavailable for SBS acting as domain controller.

New in v8
To overcome this VSS limitation, Veeam Backup & Replication utilizes the Microsoft VSS persistent snapshots technology for backup of Microsoft Exchange VMs. If Microsoft Exchange fails to be frozen within the allowed period of time, Veeam Backup & Replication automatically fails over to the persistent snapshot mechanism. To learn more about this new feature please read:
http://helpcenter.veeam.com/backup/80/v ... shots.html
Shunx239
Novice
 
Posts: 7
Liked: 1 time
Joined: Sat Feb 11, 2017 9:39 pm
Location: Milan
Full Name: Nikolay

Re: SBS2011/Exchange VSS Issue - 9.5U3

Veeam Logoby Shunx239 » Wed Jan 03, 2018 5:25 pm

Rather clear now, seems to be a real bug in the new build.

Look what we had with 9.5U2 (Task.VMName.vm-223.log):
Code: Select all
[20.12.2017 21:02:39] <24> Info     VSSControl: Frozen in 12sec
[20.12.2017 21:02:40] <24> Info     [VimApi] Create snapshot, ref "vm-223", name "VEEAM BACKUP TEMPORARY SNAPSHOT", description "Please do not delete this snapshot. It is being used by Veeam Backup.", memory "False", quiesce "False"
[20.12.2017 21:02:42] <24> Info     [Soap] Snapshot VM 'vm-223' was created. Ref: 'snapshot-3272'
[20.12.2017 21:02:42] <24> Info     VSSControl: Unfreezing
[20.12.2017 21:02:48] <24> Info     CGuestVssSnapshotKeeper 2: Created, ttl 1200sec


The same with 9.5U3 looks like this:
Code: Select all
[03.01.2018 13:41:06] <18> Info     VSSControl: Frozen in 20sec
[03.01.2018 13:41:06] <18> Info     [VimApi] Create snapshot, ref "vm-223", name "VEEAM BACKUP TEMPORARY SNAPSHOT", description "Please do not delete this snapshot. It is being used by Veeam Backup.", memory "False", quiesce "False"
[03.01.2018 13:41:26] <18> Info     [Soap] Snapshot VM 'vm-223' was created. Ref: 'snapshot-3373'
[03.01.2018 13:41:26] <18> Info     VSSControl: Unfreezing
[03.01.2018 13:41:32] <18> Error    Failed to call RPC function 'Vss.Unfreeze': Error code: 0x80004005. Failed to invoke func [Unfreeze]: Unspecified error. Unfreeze error: [Backup job failed. Cannot create a shadow copy of the volumes containing writer's data. A VSS critical writer has failed. Writer name: [Microsoft Exchange Writer]. Class ID: [{76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}]. Instance ID: [{eaaa415f-aefd-4d13-a357-79ea5f00b4e6}]. Writer's state: [VSS_WS_FAILED_AT_FREEZE]. Error code: [0x800423f2].].. InParams: '<InputArguments><ttl value="1200" /><VssSessionId value="{d04802b5-bdd8-40bb-930b-48fed11d211e}" /></InputArguments>'.


With 9.5U3 the SOAP returns success but not immediately, on 20sec timeout.
Unfortunately it is too late for issuing 'Vss.Unfreeze'

With 9.5U2 the timeout was 12sec, it was reasonable.

Another question is why it returned quicker with 9.5U2, but the 20sec timeout of 9.5U3 can't work at all!
Shunx239
Novice
 
Posts: 7
Liked: 1 time
Joined: Sat Feb 11, 2017 9:39 pm
Location: Milan
Full Name: Nikolay

Re: SBS2011/Exchange VSS Issue - 9.5U3

Veeam Logoby Dima P. » Wed Jan 03, 2018 8:10 pm 1 person likes this post

Hi folks,

All cases mentioned in this thread are escalated, so please keep working with support team. If you face similar problem please submit a case and update this thread with your case ID. Please also doublecheck if the described workaround works in your environment. Thank you in advance.
Dima P.
Veeam Software
 
Posts: 7015
Liked: 507 times
Joined: Mon Feb 04, 2013 2:07 pm
Location: SPb
Full Name: Dmitry Popov

Re: SBS2011/Exchange VSS Issue - 9.5U3

Veeam Logoby Shunx239 » Sun Jan 07, 2018 10:29 am

I was probably wrong stating the max timeout was determined by the call to VIM API.

Still confused by the timestamps. Here is the Veeam log excerpt:
Code: Select all
[03.01.2018 13:40:46] <18> Info     VSSControl: Freezing. Job id: a8d52fd7-6676-4eeb-9cb3-5550dc8143b1. Job Name: Afrodite Backup. VssCopyOnly: False
[03.01.2018 13:40:46] <18> Info     VSSControl: Snapshots will be use for indexing. Volumes spec: <VolumeFilter vssWillBeUsedForIndexing="True"><Include><Path>F:\</Path><Path>D:\</Path><Path>E:\</Path><Path>Z:\</Path><Path>C:\WSUS\</Path><Path>C:\</Path></Include><Exclude><Path>C:\WSUS\</Path><Path>%windir%</Path><Path>%ProgramFiles%</Path><Path>%ProgramFiles(x86)%</Path><Path>%ProgramW6432%</Path><Path>%TEMP%</Path></Exclude></VolumeFilter>
[03.01.2018 13:41:06] <18> Info     VSSControl: Frozen in 20sec
[03.01.2018 13:41:06] <18> Info     [VimApi] Create snapshot, ref "vm-223", name "VEEAM BACKUP TEMPORARY SNAPSHOT", description "Please do not delete this snapshot. It is being used by Veeam Backup.", memory "False", quiesce "False"
[03.01.2018 13:41:26] <18> Info     [Soap] Snapshot VM 'vm-223' was created. Ref: 'snapshot-3373'
[03.01.2018 13:41:26] <18> Info     VSSControl: Unfreezing
[03.01.2018 13:41:32] <18> Error    Failed to call RPC function 'Vss.Unfreeze': Error code: 0x80004005. Failed to invoke func [Unfreeze]: Unspecified error. Unfreeze error: [Backup job failed. Cannot create a shadow copy of the volumes containing writer's data. A VSS critical writer has failed. Writer name: [Microsoft Exchange Writer]. Class ID: [{76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}]. Instance ID: [{eaaa415f-aefd-4d13-a357-79ea5f00b4e6}]. Writer's state: [VSS_WS_FAILED_AT_FREEZE]. Error code: [0x800423f2].].. InParams: '<InputArguments><ttl value="1200" /><VssSessionId value="{d04802b5-bdd8-40bb-930b-48fed11d211e}" /></InputArguments>'.
[03.01.2018 13:41:32] <18> Error     Error code: 0x80004005
[03.01.2018 13:41:32] <18> Error    Failed to invoke func [Unfreeze]: Unspecified error. Unfreeze error: [Backup job failed.
[03.01.2018 13:41:32] <18> Error    Cannot create a shadow copy of the volumes containing writer's data.
[03.01.2018 13:41:32] <18> Error    A VSS critical writer has failed. Writer name: [Microsoft Exchange Writer]. Class ID: [{76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}]. Instance ID: [{eaaa415f-aefd-4d13-a357-79ea5f00b4e6}]. Writer's state: [VSS_WS_FAILED_AT_FREEZE]. Error code: [0x800423f2].] (System.Runtime.InteropServices.COMException)
[03.01.2018 13:41:32] <18> Error       at VeeamProxyClient.GenerateComException(basic_string<char\,std::char_traits<char>\,std::allocator<char> >* excMsg, Int32 errHrCode)
[03.01.2018 13:41:32] <18> Error       at VeeamProxyClient.RegenerateCLRException(exception* excBase)
[03.01.2018 13:41:32] <18> Error       at VeeamProxyClient.CCliProxyRpcInvoker.DoRpc(String cmdName, String inXml, String& outXml)
[03.01.2018 13:41:32] <18> Error       at Veeam.Backup.ProxyProvider.CProxyRpcInvoker.Call(String methodName, CProxyInvokeInputArg inputArgs, Boolean secureCall)
[03.01.2018 13:41:32] <18> Error     Error code: 0x80004005 (Veeam.Backup.Common.CCppCOMException)
[03.01.2018 13:41:32] <18> Error    Failed to invoke func [Unfreeze]: Unspecified error. Unfreeze error: [Backup job failed.
[03.01.2018 13:41:32] <18> Error     (Veeam.Backup.Common.CCppComponentException)
[03.01.2018 13:41:32] <18> Error    Cannot create a shadow copy of the volumes containing writer's data.
[03.01.2018 13:41:32] <18> Error     (Veeam.Backup.Common.CCppComponentException)
[03.01.2018 13:41:32] <18> Error    A VSS critical writer has failed. Writer name: [Microsoft Exchange Writer]. Class ID: [{76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}]. Instance ID: [{eaaa415f-aefd-4d13-a357-79ea5f00b4e6}]. Writer's state: [VSS_WS_FAILED_AT_FREEZE]. Error code: [0x800423f2].] (Veeam.Backup.Common.CCppComponentException)
[03.01.2018 13:41:32] <18> Error       at Veeam.Backup.ProxyProvider.CProxyRpcInvoker.Call(String methodName, CProxyInvokeInputArg inputArgs, Boolean secureCall)
[03.01.2018 13:41:32] <18> Error       at Veeam.Backup.Core.CVssProxyInvoker.Invoke(String funcName, TimeSpan timeout, IVssProxyArg arg, Boolean secureCall)
[03.01.2018 13:41:32] <18> Error       at Veeam.Backup.Core.CVssProxyInvoker.Invoke(String funcName, IVssProxyArg arg, Boolean secureCall)
[03.01.2018 13:41:32] <18> Error       at Veeam.Backup.VssProvider.CVssControl.Unfreeze(Guid jobId, UInt32 ttl)
[03.01.2018 13:41:32] <18> Error    VMware snapshot creation time 19 seconds. If vCenter database is present in guest, exclusion may need to be added manually. Follow Veeam KB1051
[03.01.2018 13:41:32] <18> Error    VSSControl: Failed to freeze guest over network, wait timeout   at Veeam.Backup.VssProvider.CVssControl.Unfreeze(Guid jobId, UInt32 ttl)
[03.01.2018 13:41:32] <18> Error       at Veeam.Backup.Core.CViGuestVssFreezer.Unfreeze(Boolean createPersistentVssSnapshot, Boolean snapshotSuccessed)
[03.01.2018 13:41:32] <18> Error    VSSControl: Failed to freeze guest over network, wait timeout (System.TimeoutException)
[03.01.2018 13:41:32] <18> Error       at Veeam.Backup.VssProvider.CVssControl.Unfreeze(Guid jobId, UInt32 ttl)
[03.01.2018 13:41:32] <18> Error       at Veeam.Backup.Core.CViGuestVssFreezer.Unfreeze(Boolean createPersistentVssSnapshot, Boolean snapshotSuccessed)
[03.01.2018 13:41:34] <18> Info     Checking if VSS persistent snapshot allowed. Result: False

When OS application log starts reporting ESE/ESENT Event ID 2005/2001 at 13:41:05 and times out after 20sec.

It would be interesting to reproduce the issue with clean Exchange 2010 installation. It should go in timeout the same way and then fail over to VSS persistent snapshot (not possible with SBS2011 - see the last line of the exerpt above).

We have retargeted the job to run against VMWare host directly.
It works: this way VSS freeze takes between 7 and 13 secs (as it has been against VC with 9.5U2).

I understand it could not be an option for everybody.

so please keep working with support team.

In our case it appears to be waste of time. They suggest me to oprimize my SBS installation. After the issue is isolated and workaround applied it has few sense. I will keep the support case open untill a fix is available from Veeam.
Shunx239
Novice
 
Posts: 7
Liked: 1 time
Joined: Sat Feb 11, 2017 9:39 pm
Location: Milan
Full Name: Nikolay

Re: SBS2011/Exchange VSS Issue - 9.5U3

Veeam Logoby Gostev » Mon Jan 15, 2018 10:30 pm

Hi all, just wanted to update that according to the internal communication I've received, the issue has been understood (elongated JIT compilation time during the first vSphere API interaction) and the hot fix is now in testing. Thanks!
Gostev
Veeam Software
 
Posts: 21733
Liked: 2459 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: SBS2011/Exchange VSS Issue - 9.5U3

Veeam Logoby gatorheelz » Mon Jan 15, 2018 10:32 pm

That's great news Gostev. We were debating between a new backup chain (bypassing vCenter) or crash consistent backups.
gatorheelz
Influencer
 
Posts: 16
Liked: 2 times
Joined: Mon Jun 19, 2017 4:38 pm
Full Name: gatorheelz

Re: SBS2011/Exchange VSS Issue - 9.5U3

Veeam Logoby ic_mcourtney » Mon Jan 15, 2018 11:59 pm

Gostev wrote:Hi all, just wanted to update that according to the internal communication I've received, the issue has been understood (elongated JIT compilation time during the first vSphere API interaction) and the hot fix is now in testing. Thanks!


That's great news Gostev! Please keep us updated about when the hotfix is available. I've held off on upgrading to U3 because I was concerned that we'd see this issue in our environment; I'd be much happier to do the upgrade know that there is a hotfix available, should I need it.
ic_mcourtney
Influencer
 
Posts: 20
Liked: 2 times
Joined: Thu Sep 22, 2016 7:15 am
Full Name: mcourtney

Re: SBS2011/Exchange VSS Issue - 9.5U3

Veeam Logoby rkovhaev » Tue Jan 16, 2018 7:11 pm

hotfix is available, feel free to open up a support case with us
rkovhaev
Veeam Software
 
Posts: 34
Liked: 12 times
Joined: Mon May 17, 2010 6:49 pm
Location: hockey night in canada
Full Name: Rustam

Previous

Return to Veeam Backup & Replication



Who is online

Users browsing this forum: MrSpock and 1 guest