SBS2011/Exchange VSS Issue - 9.5U3

Availability for the Always-On Enterprise

Re: SBS2011/Exchange VSS Issue - 9.5U3

Veeam Logoby Shunx239 » Wed Jan 03, 2018 2:25 pm

I am pessimistic, IMHO the core team must be involved. The support will help documenting issue of course.

Before the update, 6sec freeze had been sufficient on my system.

In the last run (9.5U3) the timescale is as follows:

13:41:05 VSS freeze started
13:41:07 Create virtual machine snapshot - Started
13:41:08 Create virtual machine snapshot - Completed
13:41:25 Shadow copy freeze ended (aborted)

As far as I understand the following step causes the problem (KB1680):

Return of snapshot information via VIM API
These steps should usually be near-instantaneous, but if the vCenter is heavily loaded or has a high latency to the ESXi hosts, the delay may be significant.


The vCenter is the same, I blame Veeam for this issue
Shunx239
Novice
 
Posts: 8
Liked: 2 times
Joined: Sat Feb 11, 2017 9:39 pm
Location: Milan
Full Name: Nikolay

Re: SBS2011/Exchange VSS Issue - 9.5U3

Veeam Logoby bpayne » Wed Jan 03, 2018 3:34 pm

Has anyone been able to nail down the operation systems affected (only affecting SBS2011?)? I have an Exchange 2010 environment running on Windows Server 2008 R2 and wondering if I will be affected by this. I can't risk Exchange transaction logs not truncating. I have been waiting to install Update 3 because I am following this thread. Thanks
bpayne
Enthusiast
 
Posts: 47
Liked: 7 times
Joined: Tue Jan 20, 2015 2:07 pm
Full Name: Brandon Payne

Re: SBS2011/Exchange VSS Issue - 9.5U3

Veeam Logoby Shunx239 » Wed Jan 03, 2018 4:19 pm 1 person likes this post

I believe other Exchange 2010 installations are affected as well by the freeze timeout, just not beeing reported as failures since Veeam fails over to another snapshot technology unavailable for SBS acting as domain controller.

New in v8
To overcome this VSS limitation, Veeam Backup & Replication utilizes the Microsoft VSS persistent snapshots technology for backup of Microsoft Exchange VMs. If Microsoft Exchange fails to be frozen within the allowed period of time, Veeam Backup & Replication automatically fails over to the persistent snapshot mechanism. To learn more about this new feature please read:
http://helpcenter.veeam.com/backup/80/v ... shots.html
Shunx239
Novice
 
Posts: 8
Liked: 2 times
Joined: Sat Feb 11, 2017 9:39 pm
Location: Milan
Full Name: Nikolay

Re: SBS2011/Exchange VSS Issue - 9.5U3

Veeam Logoby Shunx239 » Wed Jan 03, 2018 5:25 pm

Rather clear now, seems to be a real bug in the new build.

Look what we had with 9.5U2 (Task.VMName.vm-223.log):
Code: Select all
[20.12.2017 21:02:39] <24> Info     VSSControl: Frozen in 12sec
[20.12.2017 21:02:40] <24> Info     [VimApi] Create snapshot, ref "vm-223", name "VEEAM BACKUP TEMPORARY SNAPSHOT", description "Please do not delete this snapshot. It is being used by Veeam Backup.", memory "False", quiesce "False"
[20.12.2017 21:02:42] <24> Info     [Soap] Snapshot VM 'vm-223' was created. Ref: 'snapshot-3272'
[20.12.2017 21:02:42] <24> Info     VSSControl: Unfreezing
[20.12.2017 21:02:48] <24> Info     CGuestVssSnapshotKeeper 2: Created, ttl 1200sec


The same with 9.5U3 looks like this:
Code: Select all
[03.01.2018 13:41:06] <18> Info     VSSControl: Frozen in 20sec
[03.01.2018 13:41:06] <18> Info     [VimApi] Create snapshot, ref "vm-223", name "VEEAM BACKUP TEMPORARY SNAPSHOT", description "Please do not delete this snapshot. It is being used by Veeam Backup.", memory "False", quiesce "False"
[03.01.2018 13:41:26] <18> Info     [Soap] Snapshot VM 'vm-223' was created. Ref: 'snapshot-3373'
[03.01.2018 13:41:26] <18> Info     VSSControl: Unfreezing
[03.01.2018 13:41:32] <18> Error    Failed to call RPC function 'Vss.Unfreeze': Error code: 0x80004005. Failed to invoke func [Unfreeze]: Unspecified error. Unfreeze error: [Backup job failed. Cannot create a shadow copy of the volumes containing writer's data. A VSS critical writer has failed. Writer name: [Microsoft Exchange Writer]. Class ID: [{76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}]. Instance ID: [{eaaa415f-aefd-4d13-a357-79ea5f00b4e6}]. Writer's state: [VSS_WS_FAILED_AT_FREEZE]. Error code: [0x800423f2].].. InParams: '<InputArguments><ttl value="1200" /><VssSessionId value="{d04802b5-bdd8-40bb-930b-48fed11d211e}" /></InputArguments>'.


With 9.5U3 the SOAP returns success but not immediately, on 20sec timeout.
Unfortunately it is too late for issuing 'Vss.Unfreeze'

With 9.5U2 the timeout was 12sec, it was reasonable.

Another question is why it returned quicker with 9.5U2, but the 20sec timeout of 9.5U3 can't work at all!
Shunx239
Novice
 
Posts: 8
Liked: 2 times
Joined: Sat Feb 11, 2017 9:39 pm
Location: Milan
Full Name: Nikolay

Re: SBS2011/Exchange VSS Issue - 9.5U3

Veeam Logoby Dima P. » Wed Jan 03, 2018 8:10 pm 1 person likes this post

Hi folks,

All cases mentioned in this thread are escalated, so please keep working with support team. If you face similar problem please submit a case and update this thread with your case ID. Please also doublecheck if the described workaround works in your environment. Thank you in advance.
Dima P.
Veeam Software
 
Posts: 7789
Liked: 557 times
Joined: Mon Feb 04, 2013 2:07 pm
Location: Prague
Full Name: Dmitry Popov

Re: SBS2011/Exchange VSS Issue - 9.5U3

Veeam Logoby Shunx239 » Sun Jan 07, 2018 10:29 am

I was probably wrong stating the max timeout was determined by the call to VIM API.

Still confused by the timestamps. Here is the Veeam log excerpt:
Code: Select all
[03.01.2018 13:40:46] <18> Info     VSSControl: Freezing. Job id: a8d52fd7-6676-4eeb-9cb3-5550dc8143b1. Job Name: Afrodite Backup. VssCopyOnly: False
[03.01.2018 13:40:46] <18> Info     VSSControl: Snapshots will be use for indexing. Volumes spec: <VolumeFilter vssWillBeUsedForIndexing="True"><Include><Path>F:\</Path><Path>D:\</Path><Path>E:\</Path><Path>Z:\</Path><Path>C:\WSUS\</Path><Path>C:\</Path></Include><Exclude><Path>C:\WSUS\</Path><Path>%windir%</Path><Path>%ProgramFiles%</Path><Path>%ProgramFiles(x86)%</Path><Path>%ProgramW6432%</Path><Path>%TEMP%</Path></Exclude></VolumeFilter>
[03.01.2018 13:41:06] <18> Info     VSSControl: Frozen in 20sec
[03.01.2018 13:41:06] <18> Info     [VimApi] Create snapshot, ref "vm-223", name "VEEAM BACKUP TEMPORARY SNAPSHOT", description "Please do not delete this snapshot. It is being used by Veeam Backup.", memory "False", quiesce "False"
[03.01.2018 13:41:26] <18> Info     [Soap] Snapshot VM 'vm-223' was created. Ref: 'snapshot-3373'
[03.01.2018 13:41:26] <18> Info     VSSControl: Unfreezing
[03.01.2018 13:41:32] <18> Error    Failed to call RPC function 'Vss.Unfreeze': Error code: 0x80004005. Failed to invoke func [Unfreeze]: Unspecified error. Unfreeze error: [Backup job failed. Cannot create a shadow copy of the volumes containing writer's data. A VSS critical writer has failed. Writer name: [Microsoft Exchange Writer]. Class ID: [{76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}]. Instance ID: [{eaaa415f-aefd-4d13-a357-79ea5f00b4e6}]. Writer's state: [VSS_WS_FAILED_AT_FREEZE]. Error code: [0x800423f2].].. InParams: '<InputArguments><ttl value="1200" /><VssSessionId value="{d04802b5-bdd8-40bb-930b-48fed11d211e}" /></InputArguments>'.
[03.01.2018 13:41:32] <18> Error     Error code: 0x80004005
[03.01.2018 13:41:32] <18> Error    Failed to invoke func [Unfreeze]: Unspecified error. Unfreeze error: [Backup job failed.
[03.01.2018 13:41:32] <18> Error    Cannot create a shadow copy of the volumes containing writer's data.
[03.01.2018 13:41:32] <18> Error    A VSS critical writer has failed. Writer name: [Microsoft Exchange Writer]. Class ID: [{76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}]. Instance ID: [{eaaa415f-aefd-4d13-a357-79ea5f00b4e6}]. Writer's state: [VSS_WS_FAILED_AT_FREEZE]. Error code: [0x800423f2].] (System.Runtime.InteropServices.COMException)
[03.01.2018 13:41:32] <18> Error       at VeeamProxyClient.GenerateComException(basic_string<char\,std::char_traits<char>\,std::allocator<char> >* excMsg, Int32 errHrCode)
[03.01.2018 13:41:32] <18> Error       at VeeamProxyClient.RegenerateCLRException(exception* excBase)
[03.01.2018 13:41:32] <18> Error       at VeeamProxyClient.CCliProxyRpcInvoker.DoRpc(String cmdName, String inXml, String& outXml)
[03.01.2018 13:41:32] <18> Error       at Veeam.Backup.ProxyProvider.CProxyRpcInvoker.Call(String methodName, CProxyInvokeInputArg inputArgs, Boolean secureCall)
[03.01.2018 13:41:32] <18> Error     Error code: 0x80004005 (Veeam.Backup.Common.CCppCOMException)
[03.01.2018 13:41:32] <18> Error    Failed to invoke func [Unfreeze]: Unspecified error. Unfreeze error: [Backup job failed.
[03.01.2018 13:41:32] <18> Error     (Veeam.Backup.Common.CCppComponentException)
[03.01.2018 13:41:32] <18> Error    Cannot create a shadow copy of the volumes containing writer's data.
[03.01.2018 13:41:32] <18> Error     (Veeam.Backup.Common.CCppComponentException)
[03.01.2018 13:41:32] <18> Error    A VSS critical writer has failed. Writer name: [Microsoft Exchange Writer]. Class ID: [{76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}]. Instance ID: [{eaaa415f-aefd-4d13-a357-79ea5f00b4e6}]. Writer's state: [VSS_WS_FAILED_AT_FREEZE]. Error code: [0x800423f2].] (Veeam.Backup.Common.CCppComponentException)
[03.01.2018 13:41:32] <18> Error       at Veeam.Backup.ProxyProvider.CProxyRpcInvoker.Call(String methodName, CProxyInvokeInputArg inputArgs, Boolean secureCall)
[03.01.2018 13:41:32] <18> Error       at Veeam.Backup.Core.CVssProxyInvoker.Invoke(String funcName, TimeSpan timeout, IVssProxyArg arg, Boolean secureCall)
[03.01.2018 13:41:32] <18> Error       at Veeam.Backup.Core.CVssProxyInvoker.Invoke(String funcName, IVssProxyArg arg, Boolean secureCall)
[03.01.2018 13:41:32] <18> Error       at Veeam.Backup.VssProvider.CVssControl.Unfreeze(Guid jobId, UInt32 ttl)
[03.01.2018 13:41:32] <18> Error    VMware snapshot creation time 19 seconds. If vCenter database is present in guest, exclusion may need to be added manually. Follow Veeam KB1051
[03.01.2018 13:41:32] <18> Error    VSSControl: Failed to freeze guest over network, wait timeout   at Veeam.Backup.VssProvider.CVssControl.Unfreeze(Guid jobId, UInt32 ttl)
[03.01.2018 13:41:32] <18> Error       at Veeam.Backup.Core.CViGuestVssFreezer.Unfreeze(Boolean createPersistentVssSnapshot, Boolean snapshotSuccessed)
[03.01.2018 13:41:32] <18> Error    VSSControl: Failed to freeze guest over network, wait timeout (System.TimeoutException)
[03.01.2018 13:41:32] <18> Error       at Veeam.Backup.VssProvider.CVssControl.Unfreeze(Guid jobId, UInt32 ttl)
[03.01.2018 13:41:32] <18> Error       at Veeam.Backup.Core.CViGuestVssFreezer.Unfreeze(Boolean createPersistentVssSnapshot, Boolean snapshotSuccessed)
[03.01.2018 13:41:34] <18> Info     Checking if VSS persistent snapshot allowed. Result: False

When OS application log starts reporting ESE/ESENT Event ID 2005/2001 at 13:41:05 and times out after 20sec.

It would be interesting to reproduce the issue with clean Exchange 2010 installation. It should go in timeout the same way and then fail over to VSS persistent snapshot (not possible with SBS2011 - see the last line of the exerpt above).

We have retargeted the job to run against VMWare host directly.
It works: this way VSS freeze takes between 7 and 13 secs (as it has been against VC with 9.5U2).

I understand it could not be an option for everybody.

so please keep working with support team.

In our case it appears to be waste of time. They suggest me to oprimize my SBS installation. After the issue is isolated and workaround applied it has few sense. I will keep the support case open untill a fix is available from Veeam.
Shunx239
Novice
 
Posts: 8
Liked: 2 times
Joined: Sat Feb 11, 2017 9:39 pm
Location: Milan
Full Name: Nikolay

Re: SBS2011/Exchange VSS Issue - 9.5U3

Veeam Logoby Gostev » Mon Jan 15, 2018 10:30 pm

Hi all, just wanted to update that according to the internal communication I've received, the issue has been understood (elongated JIT compilation time during the first vSphere API interaction) and the hot fix is now in testing. Thanks!
Gostev
Veeam Software
 
Posts: 22179
Liked: 2610 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: SBS2011/Exchange VSS Issue - 9.5U3

Veeam Logoby gatorheelz » Mon Jan 15, 2018 10:32 pm

That's great news Gostev. We were debating between a new backup chain (bypassing vCenter) or crash consistent backups.
gatorheelz
Influencer
 
Posts: 18
Liked: 3 times
Joined: Mon Jun 19, 2017 4:38 pm
Full Name: gatorheelz

Re: SBS2011/Exchange VSS Issue - 9.5U3

Veeam Logoby ic_mcourtney » Mon Jan 15, 2018 11:59 pm

Gostev wrote:Hi all, just wanted to update that according to the internal communication I've received, the issue has been understood (elongated JIT compilation time during the first vSphere API interaction) and the hot fix is now in testing. Thanks!


That's great news Gostev! Please keep us updated about when the hotfix is available. I've held off on upgrading to U3 because I was concerned that we'd see this issue in our environment; I'd be much happier to do the upgrade know that there is a hotfix available, should I need it.
ic_mcourtney
Influencer
 
Posts: 20
Liked: 2 times
Joined: Thu Sep 22, 2016 7:15 am
Full Name: mcourtney

Re: SBS2011/Exchange VSS Issue - 9.5U3

Veeam Logoby rkovhaev » Tue Jan 16, 2018 7:11 pm

hotfix is available, feel free to open up a support case with us
rkovhaev
Veeam Software
 
Posts: 35
Liked: 12 times
Joined: Mon May 17, 2010 6:49 pm
Location: hockey night in canada
Full Name: Rustam

Re: SBS2011/Exchange VSS Issue - 9.5U3

Veeam Logoby gatorheelz » Wed Jan 24, 2018 11:21 pm 1 person likes this post

The hotfix worked in our case.
gatorheelz
Influencer
 
Posts: 18
Liked: 3 times
Joined: Mon Jun 19, 2017 4:38 pm
Full Name: gatorheelz

Re: SBS2011/Exchange VSS Issue - 9.5U3

Veeam Logoby Shunx239 » Thu Jan 25, 2018 6:40 pm 1 person likes this post

I confirm the hotfix worked in our case too. Not the orginal one but reissued on Jan 18. Running fine for three days already.

Seemed the issue was because of SOAP reporting VM Snapshot creation with delay, relative dll got patched.

Glad things are working now, one day somebody should explain the difference with direct interaction to ESX host.
Shunx239
Novice
 
Posts: 8
Liked: 2 times
Joined: Sat Feb 11, 2017 9:39 pm
Location: Milan
Full Name: Nikolay

Re: SBS2011/Exchange VSS Issue - 9.5U3

Veeam Logoby mcz » Fri Feb 09, 2018 9:21 am

Ibeme85 wrote:I already opened a case relating a possible VSS issue.

In my case VSS doesn't seem to fail but in a Sure Backup Job, the SBS2011 doesn't reboot out of AD Restore Mode into normal Mode and so the SureBackup Job fails.

Backups created with U2 Work. Backups created with U3 don't.

Case 02434632


I hade exactly the same issue and we were investigating this case (#02452383) for a long time and finally T3 engineer has found a change in the veeam guest helper. He wrote the following to me:

That was a change in veeam guest helper process, which takes place both in backup and restore. I'll keep you informed on the updates from RnD and QA


Please not that I'm currently waiting for the hotfix.

I'm just wondering why T2 hadn't got the information that there was a change in backup/restore-logic. Because I asked if there was a change in U3 and T2 requested this information and told me that there weren't any changes. If we had known that, it would have saved us a lot of time...

What I also wannted to mention is that this T3-engineer is fantastic. He explained everything I was asking for and had a very good and deep knowledge. And for me it is always fun to talk with people who are really "close to the code" and are directly involved in building the software. So thanks veeam for giving us the opportunity! :D
mcz
Expert
 
Posts: 203
Liked: 19 times
Joined: Tue Jul 19, 2016 8:39 am
Full Name: Michael

Re: SBS2011/Exchange VSS Issue - 9.5U3

Veeam Logoby Ibeme85 » Sat Feb 10, 2018 8:58 am

Hi there,

thank you for the information.
I also worked for about 2 month with a tier2 engineer on this issue.

We did many remote sessions and I sent many logs. I insisted that it needs to be a change that was made from U2 to U3, because nothing else changed and backups that were created in U2 did still work in SureBackup.

The case was send many times to RnD (which I supose is Tier 3) but it wasn't possible to reproduce the issue.

I invested about 10 hours in this issue that I can't bill our customer being a problem in the product.

Yesterday my Tier 2 Engineer wrote me that it's a change to the timeout of the Veeam Guest Helper Process.

Now I'm also waiting for a fix.
Ibeme85
Novice
 
Posts: 3
Liked: never
Joined: Sun Dec 03, 2017 7:43 pm
Full Name: Bjoern Ebner

Re: SBS2011/Exchange VSS Issue - 9.5U3

Veeam Logoby Ibeme85 » Wed Feb 14, 2018 1:20 pm

I received a fix for the issue with SureBackup yesterday.
Now SureBackup is working fine again.

A private fix should be available now.
Ibeme85
Novice
 
Posts: 3
Liked: never
Joined: Sun Dec 03, 2017 7:43 pm
Full Name: Bjoern Ebner

PreviousNext

Return to Veeam Backup & Replication



Who is online

Users browsing this forum: Craqcatoa, KevinJ and 41 guests