Availability for the Always-On Enterprise
Shunx239
Novice
Posts: 8
Liked: 2 times
Joined: Feb 11, 2017 9:39 pm
Full Name: Nikolay
Location: Milan
Contact:

Re: SBS2011/Exchange VSS Issue - 9.5U3

Post by Shunx239 » Jan 03, 2018 2:25 pm

I am pessimistic, IMHO the core team must be involved. The support will help documenting issue of course.

Before the update, 6sec freeze had been sufficient on my system.

In the last run (9.5U3) the timescale is as follows:

13:41:05 VSS freeze started
13:41:07 Create virtual machine snapshot - Started
13:41:08 Create virtual machine snapshot - Completed
13:41:25 Shadow copy freeze ended (aborted)

As far as I understand the following step causes the problem (KB1680):
Return of snapshot information via VIM API
These steps should usually be near-instantaneous, but if the vCenter is heavily loaded or has a high latency to the ESXi hosts, the delay may be significant.
The vCenter is the same, I blame Veeam for this issue

bpayne
Enthusiast
Posts: 52
Liked: 12 times
Joined: Jan 20, 2015 2:07 pm
Full Name: Brandon Payne
Contact:

Re: SBS2011/Exchange VSS Issue - 9.5U3

Post by bpayne » Jan 03, 2018 3:34 pm

Has anyone been able to nail down the operation systems affected (only affecting SBS2011?)? I have an Exchange 2010 environment running on Windows Server 2008 R2 and wondering if I will be affected by this. I can't risk Exchange transaction logs not truncating. I have been waiting to install Update 3 because I am following this thread. Thanks

Shunx239
Novice
Posts: 8
Liked: 2 times
Joined: Feb 11, 2017 9:39 pm
Full Name: Nikolay
Location: Milan
Contact:

Re: SBS2011/Exchange VSS Issue - 9.5U3

Post by Shunx239 » Jan 03, 2018 4:19 pm 1 person likes this post

I believe other Exchange 2010 installations are affected as well by the freeze timeout, just not beeing reported as failures since Veeam fails over to another snapshot technology unavailable for SBS acting as domain controller.
New in v8
To overcome this VSS limitation, Veeam Backup & Replication utilizes the Microsoft VSS persistent snapshots technology for backup of Microsoft Exchange VMs. If Microsoft Exchange fails to be frozen within the allowed period of time, Veeam Backup & Replication automatically fails over to the persistent snapshot mechanism. To learn more about this new feature please read:
http://helpcenter.veeam.com/backup/80/v ... shots.html

Shunx239
Novice
Posts: 8
Liked: 2 times
Joined: Feb 11, 2017 9:39 pm
Full Name: Nikolay
Location: Milan
Contact:

Re: SBS2011/Exchange VSS Issue - 9.5U3

Post by Shunx239 » Jan 03, 2018 5:25 pm

Rather clear now, seems to be a real bug in the new build.

Look what we had with 9.5U2 (Task.VMName.vm-223.log):

Code: Select all

[20.12.2017 21:02:39] <24> Info     VSSControl: Frozen in 12sec
[20.12.2017 21:02:40] <24> Info     [VimApi] Create snapshot, ref "vm-223", name "VEEAM BACKUP TEMPORARY SNAPSHOT", description "Please do not delete this snapshot. It is being used by Veeam Backup.", memory "False", quiesce "False"
[20.12.2017 21:02:42] <24> Info     [Soap] Snapshot VM 'vm-223' was created. Ref: 'snapshot-3272'
[20.12.2017 21:02:42] <24> Info     VSSControl: Unfreezing
[20.12.2017 21:02:48] <24> Info     CGuestVssSnapshotKeeper 2: Created, ttl 1200sec
The same with 9.5U3 looks like this:

Code: Select all

[03.01.2018 13:41:06] <18> Info     VSSControl: Frozen in 20sec
[03.01.2018 13:41:06] <18> Info     [VimApi] Create snapshot, ref "vm-223", name "VEEAM BACKUP TEMPORARY SNAPSHOT", description "Please do not delete this snapshot. It is being used by Veeam Backup.", memory "False", quiesce "False"
[03.01.2018 13:41:26] <18> Info     [Soap] Snapshot VM 'vm-223' was created. Ref: 'snapshot-3373'
[03.01.2018 13:41:26] <18> Info     VSSControl: Unfreezing
[03.01.2018 13:41:32] <18> Error    Failed to call RPC function 'Vss.Unfreeze': Error code: 0x80004005. Failed to invoke func [Unfreeze]: Unspecified error. Unfreeze error: [Backup job failed. Cannot create a shadow copy of the volumes containing writer's data. A VSS critical writer has failed. Writer name: [Microsoft Exchange Writer]. Class ID: [{76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}]. Instance ID: [{eaaa415f-aefd-4d13-a357-79ea5f00b4e6}]. Writer's state: [VSS_WS_FAILED_AT_FREEZE]. Error code: [0x800423f2].].. InParams: '<InputArguments><ttl value="1200" /><VssSessionId value="{d04802b5-bdd8-40bb-930b-48fed11d211e}" /></InputArguments>'.
With 9.5U3 the SOAP returns success but not immediately, on 20sec timeout.
Unfortunately it is too late for issuing 'Vss.Unfreeze'

With 9.5U2 the timeout was 12sec, it was reasonable.

Another question is why it returned quicker with 9.5U2, but the 20sec timeout of 9.5U3 can't work at all!

Dima P.
Veeam Software
Posts: 8919
Liked: 659 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: SBS2011/Exchange VSS Issue - 9.5U3

Post by Dima P. » Jan 03, 2018 8:10 pm 1 person likes this post

Hi folks,

All cases mentioned in this thread are escalated, so please keep working with support team. If you face similar problem please submit a case and update this thread with your case ID. Please also doublecheck if the described workaround works in your environment. Thank you in advance.

Shunx239
Novice
Posts: 8
Liked: 2 times
Joined: Feb 11, 2017 9:39 pm
Full Name: Nikolay
Location: Milan
Contact:

Re: SBS2011/Exchange VSS Issue - 9.5U3

Post by Shunx239 » Jan 07, 2018 10:29 am

I was probably wrong stating the max timeout was determined by the call to VIM API.

Still confused by the timestamps. Here is the Veeam log excerpt:

Code: Select all

[03.01.2018 13:40:46] <18> Info     VSSControl: Freezing. Job id: a8d52fd7-6676-4eeb-9cb3-5550dc8143b1. Job Name: Afrodite Backup. VssCopyOnly: False
[03.01.2018 13:40:46] <18> Info     VSSControl: Snapshots will be use for indexing. Volumes spec: <VolumeFilter vssWillBeUsedForIndexing="True"><Include><Path>F:\</Path><Path>D:\</Path><Path>E:\</Path><Path>Z:\</Path><Path>C:\WSUS\</Path><Path>C:\</Path></Include><Exclude><Path>C:\WSUS\</Path><Path>%windir%</Path><Path>%ProgramFiles%</Path><Path>%ProgramFiles(x86)%</Path><Path>%ProgramW6432%</Path><Path>%TEMP%</Path></Exclude></VolumeFilter>
[03.01.2018 13:41:06] <18> Info     VSSControl: Frozen in 20sec
[03.01.2018 13:41:06] <18> Info     [VimApi] Create snapshot, ref "vm-223", name "VEEAM BACKUP TEMPORARY SNAPSHOT", description "Please do not delete this snapshot. It is being used by Veeam Backup.", memory "False", quiesce "False"
[03.01.2018 13:41:26] <18> Info     [Soap] Snapshot VM 'vm-223' was created. Ref: 'snapshot-3373'
[03.01.2018 13:41:26] <18> Info     VSSControl: Unfreezing
[03.01.2018 13:41:32] <18> Error    Failed to call RPC function 'Vss.Unfreeze': Error code: 0x80004005. Failed to invoke func [Unfreeze]: Unspecified error. Unfreeze error: [Backup job failed. Cannot create a shadow copy of the volumes containing writer's data. A VSS critical writer has failed. Writer name: [Microsoft Exchange Writer]. Class ID: [{76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}]. Instance ID: [{eaaa415f-aefd-4d13-a357-79ea5f00b4e6}]. Writer's state: [VSS_WS_FAILED_AT_FREEZE]. Error code: [0x800423f2].].. InParams: '<InputArguments><ttl value="1200" /><VssSessionId value="{d04802b5-bdd8-40bb-930b-48fed11d211e}" /></InputArguments>'. 
[03.01.2018 13:41:32] <18> Error     Error code: 0x80004005
[03.01.2018 13:41:32] <18> Error    Failed to invoke func [Unfreeze]: Unspecified error. Unfreeze error: [Backup job failed.
[03.01.2018 13:41:32] <18> Error    Cannot create a shadow copy of the volumes containing writer's data.
[03.01.2018 13:41:32] <18> Error    A VSS critical writer has failed. Writer name: [Microsoft Exchange Writer]. Class ID: [{76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}]. Instance ID: [{eaaa415f-aefd-4d13-a357-79ea5f00b4e6}]. Writer's state: [VSS_WS_FAILED_AT_FREEZE]. Error code: [0x800423f2].] (System.Runtime.InteropServices.COMException)
[03.01.2018 13:41:32] <18> Error       at VeeamProxyClient.GenerateComException(basic_string<char\,std::char_traits<char>\,std::allocator<char> >* excMsg, Int32 errHrCode)
[03.01.2018 13:41:32] <18> Error       at VeeamProxyClient.RegenerateCLRException(exception* excBase)
[03.01.2018 13:41:32] <18> Error       at VeeamProxyClient.CCliProxyRpcInvoker.DoRpc(String cmdName, String inXml, String& outXml)
[03.01.2018 13:41:32] <18> Error       at Veeam.Backup.ProxyProvider.CProxyRpcInvoker.Call(String methodName, CProxyInvokeInputArg inputArgs, Boolean secureCall)
[03.01.2018 13:41:32] <18> Error     Error code: 0x80004005 (Veeam.Backup.Common.CCppCOMException)
[03.01.2018 13:41:32] <18> Error    Failed to invoke func [Unfreeze]: Unspecified error. Unfreeze error: [Backup job failed.
[03.01.2018 13:41:32] <18> Error     (Veeam.Backup.Common.CCppComponentException)
[03.01.2018 13:41:32] <18> Error    Cannot create a shadow copy of the volumes containing writer's data.
[03.01.2018 13:41:32] <18> Error     (Veeam.Backup.Common.CCppComponentException)
[03.01.2018 13:41:32] <18> Error    A VSS critical writer has failed. Writer name: [Microsoft Exchange Writer]. Class ID: [{76fe1ac4-15f7-4bcd-987e-8e1acb462fb7}]. Instance ID: [{eaaa415f-aefd-4d13-a357-79ea5f00b4e6}]. Writer's state: [VSS_WS_FAILED_AT_FREEZE]. Error code: [0x800423f2].] (Veeam.Backup.Common.CCppComponentException)
[03.01.2018 13:41:32] <18> Error       at Veeam.Backup.ProxyProvider.CProxyRpcInvoker.Call(String methodName, CProxyInvokeInputArg inputArgs, Boolean secureCall)
[03.01.2018 13:41:32] <18> Error       at Veeam.Backup.Core.CVssProxyInvoker.Invoke(String funcName, TimeSpan timeout, IVssProxyArg arg, Boolean secureCall)
[03.01.2018 13:41:32] <18> Error       at Veeam.Backup.Core.CVssProxyInvoker.Invoke(String funcName, IVssProxyArg arg, Boolean secureCall)
[03.01.2018 13:41:32] <18> Error       at Veeam.Backup.VssProvider.CVssControl.Unfreeze(Guid jobId, UInt32 ttl)
[03.01.2018 13:41:32] <18> Error    VMware snapshot creation time 19 seconds. If vCenter database is present in guest, exclusion may need to be added manually. Follow Veeam KB1051
[03.01.2018 13:41:32] <18> Error    VSSControl: Failed to freeze guest over network, wait timeout   at Veeam.Backup.VssProvider.CVssControl.Unfreeze(Guid jobId, UInt32 ttl)
[03.01.2018 13:41:32] <18> Error       at Veeam.Backup.Core.CViGuestVssFreezer.Unfreeze(Boolean createPersistentVssSnapshot, Boolean snapshotSuccessed)
[03.01.2018 13:41:32] <18> Error    VSSControl: Failed to freeze guest over network, wait timeout (System.TimeoutException)
[03.01.2018 13:41:32] <18> Error       at Veeam.Backup.VssProvider.CVssControl.Unfreeze(Guid jobId, UInt32 ttl)
[03.01.2018 13:41:32] <18> Error       at Veeam.Backup.Core.CViGuestVssFreezer.Unfreeze(Boolean createPersistentVssSnapshot, Boolean snapshotSuccessed)
[03.01.2018 13:41:34] <18> Info     Checking if VSS persistent snapshot allowed. Result: False
When OS application log starts reporting ESE/ESENT Event ID 2005/2001 at 13:41:05 and times out after 20sec.

It would be interesting to reproduce the issue with clean Exchange 2010 installation. It should go in timeout the same way and then fail over to VSS persistent snapshot (not possible with SBS2011 - see the last line of the exerpt above).

We have retargeted the job to run against VMWare host directly.
It works: this way VSS freeze takes between 7 and 13 secs (as it has been against VC with 9.5U2).

I understand it could not be an option for everybody.
so please keep working with support team.
In our case it appears to be waste of time. They suggest me to oprimize my SBS installation. After the issue is isolated and workaround applied it has few sense. I will keep the support case open untill a fix is available from Veeam.

Gostev
Veeam Software
Posts: 23199
Liked: 2963 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: SBS2011/Exchange VSS Issue - 9.5U3

Post by Gostev » Jan 15, 2018 10:30 pm

Hi all, just wanted to update that according to the internal communication I've received, the issue has been understood (elongated JIT compilation time during the first vSphere API interaction) and the hot fix is now in testing. Thanks!

gatorheelz
Influencer
Posts: 18
Liked: 3 times
Joined: Jun 19, 2017 4:38 pm
Full Name: gatorheelz
Contact:

Re: SBS2011/Exchange VSS Issue - 9.5U3

Post by gatorheelz » Jan 15, 2018 10:32 pm

That's great news Gostev. We were debating between a new backup chain (bypassing vCenter) or crash consistent backups.

ic_mcourtney
Influencer
Posts: 20
Liked: 2 times
Joined: Sep 22, 2016 7:15 am
Full Name: mcourtney
Contact:

Re: SBS2011/Exchange VSS Issue - 9.5U3

Post by ic_mcourtney » Jan 15, 2018 11:59 pm

Gostev wrote:Hi all, just wanted to update that according to the internal communication I've received, the issue has been understood (elongated JIT compilation time during the first vSphere API interaction) and the hot fix is now in testing. Thanks!
That's great news Gostev! Please keep us updated about when the hotfix is available. I've held off on upgrading to U3 because I was concerned that we'd see this issue in our environment; I'd be much happier to do the upgrade know that there is a hotfix available, should I need it.

rkovhaev
Veeam Software
Posts: 39
Liked: 21 times
Joined: May 17, 2010 6:49 pm
Full Name: Rustam
Location: hockey night in canada
Contact:

Re: SBS2011/Exchange VSS Issue - 9.5U3

Post by rkovhaev » Jan 16, 2018 7:11 pm

hotfix is available, feel free to open up a support case with us

gatorheelz
Influencer
Posts: 18
Liked: 3 times
Joined: Jun 19, 2017 4:38 pm
Full Name: gatorheelz
Contact:

Re: SBS2011/Exchange VSS Issue - 9.5U3

Post by gatorheelz » Jan 24, 2018 11:21 pm 1 person likes this post

The hotfix worked in our case.

Shunx239
Novice
Posts: 8
Liked: 2 times
Joined: Feb 11, 2017 9:39 pm
Full Name: Nikolay
Location: Milan
Contact:

Re: SBS2011/Exchange VSS Issue - 9.5U3

Post by Shunx239 » Jan 25, 2018 6:40 pm 1 person likes this post

I confirm the hotfix worked in our case too. Not the orginal one but reissued on Jan 18. Running fine for three days already.

Seemed the issue was because of SOAP reporting VM Snapshot creation with delay, relative dll got patched.

Glad things are working now, one day somebody should explain the difference with direct interaction to ESX host.

mcz
Expert
Posts: 219
Liked: 26 times
Joined: Jul 19, 2016 8:39 am
Full Name: Michael
Contact:

Re: SBS2011/Exchange VSS Issue - 9.5U3

Post by mcz » Feb 09, 2018 9:21 am

Ibeme85 wrote:I already opened a case relating a possible VSS issue.

In my case VSS doesn't seem to fail but in a Sure Backup Job, the SBS2011 doesn't reboot out of AD Restore Mode into normal Mode and so the SureBackup Job fails.

Backups created with U2 Work. Backups created with U3 don't.

Case 02434632
I hade exactly the same issue and we were investigating this case (#02452383) for a long time and finally T3 engineer has found a change in the veeam guest helper. He wrote the following to me:
That was a change in veeam guest helper process, which takes place both in backup and restore. I'll keep you informed on the updates from RnD and QA
Please not that I'm currently waiting for the hotfix.

I'm just wondering why T2 hadn't got the information that there was a change in backup/restore-logic. Because I asked if there was a change in U3 and T2 requested this information and told me that there weren't any changes. If we had known that, it would have saved us a lot of time...

What I also wannted to mention is that this T3-engineer is fantastic. He explained everything I was asking for and had a very good and deep knowledge. And for me it is always fun to talk with people who are really "close to the code" and are directly involved in building the software. So thanks veeam for giving us the opportunity! :D

Ibeme85
Novice
Posts: 3
Liked: never
Joined: Dec 03, 2017 7:43 pm
Full Name: Bjoern Ebner
Contact:

Re: SBS2011/Exchange VSS Issue - 9.5U3

Post by Ibeme85 » Feb 10, 2018 8:58 am

Hi there,

thank you for the information.
I also worked for about 2 month with a tier2 engineer on this issue.

We did many remote sessions and I sent many logs. I insisted that it needs to be a change that was made from U2 to U3, because nothing else changed and backups that were created in U2 did still work in SureBackup.

The case was send many times to RnD (which I supose is Tier 3) but it wasn't possible to reproduce the issue.

I invested about 10 hours in this issue that I can't bill our customer being a problem in the product.

Yesterday my Tier 2 Engineer wrote me that it's a change to the timeout of the Veeam Guest Helper Process.

Now I'm also waiting for a fix.

Ibeme85
Novice
Posts: 3
Liked: never
Joined: Dec 03, 2017 7:43 pm
Full Name: Bjoern Ebner
Contact:

Re: SBS2011/Exchange VSS Issue - 9.5U3

Post by Ibeme85 » Feb 14, 2018 1:20 pm

I received a fix for the issue with SureBackup yesterday.
Now SureBackup is working fine again.

A private fix should be available now.

Post Reply

Who is online

Users browsing this forum: Bing [Bot], Google [Bot] and 46 guests