Failed VM's backup on hyper-v 2012 cluster with smb3 storage

[pibutaviz]

ID 00622311

Hi good people.
We are looking for solution to backup virtual machines in our hyper-v failover cluster.
As i learned from internet veeam backup is the best solution for this.
Now we are trying to implement it to our cluster but have troubles with vss.

Our cluster is based on SMB3 storage.
The scheme is:
DC - domain controller with Windows Server 2012 Std (not R2)
ST - storage server with SMB3 share for virtual machines (\\ST\storage) Windows Server 2012 Std with File server and VSS agent for file server roles
NODE1-3 - three virtualization nodes under Hyper-V 2012
BK - backup server under Windows Server 2012 Std with veeam backup free and attached DAS for backups (disk E:\)

Cluster works fine - high availability, migration and so on are perfect between nodes. But i can't backup VMs with veeam coz i got 2 errors:

Code: Select all

15.08.2014 12:30:30 :: Unable to create snapshot (Fileshare Provider) (mode: Crash consistent). Details: Failed to check whether provider '{89300202-3cec-4981-9171-19f59559e0f2}' supports shadow copies on the volume '\\ST\STORAGE\'.
Vss error: '0x80042304'
--tr:Failed to add volumes to the snapshot set.
--tr:Failed to perform pre-backup tasks.

15.08.2014 12:30:44 :: Unable to allocate processing resources. Error: Failed to check whether provider '{89300202-3cec-4981-9171-19f59559e0f2}' supports shadow copies on the volume '\\ST\STORAGE\'.
Vss error: '0x80042304'
--tr:Failed to add volumes to the snapshot set.
--tr:Failed to perform pre-backup tasks.

As i see - vss agent software provider for file servers on storage server cant' initialize creating shadow copy. But i don't understand why?

On servers ST BK and NODE1-3 no errors in windows events are detected. Only one message:
VSS was stopped for time out. (we have russian version of OS - this is translated message)

I have checked all that i can:
1. For disk with shared folder on ST (\\st\storage) VSS is enabled and configured for unlimited storage
2. On all servers microsoft software vss provider is present
ST:

Code: Select all

PS C:\Windows\system32> vssadmin list providers
vssadmin 1.1 - Программа командной строки для администрирования службы теневого копирования томов
(C) Корпорация Майкрософт (Microsoft Corportion), 2001-2012.

Имя поставщика: "Microsoft File Share Shadow Copy provider"
   Тип поставщика: Общая папка
   Id поставщика: {89300202-3cec-4981-9171-19f59559e0f2}
   Версия: 1.0.0.1

Имя поставщика: "Microsoft Software Shadow Copy provider 1.0"
   Тип поставщика: Системный
   Id поставщика: {b5946137-7b9f-4925-af80-51abd60b20d5}
   Версия: 1.0.0.7

NODE1-3:

Code: Select all

C:\Users\vadm>vssadmin list providers
vssadmin 1.1 - Программа командной строки для администрирования службы теневого
копирования томов
(C) Корпорация Майкрософт (Microsoft Corportion), 2001-2012.

Имя поставщика: "Microsoft CSV Shadow Copy Helper Provider"
   Тип поставщика: Программное обеспечение
   Id поставщика: {26d02d81-6aac-4275-8504-b9c6edc5261d}
   Версия: 1.0.0.1

Имя поставщика: "Microsoft CSV Shadow Copy Provider"
   Тип поставщика: Программное обеспечение
   Id поставщика: {400a2ff4-5eb1-44b0-8a05-1fcac0bcf9ff}
   Версия: 1.0.0.1

Имя поставщика: "Microsoft Software Shadow Copy provider 1.0"
   Тип поставщика: Системный
   Id поставщика: {b5946137-7b9f-4925-af80-51abd60b20d5}
   Версия: 1.0.0.7

3. STG1 is added to veeam as smb3 server and node1-3 as hyper-v cluster
4. firewall is disabled for clear test
5. all resources are accessed by domain administrator and veeam software runs from domain administrator account

I would be grateful for any help to resolve this issue.

P.S. On another lab stand (copy of this topology) Microsoft System Center 2012 SP1 DPM backups virtual machines without problem. We have tried altaro hyper-v backup and on first stand it works, but on this not. Same error about vss.

[Dima P.]

Hello Vladimir,

I noticed that Support team asked you to create shadow copies of the volumes manually, what is the result? In addition to "vssadmin list providers" I would check the registry at HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VSS\Providers\ - only one entry from MS VSS provider should be in the list.

[dynanet]

hi pibutaviz

i had the same error, in our case, the SOFS cluster (2 clustered fileserver) where having 3 ip adresses. 2 for smb and 1 for management.
our veeam server only had a management ip.

As i resolved the hostname of both sofs server, i got all three ips in roundrobin.

I made a hostfile entry on the veeam server, so the sofs only resolved to the management ip.

since then backup started working.

maybe this helps you.

greets

dynanet

[pibutaviz]

Hello Vladimir,

I noticed that Support team asked you to create shadow copies of the volumes manually, what is the result? In addition to "vssadmin list providers" I would check the registry at HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VSS\Providers\ - only one entry from MS VSS provider should be in the list.

Have reinstalled cluster from scratch - same thing.
There is result of your and support requests.

hyper-v node:

C:\Users\vadm>diskshadow
Microsoft DiskShadow, версия 1.0
(C) Корпорация Майкрософт (Microsoft Corporation), 2012
На компьютере: NODE1, 21.08.2014 16:19:48

DISKSHADOW> add volume \\ST\STORAGE\ provider {89300202-3cec-4981-9171-19f59559e
0f2}
Код поставщика не соответствует зарегистрированному поставщику VSS.
Чтобы зарегистрировать поставщика с VSS, см. инструкции по установке поставщиков
.

DISKSHADOW> exit

C:\Users\vadm>vssadmin list providers
vssadmin 1.1 - Программа командной строки для администрирования службы теневого
копирования томов
(C) Корпорация Майкрософт (Microsoft Corportion), 2001-2012.

Имя поставщика: "Microsoft CSV Shadow Copy Helper Provider"
Тип поставщика: Программное обеспечение
Id поставщика: {26d02d81-6aac-4275-8504-b9c6edc5261d}
Версия: 1.0.0.1

Имя поставщика: "Microsoft CSV Shadow Copy Provider"
Тип поставщика: Программное обеспечение
Id поставщика: {400a2ff4-5eb1-44b0-8a05-1fcac0bcf9ff}
Версия: 1.0.0.1

Имя поставщика: "Microsoft Software Shadow Copy provider 1.0"
Тип поставщика: Системный
Id поставщика: {b5946137-7b9f-4925-af80-51abd60b20d5}
Версия: 1.0.0.7


SMB3 shared storage node ST:

C:\Windows\system32>vssadmin list providers
vssadmin 1.1 - Программа командной строки для администрирования службы теневого
копирования томов
(C) Корпорация Майкрософт (Microsoft Corportion), 2001-2012.

Имя поставщика: "Microsoft File Share Shadow Copy provider"
Тип поставщика: Общая папка
Id поставщика: {89300202-3cec-4981-9171-19f59559e0f2}
Версия: 1.0.0.1

Имя поставщика: "Microsoft Software Shadow Copy provider 1.0"
Тип поставщика: Системный
Id поставщика: {b5946137-7b9f-4925-af80-51abd60b20d5}
Версия: 1.0.0.7

About registry - on what host I need to check this HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Services\VSS\Providers path? On storage or hyper-v host? I'v checked on both - and there are same providers as vssadmin shows.

[pibutaviz]

hi pibutaviz

i had the same error, in our case, the SOFS cluster (2 clustered fileserver) where having 3 ip adresses. 2 for smb and 1 for management.
our veeam server only had a management ip.

As i resolved the hostname of both sofs server, i got all three ips in roundrobin.

I made a hostfile entry on the veeam server, so the sofs only resolved to the management ip.

since then backup started working.

maybe this helps you.

greets

dynanet

I am using another scheme in my clusters - we have 4 x 1Gbps interfaces in all servers. I'm bounding all interfaces to one aggregated port using LACP on windows servers and cisco switch. All traffic (management, cluster, client) is managed by QoS. Btw u don't have SOFS - i'm using only simple smb3 FS.
Each server has only 1 ip address. And all cluster nodes are perfectly resolved in DNS and AD.

[dynanet]

Hi

bad news again

after the backups worked yesterday, i ran into problems again today.

Our Veeam server has now the same network config as the hyper-v and sofs servers. (all win srv 2012 r2)

I made a backup job with 6 vm's. some got backed up and some not. after i rerun the job others get backed up and others not...

Warning: 22.08.2014 12:17:05 :: Unable to create snapshot (Fileshare Provider) (mode: Crash consistent). Details: Provider [{89300202-3cec-4981-9171-19f59559e0f2}] does not support shadow copies of volume [\\SOFS\CSV1\]. --tr:Failed to add volumes to the snapshot set. --tr:Failed to perform pre-backup tasks.
Error: 22.08.2014 12:17:28 :: Unable to allocate processing resources. Error: Provider [{89300202-3cec-4981-9171-19f59559e0f2}] does not support shadow copies of volume [\\SOFS\CSV1\]. --tr:Failed to add volumes to the snapshot set. --tr:Failed to perform pre-backup tasks.

Code: Select all

vssadmin list providers
vssadmin 1.1 - Volume Shadow Copy Service administrative comma
(C) Copyright 2001-2013 Microsoft Corp.

Provider name: 'Microsoft CSV Shadow Copy Helper Provider'
   Provider type: Software
   Provider Id: {26d02d81-6aac-4275-8504-b9c6edc5261d}
   Version: 1.0.0.1

Provider name: 'Microsoft CSV Shadow Copy Provider'
   Provider type: Software
   Provider Id: {400a2ff4-5eb1-44b0-8a05-1fcac0bcf9ff}
   Version: 1.0.0.1

Provider name: 'Microsoft File Share Shadow Copy provider'
   Provider type: Fileshare
   Provider Id: {89300202-3cec-4981-9171-19f59559e0f2}
   Version: 1.0.0.1

Provider name: 'Microsoft Software Shadow Copy provider 1.0'
   Provider type: System
   Provider Id: {b5946137-7b9f-4925-af80-51abd60b20d5}
   Version: 1.0.0.7

this looks the same on both sofs servers.

i already changed the max. concurrent snapshots on the shared folders to "1".

thanks for your help.

dynanet

[Dima P.]

dynanet,
Thanks for the info. I found the similar thread, the resolution was to re add hosts to VBR console (and re installing all Veeam services as well). Let me know if that works in you case.