We are running HyperV 2012 R2 and Veeam 9.5 We have been using this for a while backing up VM's on our ISCSI CSV volumes with no issues.
We've recently started testing NEtapp SMB 3 shares with HyperV. The Hosts all have the Snapdrive installed (which installed the Ontap Hardware provider).
Now when running a backup of a VM we migrate to SMB we get the VSS_WS_FAILED_AT_PREPARE_SNAPSHOT error and backups fail. Inside the VM (thats always backed up just fine previously we see VSS errors)
I can actually manually run a diskshadow snapshot inside the vm successfully...it seems to be only the vss thats called by the Veeam backup job that seems to fail...
If I edit the job and remove the Guest Quiese, then the jobs run successfully with Crash consistent option.
If I goto any hyperv host and manually run the diskshadow commands to do a snapshot I can see the host using the OnTap Hardware provider so I know the host is able to instruct the netapp to do the VSS ok.
If I edit the VM's and remove the Backup integration, but leave the Guest Quiese enabled, the VM's also backup with crash consistent mode successfully.
So...what piece of the puzzle am I missing that only on SMB shares do i get vss errors inside my VM's when trying to do Guest Quiese? It makes no sense...
-
Rumple
- Service Provider
- Posts: 81
- Liked: 14 times
- Joined: Mar 10, 2010 7:50 pm
- Full Name: Mark Hodges
- Contact:
-
Rumple
- Service Provider
- Posts: 81
- Liked: 14 times
- Joined: Mar 10, 2010 7:50 pm
- Full Name: Mark Hodges
- Contact:
Re: VSS_WS_FAILED_AT_PREPARE_SNAPSHOT on SMB
am I the only one using SMB shares with hyperv cluster..or just the only one to have this problem?
-
nmdange
- Veteran
- Posts: 528
- Liked: 144 times
- Joined: Aug 20, 2015 9:30 pm
- Contact:
Re: VSS_WS_FAILED_AT_PREPARE_SNAPSHOT on SMB
Can you clarify, do you have a Scale-out File Server cluster running Windows Server, or are your Hyper-V hosts talking to NetApp directly via SMB?
I'm not sure how many users use NetApp's SMB implementation. Also, not specific to NetApp, but there are many cases where hardware VSS providers may not work properly, so that may be related to your issues. There are plenty of people out there (including myself) using Windows Scale Out File Servers with SMB (and in many cases SMB Direct/RDMA using Chelsio or Mellanox NICs).
Veeam does not support Change Tracking non-Windows SMB servers unless you are running Windows Server 2016 on your Hyper-V hosts and your VMs are updated to version 8 to use the new RCT feature added in Hyper-V 2016. This is because Veeam cannot install their Change Tracking driver on the SMB servers if they aren't running Windows. This means Veeam would need to do a full backup every time. With that in mind, I would suggest staying with iSCSI until your Hyper-V environment is fully on Windows Server 2016, unless you want to stand up a scale-out file server cluster running Windows Server that sits between the NetApp and the Hyper-V hosts. There are use cases for doing that but if you only have a single small Hyper-V cluster it is probably not worth the effort.
I'm not sure how many users use NetApp's SMB implementation. Also, not specific to NetApp, but there are many cases where hardware VSS providers may not work properly, so that may be related to your issues. There are plenty of people out there (including myself) using Windows Scale Out File Servers with SMB (and in many cases SMB Direct/RDMA using Chelsio or Mellanox NICs).
Veeam does not support Change Tracking non-Windows SMB servers unless you are running Windows Server 2016 on your Hyper-V hosts and your VMs are updated to version 8 to use the new RCT feature added in Hyper-V 2016. This is because Veeam cannot install their Change Tracking driver on the SMB servers if they aren't running Windows. This means Veeam would need to do a full backup every time. With that in mind, I would suggest staying with iSCSI until your Hyper-V environment is fully on Windows Server 2016, unless you want to stand up a scale-out file server cluster running Windows Server that sits between the NetApp and the Hyper-V hosts. There are use cases for doing that but if you only have a single small Hyper-V cluster it is probably not worth the effort.
-
Rumple
- Service Provider
- Posts: 81
- Liked: 14 times
- Joined: Mar 10, 2010 7:50 pm
- Full Name: Mark Hodges
- Contact:
Re: VSS_WS_FAILED_AT_PREPARE_SNAPSHOT on SMB
We are running Windows 2012 R2 at the moment...Since all the VSS happens through the host (be it onhost or offhost, and the CBT data is ALWAYS stored on the hyperV host itself I can't see why it should not work the same as a CSV.
I was looking at my CSV volume backups and I see that Veeam calls the CSV shadow copy provider (which then does the backup. I don't have integration with the VM's (as they are customer VM's) and noticed I do not see a VSS entry in the logs for the VM
So basically it looks like Veeam with NetApp SMB shares is basically useless at the moment and doesn't really work?
I was looking at my CSV volume backups and I see that Veeam calls the CSV shadow copy provider (which then does the backup. I don't have integration with the VM's (as they are customer VM's) and noticed I do not see a VSS entry in the logs for the VM
So basically it looks like Veeam with NetApp SMB shares is basically useless at the moment and doesn't really work?
-
nmdange
- Veteran
- Posts: 528
- Liked: 144 times
- Joined: Aug 20, 2015 9:30 pm
- Contact:
Re: VSS_WS_FAILED_AT_PREPARE_SNAPSHOT on SMB
You are correct that the VSS happens on the host, but Veeam needs components to be installed on the SMB servers for it to capture CBT data through Veeam's file system filter. I'm not sure why it's failing, since I've never worked with NetApp SMB but without CBT even if you could get it to work it's not all that useful.
From https://helpcenter.veeam.com/docs/backu ... tml?ver=95
https://library.netapp.com/ecmdocs/ECMP ... A7AC9.html
https://library.netapp.com/ecmdocs/ECMP ... 65342.html
From https://helpcenter.veeam.com/docs/backu ... tml?ver=95
If you do still want to get it to work, I'd go through NetApp's documentation if you haven't. These seems like they might be helpfulAn SMB3 server or SMB3 cluster hosting the necessary file shares must be added to the backup infrastructure. Otherwise Veeam Backup & Replication will not be able to use changed block tracking for VMs residing on SMB3 shares.
https://library.netapp.com/ecmdocs/ECMP ... A7AC9.html
https://library.netapp.com/ecmdocs/ECMP ... 65342.html
-
Rumple
- Service Provider
- Posts: 81
- Liked: 14 times
- Joined: Mar 10, 2010 7:50 pm
- Full Name: Mark Hodges
- Contact:
Re: VSS_WS_FAILED_AT_PREPARE_SNAPSHOT on SMB
yeah, had another ticket open and after supplying logs and other things, one of the options was to mount the share on a server and add that server as a managed server.
Explained that you can't MOUNT a file share as a disk on a windows server but I don't think they actually understand how netapp filers work (or any other non windows based file server works)
CBT I can probably live without but what makes no sense is that from hyperv I can manually use diskshadow and voila, I get an Ontap Hardware snapshot created and the VM shows that vss worked fine inside the VM.
Running veeam (which technically should do the exact same thing) gives me an error inside the VM (which is the same VSS error in other posts). So why does VSS inside the VM (not application consistent) cause vss to fail in the guest?
Explained that you can't MOUNT a file share as a disk on a windows server but I don't think they actually understand how netapp filers work (or any other non windows based file server works)
CBT I can probably live without but what makes no sense is that from hyperv I can manually use diskshadow and voila, I get an Ontap Hardware snapshot created and the VM shows that vss worked fine inside the VM.
Running veeam (which technically should do the exact same thing) gives me an error inside the VM (which is the same VSS error in other posts). So why does VSS inside the VM (not application consistent) cause vss to fail in the guest?
-
billcouper
- Service Provider
- Posts: 153
- Liked: 34 times
- Joined: Dec 18, 2017 8:58 am
- Full Name: Bill Couper
- Contact:
Re: VSS_WS_FAILED_AT_PREPARE_SNAPSHOT on SMB
I think you are being a bit hard on the support team. Veeam mentions storage integration with NetApp, but only in the VMware user guide - not the Hyper-V guide. The Veeam support team for Hyper-V may have limited exposure to NetApp devices. It should be noted that depending on your model of filer it is possible to present the storage as iSCSI to an intermediary Windows server that could then present the SMB3 shares to your hyper-v cluster, so their suggestion isn't 100% wrong. I have never done this and can't comment on how well it would work, or if you would have to wipe/reset the filer completely in the process.
Sometimes it can be hard to integrate any backup product without significant teething issues depending how an environment is built. I had many issues integrating Veeam and was cursing it the whole time - until I finally got everything to 'click' and now I am extremely happy, especially when I compare the capabilities it provides compared to other solutions. If you can live without CBT then persist with current setup until you get it to work. Losing CBT is huge though, if you go down that path you may regret it later.
I haven't used Veeam with a Hyper-V before, so this probably not useful at all since I have never seen which options are available to you and I apologize if I am simply wasting your time (more than I already have)...... in a VMware environment there are two type of options for backup. Number one is to enable 'Quiescence' in the job settings. Number two is to enable 'Application Aware' in the job settings. Application Aware will connect inside guest and use VSS (with credentials you provide) while the Quiescence option uses hypervisor tools. I would try swapping modes for this VM to check if any improvement in the VSS error state. I have more than a few VM's that will only backup if I flip modes.
Sometimes it can be hard to integrate any backup product without significant teething issues depending how an environment is built. I had many issues integrating Veeam and was cursing it the whole time - until I finally got everything to 'click' and now I am extremely happy, especially when I compare the capabilities it provides compared to other solutions. If you can live without CBT then persist with current setup until you get it to work. Losing CBT is huge though, if you go down that path you may regret it later.
I haven't used Veeam with a Hyper-V before, so this probably not useful at all since I have never seen which options are available to you and I apologize if I am simply wasting your time (more than I already have)...... in a VMware environment there are two type of options for backup. Number one is to enable 'Quiescence' in the job settings. Number two is to enable 'Application Aware' in the job settings. Application Aware will connect inside guest and use VSS (with credentials you provide) while the Quiescence option uses hypervisor tools. I would try swapping modes for this VM to check if any improvement in the VSS error state. I have more than a few VM's that will only backup if I flip modes.
-
Rumple
- Service Provider
- Posts: 81
- Liked: 14 times
- Joined: Mar 10, 2010 7:50 pm
- Full Name: Mark Hodges
- Contact:
Re: VSS_WS_FAILED_AT_PREPARE_SNAPSHOT on SMB
Not trying to be hard on them, but suggesting someone put in a Microsoft file server infront of a double/triple digit cost tier 1 NAS/SAN provider really isn't really a viable or even logical solution.
I am perfectly happy for veeam to come back and say, using NetApp SMB as a file share location for Hyperv is not a supported configuration by Veeam.
At the moment, it seems like I am the first person in the world to setup this configuration and try and back it up with veeam.
I can always load up snapmanager for hyperV and do the backups to snapshots but at moment I only have a single NetApp so I also need to figure a way to get these off disk...which if maybe Veeam integrated with snapmanager to do the snapshot and then backed it up from the snap, I'd be ok with too.
also veeam seems to have a very strong partnership with NetApp so they must have at least 1 customer with my rather simple setup...scvmm, smb share on NetApp and hyperv cluster...
I am perfectly happy for veeam to come back and say, using NetApp SMB as a file share location for Hyperv is not a supported configuration by Veeam.
At the moment, it seems like I am the first person in the world to setup this configuration and try and back it up with veeam.
I can always load up snapmanager for hyperV and do the backups to snapshots but at moment I only have a single NetApp so I also need to figure a way to get these off disk...which if maybe Veeam integrated with snapmanager to do the snapshot and then backed it up from the snap, I'd be ok with too.
also veeam seems to have a very strong partnership with NetApp so they must have at least 1 customer with my rather simple setup...scvmm, smb share on NetApp and hyperv cluster...
-
csydas
- Expert
- Posts: 193
- Liked: 47 times
- Joined: Jan 16, 2018 5:14 pm
- Full Name: Harvey Carel
- Contact:
Re: VSS_WS_FAILED_AT_PREPARE_SNAPSHOT on SMB
Hey Mark,
I'm not exactly a netapp guru, but peeking around online there easily are others who have a set up like this, but it does need some finessing from what I see from some google-fu. Just looking at the netapp document on SMB3 shares for backups, Netapp comes with its own laundry list or requirements, on top of Microsoft's own list of requirements to get it going.
I'm guessing support's recommendation probably comes from the idea that they mostly deal with the intended Microsoft way of running it all Windows; since you can't get the necessary services running on Netapp for CBT, I don't think you're ever going to have that (and honestly I think you'll find that's more of a dealbreaker than you realize in the long run)
If I had a guess, I'd say check that the account Veeam is using and its services are running as are compatible with the user requirements from Netapp/Microsoft for such a set up (in the past we've had issues where the account running the Veeam services wasn't privileged enough for some SMB shares we were trying to offload to tape, and it's not hard for me to imagine VSS getting fussy about it either).
The few success stories I'm seeing mention they had to use Flexclone to get it working (which netapp's laundry list mentions as well), and I see you mention you installed Snapdrive. Again, I'm not Netapp guru and have only worked with VMWare+Netapp environments in the past, but more to the point, there just seems to be a bit of a laundry list of things to confirm I think before I'd talk at Veeam Engineers; likely this more needs a system engineer than technical support though.
Regarding the on-host disk shadow though, did you compare the event logs between the two events and just see what gets procc'd differently?
I'm not exactly a netapp guru, but peeking around online there easily are others who have a set up like this, but it does need some finessing from what I see from some google-fu. Just looking at the netapp document on SMB3 shares for backups, Netapp comes with its own laundry list or requirements, on top of Microsoft's own list of requirements to get it going.
I'm guessing support's recommendation probably comes from the idea that they mostly deal with the intended Microsoft way of running it all Windows; since you can't get the necessary services running on Netapp for CBT, I don't think you're ever going to have that (and honestly I think you'll find that's more of a dealbreaker than you realize in the long run)
If I had a guess, I'd say check that the account Veeam is using and its services are running as are compatible with the user requirements from Netapp/Microsoft for such a set up (in the past we've had issues where the account running the Veeam services wasn't privileged enough for some SMB shares we were trying to offload to tape, and it's not hard for me to imagine VSS getting fussy about it either).
The few success stories I'm seeing mention they had to use Flexclone to get it working (which netapp's laundry list mentions as well), and I see you mention you installed Snapdrive. Again, I'm not Netapp guru and have only worked with VMWare+Netapp environments in the past, but more to the point, there just seems to be a bit of a laundry list of things to confirm I think before I'd talk at Veeam Engineers; likely this more needs a system engineer than technical support though.
Regarding the on-host disk shadow though, did you compare the event logs between the two events and just see what gets procc'd differently?
-
Mike Resseler
- Product Manager
- Posts: 8191
- Liked: 1322 times
- Joined: Feb 08, 2013 3:08 pm
- Full Name: Mike Resseler
- Location: Belgium
- Contact:
Re: VSS_WS_FAILED_AT_PREPARE_SNAPSHOT on SMB
Hi Rumple,
From the past (if my memory serves me well) this mostly had to do with configuration. Do you have flexclone licensed on the filer?
Second. The proxy that does the heavy lifting, under what account is that service running? I also believe that the local system could have issues to the netapp in some cases and you would get VSS issues there. I believe the solution for that was to use a domain account for that service
To be honest, I am not 100 percent certain anymore about this, but checking those two things might be good
Cheers
Mike
From the past (if my memory serves me well) this mostly had to do with configuration. Do you have flexclone licensed on the filer?
Second. The proxy that does the heavy lifting, under what account is that service running? I also believe that the local system could have issues to the netapp in some cases and you would get VSS issues there. I believe the solution for that was to use a domain account for that service
To be honest, I am not 100 percent certain anymore about this, but checking those two things might be good
Cheers
Mike
-
Rumple
- Service Provider
- Posts: 81
- Liked: 14 times
- Joined: Mar 10, 2010 7:50 pm
- Full Name: Mark Hodges
- Contact:
Re: VSS_WS_FAILED_AT_PREPARE_SNAPSHOT on SMB
I am currently testing with on host proxy and I have snapdrive installed on the hosts running as a domain account (and local admin on boxes) and have defined the vsadmin account credentials.
I have the full compliment of licenses for the AFF array.
I have a ticket from netapp open as well but that's going no where slowly...
Interestingly, I am able to run diskshadow on the host and it completes successfully (although it takes over 2 minutes or so).
I've shortened the log to the relevant VSS writer.
* Including writer "Microsoft Hyper-V VSS Writer":
+ Adding component: \47AA0EE9-6DF0-4A2D-BFF5-FD3BCB194BE1
Alias VSS_SHADOW_1 for shadow ID {8d282e66-f746-4904-8430-33bf7018367b} set as environment variable.
Alias VSS_SHADOW_SET for shadow set ID {99ef9c2a-9bd8-4f1c-ab96-1bfadd01c6a8} set as environment variable.
Inserted file Manifest.xml into .cab file 2018-02-08_13-45-53_HYPERV03.cab
Inserted file BCDocument.xml into .cab file 2018-02-08_13-45-53_HYPERV03.cab
Inserted file WM0.xml into .cab file 2018-02-08_13-45-53_HYPERV03.cab
Inserted file WM1.xml into .cab file 2018-02-08_13-45-53_HYPERV03.cab
Inserted file WM2.xml into .cab file 2018-02-08_13-45-53_HYPERV03.cab
Inserted file WM3.xml into .cab file 2018-02-08_13-45-53_HYPERV03.cab
Inserted file WM4.xml into .cab file 2018-02-08_13-45-53_HYPERV03.cab
Inserted file WM5.xml into .cab file 2018-02-08_13-45-53_HYPERV03.cab
Inserted file WM6.xml into .cab file 2018-02-08_13-45-53_HYPERV03.cab
Inserted file WM7.xml into .cab file 2018-02-08_13-45-53_HYPERV03.cab
Inserted file WM8.xml into .cab file 2018-02-08_13-45-53_HYPERV03.cab
Inserted file WM9.xml into .cab file 2018-02-08_13-45-53_HYPERV03.cab
Inserted file WM10.xml into .cab file 2018-02-08_13-45-53_HYPERV03.cab
Inserted file WM11.xml into .cab file 2018-02-08_13-45-53_HYPERV03.cab
Inserted file WM12.xml into .cab file 2018-02-08_13-45-53_HYPERV03.cab
Inserted file vsstest.log into .cab file 2018-02-08_13-45-53_HYPERV03.cab
Querying all shadow copies with the shadow copy set ID {99ef9c2a-9bd8-4f1c-ab96-1bfadd01c6a8}
* Shadow copy ID = {8d282e66-f746-4904-8430-33bf7018367b} %VSS_SHADOW_1%
- Shadow copy set: {99ef9c2a-9bd8-4f1c-ab96-1bfadd01c6a8} %VSS_SHADOW_SET%
- Original count of shadow copies = 1
- Original volume name: \\myfileshare_SHARES\N_HYPERV_LUN1\ [volume not on this machine]
- Creation time: 2/8/2018 1:45:46 PM
- Shadow copy device name: \\myfileshare_SHARES\N_HYPERV_LUN1@{bce6d119-0d10-11e8-82e7-00a098befeea}
- Originating machine: myfileshare_SHARES
- Service machine: HYPERV03.myfileshare.co
- Not exposed
- Provider ID: {89300202-3cec-4981-9171-19f59559e0f2}
- Attributes: Auto_Release FileShare
when veeam runs the job though...
2/8/2018 1:24:11 PM :: Queued for processing at 2/8/2018 1:24:11 PM
2/8/2018 1:24:11 PM :: Required backup infrastructure resources have been assigned
2/8/2018 1:24:16 PM :: VM processing started at 2/8/2018 1:24:16 PM
2/8/2018 1:24:16 PM :: VM size: 60.0 GB (17.8 GB used)
2/8/2018 1:24:19 PM :: Preparing to create snapshot
2/8/2018 1:24:19 PM :: Failed to create snapshot (Fileshare Provider) (mode: Hyper-V child partition snapshot) Details: Writer 'Microsoft Hyper-V VSS Writer' is failed at 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT'.
The writer experienced a non-transient error. If the backup process is retried,
the error is likely to reoccur.
--tr:Failed to verify writers state.
--tr:Failed to perform pre-backup tasks.
2/8/2018 1:26:51 PM :: Retrying snapshot creation attempt (Writer 'Microsoft Hyper-V VSS Writer' is failed at 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT'.
The writer experienced a non-transient error. If the backup process is retried,
the error is likely to reoccur.
--tr:Failed to verify writers state.
--tr:Failed to perform pre-backup tasks.)
2/8/2018 1:26:51 PM :: Task has been rescheduled
2/8/2018 1:26:51 PM :: Queued for processing at 2/8/2018 1:26:51 PM
2/8/2018 1:26:52 PM :: Unable to allocate processing resources. Error: Writer 'Microsoft Hyper-V VSS Writer' is failed at 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT'.
The writer experienced a non-transient error. If the backup process is retried,
the error is likely to reoccur.
--tr:Failed to verify writers state.
--tr:Failed to perform pre-backup tasks.
2/8/2018 1:26:52 PM :: Network traffic verification detected no corrupted blocks
2/8/2018 1:26:52 PM :: Processing finished with errors at 2/8/2018 1:26:52 PM
I have the full compliment of licenses for the AFF array.
I have a ticket from netapp open as well but that's going no where slowly...
Interestingly, I am able to run diskshadow on the host and it completes successfully (although it takes over 2 minutes or so).
I've shortened the log to the relevant VSS writer.
* Including writer "Microsoft Hyper-V VSS Writer":
+ Adding component: \47AA0EE9-6DF0-4A2D-BFF5-FD3BCB194BE1
Alias VSS_SHADOW_1 for shadow ID {8d282e66-f746-4904-8430-33bf7018367b} set as environment variable.
Alias VSS_SHADOW_SET for shadow set ID {99ef9c2a-9bd8-4f1c-ab96-1bfadd01c6a8} set as environment variable.
Inserted file Manifest.xml into .cab file 2018-02-08_13-45-53_HYPERV03.cab
Inserted file BCDocument.xml into .cab file 2018-02-08_13-45-53_HYPERV03.cab
Inserted file WM0.xml into .cab file 2018-02-08_13-45-53_HYPERV03.cab
Inserted file WM1.xml into .cab file 2018-02-08_13-45-53_HYPERV03.cab
Inserted file WM2.xml into .cab file 2018-02-08_13-45-53_HYPERV03.cab
Inserted file WM3.xml into .cab file 2018-02-08_13-45-53_HYPERV03.cab
Inserted file WM4.xml into .cab file 2018-02-08_13-45-53_HYPERV03.cab
Inserted file WM5.xml into .cab file 2018-02-08_13-45-53_HYPERV03.cab
Inserted file WM6.xml into .cab file 2018-02-08_13-45-53_HYPERV03.cab
Inserted file WM7.xml into .cab file 2018-02-08_13-45-53_HYPERV03.cab
Inserted file WM8.xml into .cab file 2018-02-08_13-45-53_HYPERV03.cab
Inserted file WM9.xml into .cab file 2018-02-08_13-45-53_HYPERV03.cab
Inserted file WM10.xml into .cab file 2018-02-08_13-45-53_HYPERV03.cab
Inserted file WM11.xml into .cab file 2018-02-08_13-45-53_HYPERV03.cab
Inserted file WM12.xml into .cab file 2018-02-08_13-45-53_HYPERV03.cab
Inserted file vsstest.log into .cab file 2018-02-08_13-45-53_HYPERV03.cab
Querying all shadow copies with the shadow copy set ID {99ef9c2a-9bd8-4f1c-ab96-1bfadd01c6a8}
* Shadow copy ID = {8d282e66-f746-4904-8430-33bf7018367b} %VSS_SHADOW_1%
- Shadow copy set: {99ef9c2a-9bd8-4f1c-ab96-1bfadd01c6a8} %VSS_SHADOW_SET%
- Original count of shadow copies = 1
- Original volume name: \\myfileshare_SHARES\N_HYPERV_LUN1\ [volume not on this machine]
- Creation time: 2/8/2018 1:45:46 PM
- Shadow copy device name: \\myfileshare_SHARES\N_HYPERV_LUN1@{bce6d119-0d10-11e8-82e7-00a098befeea}
- Originating machine: myfileshare_SHARES
- Service machine: HYPERV03.myfileshare.co
- Not exposed
- Provider ID: {89300202-3cec-4981-9171-19f59559e0f2}
- Attributes: Auto_Release FileShare
when veeam runs the job though...
2/8/2018 1:24:11 PM :: Queued for processing at 2/8/2018 1:24:11 PM
2/8/2018 1:24:11 PM :: Required backup infrastructure resources have been assigned
2/8/2018 1:24:16 PM :: VM processing started at 2/8/2018 1:24:16 PM
2/8/2018 1:24:16 PM :: VM size: 60.0 GB (17.8 GB used)
2/8/2018 1:24:19 PM :: Preparing to create snapshot
2/8/2018 1:24:19 PM :: Failed to create snapshot (Fileshare Provider) (mode: Hyper-V child partition snapshot) Details: Writer 'Microsoft Hyper-V VSS Writer' is failed at 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT'.
The writer experienced a non-transient error. If the backup process is retried,
the error is likely to reoccur.
--tr:Failed to verify writers state.
--tr:Failed to perform pre-backup tasks.
2/8/2018 1:26:51 PM :: Retrying snapshot creation attempt (Writer 'Microsoft Hyper-V VSS Writer' is failed at 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT'.
The writer experienced a non-transient error. If the backup process is retried,
the error is likely to reoccur.
--tr:Failed to verify writers state.
--tr:Failed to perform pre-backup tasks.)
2/8/2018 1:26:51 PM :: Task has been rescheduled
2/8/2018 1:26:51 PM :: Queued for processing at 2/8/2018 1:26:51 PM
2/8/2018 1:26:52 PM :: Unable to allocate processing resources. Error: Writer 'Microsoft Hyper-V VSS Writer' is failed at 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT'.
The writer experienced a non-transient error. If the backup process is retried,
the error is likely to reoccur.
--tr:Failed to verify writers state.
--tr:Failed to perform pre-backup tasks.
2/8/2018 1:26:52 PM :: Network traffic verification detected no corrupted blocks
2/8/2018 1:26:52 PM :: Processing finished with errors at 2/8/2018 1:26:52 PM
-
Mike Resseler
- Product Manager
- Posts: 8191
- Liked: 1322 times
- Joined: Feb 08, 2013 3:08 pm
- Full Name: Mike Resseler
- Location: Belgium
- Contact:
Re: VSS_WS_FAILED_AT_PREPARE_SNAPSHOT on SMB
Hi,
Seeing this I would suggest you take a peek at the File Server VSS Agent Service. (Is it installed?).
Did you updated your ticket with this information?
Seeing this I would suggest you take a peek at the File Server VSS Agent Service. (Is it installed?).
Did you updated your ticket with this information?
-
Rumple
- Service Provider
- Posts: 81
- Liked: 14 times
- Joined: Mar 10, 2010 7:50 pm
- Full Name: Mark Hodges
- Contact:
Re: VSS_WS_FAILED_AT_PREPARE_SNAPSHOT on SMB
I just updated my ticket with this information, but you can't install the file server VSS agent on a netapp filer...
-
Rumple
- Service Provider
- Posts: 81
- Liked: 14 times
- Joined: Mar 10, 2010 7:50 pm
- Full Name: Mark Hodges
- Contact:
Re: VSS_WS_FAILED_AT_PREPARE_SNAPSHOT on SMB
For anyone still remotely interested, I am still working with netapp and we've been able to successfully call diskshadow commands on the host and perform the backup successfully.
I am now just waiting for a Level 2 resource to be assigned to my ticket so that we can get on a conference call with Netapp and work through the configuration.
Case # 02477263 if anyone in the future needs it....
I am now just waiting for a Level 2 resource to be assigned to my ticket so that we can get on a conference call with Netapp and work through the configuration.
Case # 02477263 if anyone in the future needs it....
-
Mike Resseler
- Product Manager
- Posts: 8191
- Liked: 1322 times
- Joined: Feb 08, 2013 3:08 pm
- Full Name: Mike Resseler
- Location: Belgium
- Contact:
Re: VSS_WS_FAILED_AT_PREPARE_SNAPSHOT on SMB
Just so I understand it clearly:
The level 2 resource is from NetApp?
And the fact that you can do a backup successfully comes after some changes correct? That didn't work before?
The level 2 resource is from NetApp?
And the fact that you can do a backup successfully comes after some changes correct? That didn't work before?
-
Rumple
- Service Provider
- Posts: 81
- Liked: 14 times
- Joined: Mar 10, 2010 7:50 pm
- Full Name: Mark Hodges
- Contact:
Re: VSS_WS_FAILED_AT_PREPARE_SNAPSHOT on SMB
The Level 2 resource is from Veeam that I am waiting to hear from.
I could always do a diskshadow vss backup from the hyperv servers (once I knew the command lines to use)
Veeam backups only work if I run them crash consistent but if they call VSS it always fails. Doesn't make much sense as Veeam should basically be doing the same thing as the command line (using API's I assume).
I could always do a diskshadow vss backup from the hyperv servers (once I knew the command lines to use)
Veeam backups only work if I run them crash consistent but if they call VSS it always fails. Doesn't make much sense as Veeam should basically be doing the same thing as the command line (using API's I assume).
-
Rumple
- Service Provider
- Posts: 81
- Liked: 14 times
- Joined: Mar 10, 2010 7:50 pm
- Full Name: Mark Hodges
- Contact:
Re: VSS_WS_FAILED_AT_PREPARE_SNAPSHOT on SMB
Wanted to provide an update on this issue. It appears we've hit a netapp bug that's getting fixed soon (thankfully I am on 9.1P7 at moment and not 9.2) so after the 24th I should be able to confirm if the issue is resolved. Its been a long painful road to get to this point but we are hopefully at the end of this road.
Hi Mark,
As discussed over phone last week, looks like you are hitting bug 1134071. Unfortunately there is no work around suggested for it as of now, however, it states that – “The Windows application that is trying to perform the byte lock downgrades can work around this issue by changing its locking sequence to release the exclusive byte lock first before establishing the shared byte lock for the file offset range”.
One other way to get around this would be upgrading to 9.1 p12. I am listing the public report for the bug below for your reference.
https://mysupport.netapp.com/NOW/cgi-bi ... ay=1134071
Hi Mark,
As discussed over phone last week, looks like you are hitting bug 1134071. Unfortunately there is no work around suggested for it as of now, however, it states that – “The Windows application that is trying to perform the byte lock downgrades can work around this issue by changing its locking sequence to release the exclusive byte lock first before establishing the shared byte lock for the file offset range”.
One other way to get around this would be upgrading to 9.1 p12. I am listing the public report for the bug below for your reference.
https://mysupport.netapp.com/NOW/cgi-bi ... ay=1134071
-
Mike Resseler
- Product Manager
- Posts: 8191
- Liked: 1322 times
- Joined: Feb 08, 2013 3:08 pm
- Full Name: Mike Resseler
- Location: Belgium
- Contact:
Re: VSS_WS_FAILED_AT_PREPARE_SNAPSHOT on SMB
Mark,
Many thanks for letting us know. Unfortunately I cannot go to the link (I don't have a username/ pwd for Netapp but colleagues have so I will ask them
). So 10 more days and it should get fixed, did I get that correct?
Keep us informed
Many thanks for letting us know. Unfortunately I cannot go to the link (I don't have a username/ pwd for Netapp but colleagues have so I will ask them
). So 10 more days and it should get fixed, did I get that correct?Keep us informed
-
Rumple
- Service Provider
- Posts: 81
- Liked: 14 times
- Joined: Mar 10, 2010 7:50 pm
- Full Name: Mark Hodges
- Contact:
Re: VSS_WS_FAILED_AT_PREPARE_SNAPSHOT on SMB
oops..forgot that was paywalled
They say its going to be fixed in the MArch 23rd release of 9.1 and may release of 9.2 so we will soon see...
herei s bug details.
Bug ID
1134071
Title SMB byte lock downgrades do not work on ONTAP
Duplicate of
Bug Severity 2 - System barely usable
Bug Status
Product Data ONTAP
Bug Type Unknown
Description
Formatted An issue in ONTAP causes Server Message Block (SMB) byte lock downgrades
to work incorrectly on ONTAP. When an SMB client establishes an exclusive
byte lock on a file offset range, and then subsequently wants to downgrade
the byte lock to a shared byte lock on the same file offset range -- while at
the same time does not want any other SMB client to acquire an exclusive byte
lock on the same file offset range -- the following occurs:
1. The SMB client will acquire a shared byte lock on the same file offset range
and this will be granted without any waiting period since the SMB client
already has an exclusive byte lock.
2. The SMB client will release the byte lock for that file offset
range. At this point the SMB server should release the exclusive byte lock.
Instead ONTAP misbehaves by releasing the shared byte lock that was established
on that file offset range.
3. While the application assumes that the file is left with a shared byte lock,
ONTAP has left an exclusive byte lock on that file offset range. This causes
other applications wanting to establish the shared byte lock to indefinitely
wait until the byte lock is released by the first application.
Workaround
Formatted There are no workarounds available within the ONTAP software. The Windows
application that is trying to perform the byte lock downgrades can work around
this issue by changing its locking sequence to release the exclusive byte lock
first before establishing the shared byte lock for the file offset range.
Notes
Formatted This issue has been seen with customers using SnapManager for Hyper-V to create a
VSS snapshot for VHDX files stored on SMB shares.
Note that the locking and unlocking offset ranges for the exclusive and the
shared byte lock sequences need not be exact matching ranges. They can be
overlapping each other and experience the same issue.
They say its going to be fixed in the MArch 23rd release of 9.1 and may release of 9.2 so we will soon see...
herei s bug details.
Bug ID
1134071
Title SMB byte lock downgrades do not work on ONTAP
Duplicate of
Bug Severity 2 - System barely usable
Bug Status
Product Data ONTAP
Bug Type Unknown
Description
Formatted An issue in ONTAP causes Server Message Block (SMB) byte lock downgrades
to work incorrectly on ONTAP. When an SMB client establishes an exclusive
byte lock on a file offset range, and then subsequently wants to downgrade
the byte lock to a shared byte lock on the same file offset range -- while at
the same time does not want any other SMB client to acquire an exclusive byte
lock on the same file offset range -- the following occurs:
1. The SMB client will acquire a shared byte lock on the same file offset range
and this will be granted without any waiting period since the SMB client
already has an exclusive byte lock.
2. The SMB client will release the byte lock for that file offset
range. At this point the SMB server should release the exclusive byte lock.
Instead ONTAP misbehaves by releasing the shared byte lock that was established
on that file offset range.
3. While the application assumes that the file is left with a shared byte lock,
ONTAP has left an exclusive byte lock on that file offset range. This causes
other applications wanting to establish the shared byte lock to indefinitely
wait until the byte lock is released by the first application.
Workaround
Formatted There are no workarounds available within the ONTAP software. The Windows
application that is trying to perform the byte lock downgrades can work around
this issue by changing its locking sequence to release the exclusive byte lock
first before establishing the shared byte lock for the file offset range.
Notes
Formatted This issue has been seen with customers using SnapManager for Hyper-V to create a
VSS snapshot for VHDX files stored on SMB shares.
Note that the locking and unlocking offset ranges for the exclusive and the
shared byte lock sequences need not be exact matching ranges. They can be
overlapping each other and experience the same issue.
-
Mike Resseler
- Product Manager
- Posts: 8191
- Liked: 1322 times
- Joined: Feb 08, 2013 3:08 pm
- Full Name: Mike Resseler
- Location: Belgium
- Contact:
Re: VSS_WS_FAILED_AT_PREPARE_SNAPSHOT on SMB
No worries. Thanks for the copy/paste. Keep us informed if the patch in a few weeks solves the issue
-
Rumple
- Service Provider
- Posts: 81
- Liked: 14 times
- Joined: Mar 10, 2010 7:50 pm
- Full Name: Mark Hodges
- Contact:
Re: VSS_WS_FAILED_AT_PREPARE_SNAPSHOT on SMB
As a follow up to this, we installed the latest 9.1 p12 from NetApp and voila, VMs on SMB magically backup successfully now
Apparently this fix is coming soon to 9.2 and 9.3 NetApp versions as well
Apparently this fix is coming soon to 9.2 and 9.3 NetApp versions as well
-
Mike Resseler
- Product Manager
- Posts: 8191
- Liked: 1322 times
- Joined: Feb 08, 2013 3:08 pm
- Full Name: Mike Resseler
- Location: Belgium
- Contact:
Re: VSS_WS_FAILED_AT_PREPARE_SNAPSHOT on SMB
Thanks Mark!
Really appreciated. If you happen to hear when that fix is available for 9.2 and .3 also, it would be greatly appreciated if you tell us!
Mike
Really appreciated. If you happen to hear when that fix is available for 9.2 and .3 also, it would be greatly appreciated if you tell us!
Mike
Who is online
Users browsing this forum: No registered users and 11 guests