Netapp CIFS/NFS Share Backup via Snapshot

[DaStivi]

Hi,
anyone using the new File Share (NAS) Backup Feature of v10 to Backup Files from ah Netapp Share allready?
i'm trying to use the NetApp Snapshot as Source, as some Files are always locked on the share for specific reasons and so cannot be back'd up.. so using VSS/Snapshot should fix this..
when configuring the Share there is the advanced option to choose from direct, VSS or using specific snapshot path...

the last option, "specific" Path is some kinda weird, i could trigger of course a snapshot via script, or even via netapp snapshot sheduling, but the next issue is that the snapshot names are date & time specific, so there is another line in the script needed to change the "latest" snapshot creation to ah common name that doesn't change...

using the VSS option could be an option, but is failing here on "access denied"... i see there is some "remote vss options" on netapp side available, they using this with snapmanager SQL and HyperV to backup from SMB3 shares... but clearly this states that on netapp side the "continuously available share" must be enabled, one doneside of this, its not recommended to enable this feature on an "normal user" share.. as every client is then sending heartbeats to the share whats stressing the netapp storage controllers alot! if tested this now with ah 2nd Share to the same folder on my netapp where i enabled the feature.. still getting the access denied error... my backup user should have every rights needed...

another thing i found in netapp kb articles. this should only work with hyperv vms (vhd) and sql server files anyway, at least its reads like that:

SnapManager for Hyper-V specifies the list of virtual machine files in the SMB shares that the shadow copy service needs to copy.
By providing a targeted list of virtual machine files, the shadow copy service does not need to creates shadow copies of all the files in the share.

You can use Remote VSS to perform share-based backups of Hyper-V virtual machine files that are stored on a CIFS server.

Microsoft Remote VSS (Volume Shadow Copy Services) is an extension of the existing Microsoft VSS infrastructure. Previously, VSS could be used for backup services only for data stored on local disk. This limited the use of VSS to applications that store data either on a local disk or on SAN-based storage. With Remote VSS, Microsoft has extended the VSS infrastructure to support shadow copy of SMB shares. Server applications such as Hyper-V are now storing VHD files on SMB file shares. With these new extensions, it is possible to take application consistent shadow copies for virtual machines that store data and configuration files on shares.

Microsoft Hyper-V over SMB 3.0 with ONTAP: Best Practices PDF

[DaStivi]

i found another limitation for remote vss:

A maximum directory depth of five subdirectories is allowed for shadow copy creation.
This is the directory depth over which the shadow copy service creates a shadow copy backup set. Shadow copy creation fails if directories containing virtual machine file are nested deeper than five levels. This is intended to limit the directory traversal when cloning the share. The maximum directory depth can be changed by using a CIFS server option.

[Dima P.]

Hello DaStivi,

Take a look at this blog post Advanced Veeam NAS Backup from Marko. It explains how backup via alternative path should be configured with NetApp.

[Adam.Bergh]

Hi Stephan,

I'm Veeam's Global NetApp Architect. Happy to assist with any questions you might have on using ONTAP snapshots as a source for Veeam NAS backups. Also @marco.horstmann is an expert here as well.

Using VSS to create/remove a snapshot is the recommended approach for SMB share backup from ONTAP. Be sure that you have DNS set up properly and are adding the share to Veeam with DNS and not IP address, for example "\\svm-name\share-name" and not "\\192.168.100.200\share-name". VSS needs DNS to work properly.

On my lab ONTAP systems, I did not have to change any default settings to get VSS to work with Veeam. I've tested ONTAP 9.x.

On the link you posted regarding directory depth I don't think applies to what Veeam is doing. I've got test shares that are much deeper and have not had any issues with the snapshot creation that we are doing.

Please let us know if you're still having issues with VSS snapshot creation with SMB shares. For NFS shares, please see Marco's powershell scripts over at https://github.com/VeeamHub/powershell/ ... kup4NetApp

Regards,
Adam Bergh

[DaStivi]

Hi Adam,
i've used the DNS Name, its resolvable to the ip of the SVM...
i've done some tests, installed snapdrive and the snapmanager HyperV as i thought maybe something is missing to trigger the remote vss, but still i receive the "Access Denied Error code: 0x00000005 Failed to obtain shadow path for backup item ....... failed to perform pre-backup tasks"

the user i'm using to access the smb file share is administrator on the netapp and svm are there any special "privileges" needed? i can change something in the svm settings for the groups...

[rpop]

Hello,

I followed th very good post of Marco Horstmann, very interesting.
But unable to reply in his post, seems to be a Captcha problem.

Just one question, i create a batch file to invoke snapshot NetApp + credentials, started manually everything works very well, but from the veeam backup job, it don't.
Veeam seems to be not able to create the snapshot NetApp,
in the post Marco mean to be careffully with the credential part (Local system), i am not understand, sorry

I created de crendentials xml file from my admin local account on the Veeam B&R server, i think it's the problem because all Veeam services starts with "LOCAL SYSTEM" , right ?

Thank you for your help

Best regards
Daniel

[Adam.Bergh]

Stephan,

I've done some tests, installed snapdrive and the snapmanager HyperV

Veeam's NAS backup is strictly for unstructured data, and not for Hyper-V, and does not require any NetApp software. Maybe there is some confusion on what you're trying to protect, and the Veeam capability you are selecting.

Can you describe the data you are trying to protect? Where does this data reside? On a Windows Server or directly off of an SMB share in ONTAP?

[DaStivi]

Veeam's NAS backup is strictly for unstructured data, and not for Hyper-V, and does not require any NetApp software. Maybe there is some confusion on what you're trying to protect, and the Veeam capability you are selecting.

Can you describe the data you are trying to protect? Where does this data reside? On a Windows Server or directly off of an SMB share in ONTAP?

Hi, no sorry for causing confusions... i'm trying to protect ah SMB/Cifs Fileshare (of course directly) on an NetApp Ontap (9.1p17)
i just read through netapp requirements for Remote VSS and Netapp states specifically HyperV and SQL (as these services are alowed to run on an SMB3.0 Share) and just because of this i tried all the requirements for "Remote VSS" in Veeam too... as i don't see what could be wrong on the other side as it just don't work here, and everyone is telling me there should be nothing special that this gonna work...

as i wrote allready, the user i'm accessing the share is full administrator on the netapp, the share/Cifs SVM is SMB3.0, the share is browsable, continously-available (only for testing right now, but doesn't make any difference, but as i also wrote earlier this shouldn't be setting for normal user accessing shares, reason see above, and show-previous-versions and showsnapshot is enabled too...

[marco.horstmann]

Hi Stephan, sorry for late reply. I had not seen before that Adam has mentioned me.

The only time I had really problems to get access to a share was a name resolution problem.
I had added the share to VBR using the ip address and my server was not able to resolve it to
a NetBIOS name which is needed for vssadmin.

You can try the vss part manually to see if this is the initial problem.

Code: Select all

begin backup
add volume \\svm-cifs\share1
create
delete shadows volume \\svm-cifs\share1

@rpop The problem with my script is that it always runs from the security context of VBR which is by default LOCALSYSTEM. So in my blog post I used psexec to start a shell within LOCALSYSTEM context to save the encrypted password to a file. With this credentials you're able to start the script from VBR. See how I did it: https://horstmann.in/store-powershell-c ... encrypted/

[marco.horstmann]

Hi Stephan,

I just had a call with @Herbert Sz and figured out that in his lab the VBR server is not
a domain member. For this reason he was in my opinion not able to request remote VSS snapshots.
Herbert will test if it is needed that VBR server is in domain or just the file proxys.

He will give you feedback about the results.

For all how maybe read this later. Remote VSS Support seems to be only supported by Microsoft if both
the requester and the server how hosts the share are joined in the same domain. See
https://docs.microsoft.com/en-us/previo ... v%3Dws.11)
for reference.

Kind regards
Marco

[Herbert Sz]

I just finalized my tests, so collecting @marco.horstmann's remarks and my findings:
.) filerproxy and netapp's cifs-server's system name must be in the same AD-domain (VBR server can be in a workgroup or other domain),
.) cifs-server's system name must be resolvable via reverse DNS,
Everything else should work out of the box, at least it worked for me with Ontap 9.4:
created CIFS-Server on SVM connecting it to AD-Domain, added fileshare.
Configured this fileshare into VBR server with advanced option "backup from Microsoft VSS snapshot", defined a Windows Server as filerproxy in above mentioned AD-Domain, created filer backupjob using this filerproxy, and RVSS worked fine.
No further specialities were necessary in my case.
Fileshare must be SMB3, as described in the docs.

[DaStivi]

hey again, i've done some testing yesterday with Herbert, one of the tests was to create ah new share on the same SVM that does not like rvss (still) and this share did worked well sofar!
i checked every setting now on the other share, and cross referenced it with my newly created share... don't see any differences .. still the other won't rvss successfully...

i will open an case soon so that support can check this...

one test i will do before opening a case, i gonna copy everything from my othershare to the newly created one, maybe somewhere in the files or filestructure (folder depth?) is the key to the issue...

[rpop]

Hi marco.horstmann
Thank you for your reply, it's ok, it works very well !

rpop

[D.Lee]

I've got error on calling snapshots from NetApp through VBR like Daniels' case here: file-shares-f57/netapp-cifs-nfs-share-b ... 64899.html

use script on Netapp powershell to create snapshot - ok
put same script on VBR as pre-scripte - exit with failure (see below extract from log)

a support case 04291323 was created and support replied below:
It seems the storage snapshot script is not run directly through Veeam platform, that is not supported, please make sure all operations are run directly through script.

I am wondering if there's any error to execute the pre-job scripts to call Netapp ONTAP for snapshot creation (connection error/credentials/etc...)
already followed the suggested step for powershell credential creation
https://horstmann.in/store-powershell-c ... encrypted/

Any idea on the error is appreciated, thanks.

Code: Select all

[28.07.2020 16:32:46] <01> Info     Starting job mode: 'Normal'
[28.07.2020 16:32:46] <01> Info     [Session] Id 'eaac9dbf-725a-4e3b-bc6a-9866d54657fe', State 'Working'
[28.07.2020 16:32:46] <01> Info     Running pre-job script
[28.07.2020 16:32:46] <01> Info     Executing custom command [powershell.exe], arguments [ -ExecutionPolicy ByPass -Command " try {& 'C:\backup\Invoke-NASBackup.ps1' -PrimaryCluster "192.168.8.220" -PrimarySVM "petal_svm1" -PrimaryShare "petal_cifs" -PrimaryClusterCredentials "C:\backup\1.xml" -ErrorAction Stop } catch { exit 1 } "]
[28.07.2020 16:32:47] <01> Warning  Pre-job script terminated with exit code 1
[28.07.2020 16:32:47] <01> Info     [CCloudConnectionHolder] Target repository is not CloudRepository
[28.07.2020 16:32:47] <01> Info     [JobLease] Creating job lease keeper. Lease timeout: '01:00:00', update timeout '60'
[28.07.2020 16:32:47] <01> Info     [LeaseKeeper] Created, ttl 60sec

[MaPf]

I just wanted to submit my findings, as I also had an exit code 1 on script execution.

I also used the script from Marco Horstmann (thanks for that) to create a snapshot from a NFS volume.
Looking through logs I found the following in the NASBackup.log:

Code: Select all

Trying to connect to SVM conna_NFS_CIFS_FR4 on cluster connac1.conacc.local 
The term 'Connect-NcController' is not recognized as the name of a cmdlet, function, script file, or operable program. Check the spelling of the name, or if a path was included, verify that the path is correct and try again.

My suspicion then was, that the Netapp Tools were not correctly installed.
So just to be sure I copied the Modules folder from C:\Program Files (x86)\NetApp\NetApp PowerShell Toolkit to C:\Program Files (x86)\WindowsPowerShell and C:\Program Files\WindowsPowerShell.

After that the Pre-job and Post-job scripts went fine

[virtualdon]

If you don't want to have to deal with scripts to rename the NetApp snapshots, you can also change the snapshot names to something predictable using the vol modify command :

cluster::> vol modify -volume vol1 -sched-snap-name ordinal -vserver vsm1

This will create snapshots with more traditional 7-mode naming such as:

hourly.0
hourly.1
nightly.0
nightly.1
weekly.0

Details here:

https://kb.netapp.com/Advice_and_Troubl ... _ONTAP_9.0

[Dima P.]

Hello folks,

Wanted to share some good news: with latest Veeam B&R v11 it's no logger required to rely on scripts - you can register NetApp (or Isilon) as a Filer device, add it to the job and backup job itself will trigger the snapshot creation, use it as alternative source for backup and delete the snapshot whenever backup is done.

Additionally, you can convert your existing backup files to new Filer source via PowerShell scrip to avoid new backup chain creation. Happy testing!

[Herbert Sz]

I already tried it, works like a charm!

[DaStivi]

Hello, whats the real difference here now compared to the "old" method where the snapshot was triggert through SMB3 protocol? i see that now on NFS Share as benefit, because there you had to use the script isn't it? the downside, you need now Enterprise+ License for the Storage Integration ?!

edit2: i'd reread the whole post... and might see also some other benefits now... so we had the discussion about the limititions, that the vbr server and cifs shares needs to be in the same domain etc... so the new way of handling the snapshot would help here to ... still please read my next post, as i still miss some additional snapshot handling features now that could be possible now...

edit:
btw. converting was some troublesome process... i think the description is ah little bit odd... as the docs (https://helpcenter.veeam.com/docs/backu ... ml?ver=110) state

Code: Select all

$nasBackup = Get-VBRNASBackup -Name "Daily SMB1 Backup"
$nasServer = Get-VBRNASServer -Name "\\WinSRV2020\Documents"
Convert-VBRNASBackupSANFormat -Backup $nasBackup -Server $nasServer

for the $nasServer i did find that i still got the old SMB value back.. but that didn't worked on the convert cmd then.. instead i had to parse the "SANSMB" Object, but that was also some trouble as, as running the get-vbrnasserver don't show names... still filtering works..

Code: Select all

PS C:\Windows\system32> Get-VBRNASServer


Type                        : SMB
Path                        : \\srvfile01\Users
AccessCredentials           : domain\usr-backup-service
ProcessingMode              : VSSSnapshot
StorageSnapshotPath         :
ProxyMode                   : Automatic
SelectedProxyServer         : {}
DirectBackupFailoverEnabled : False
Id                          : d51d4819-96cf-4314-8f5e-d33e2c6d9ca6
CacheRepository             : Veeam.Backup.Core.CBackupRepository
BackupIOControlLevel        : Highest

Type                     : SANSMB
AccessCredentials        : domain\usr-backup-service
ProxyMode                : Automatic
SelectedProxyServers     : {}
EnableSnapDiff           : False
InheritSettingsFromFiler : True
Id                       : 9d42e3d4-ccf0-4e15-a1b3-088a7083fb2e
CacheRepository          : Veeam.Backup.Core.CBackupRepository
BackupIOControlLevel     : Highest

Type                     : SANSMB
AccessCredentials        : domain\usr-backup-service
ProxyMode                : Automatic
SelectedProxyServers     : {}
EnableSnapDiff           : False
InheritSettingsFromFiler : True
Id                       : 16faedbe-80a8-4e63-b97c-14968ba19e15
CacheRepository          : Veeam.Backup.Core.CBackupRepository
BackupIOControlLevel     : Highest

Type                     : SANSMB
AccessCredentials        : domain\usr-backup-service
ProxyMode                : Automatic
SelectedProxyServers     : {}

etc, all "shares" got listet on the SAN Server....

running the command with -name *user* i only got the expected 2 outputs...

Code: Select all

Get-VBRNASServer -Name *Users*


Type                        : SMB
Path                        : \\srvfile01\Users
AccessCredentials           : acp\usr-backup-service
ProcessingMode              : VSSSnapshot
StorageSnapshotPath         :
ProxyMode                   : Automatic
SelectedProxyServer         : {}
DirectBackupFailoverEnabled : False
Id                          : d51d4819-96cf-4314-8f5e-d33e2c6d9ca6
CacheRepository             : Veeam.Backup.Core.CBackupRepository
BackupIOControlLevel        : Highest

Type                     : SANSMB
AccessCredentials        : acp\usr-backup-service
ProxyMode                : Automatic
SelectedProxyServers     : {}
EnableSnapDiff           : False
InheritSettingsFromFiler : True
Id                       : 6f34fe16-15d8-488a-a489-7b9ea0eb4bb9
CacheRepository          : Veeam.Backup.Core.CBackupRepository
BackupIOControlLevel     : Highest

[DaStivi]

one other thing i still miss, if now the storage integration is used, i would have assumed it would be possible to trigger "additional" snapshot creation on the volume, like in the vm jobs... that veeam creates an storage snapshot and don't delete them afterbackup finish... or even might trigger an snapvaul/mirror update....

[restore-helper]

Thank you Dastivi, I finally managed to convert from SMB to the new Backup Mode. Only to find out that the Backup Copy Job was not converted and there for was a full Backup in size.
Is there a way to convert a Backup Copy Job to?
If not, am I right that then the Backup Copy will always be a full Backup in size.

[DaStivi]

you sure you converted the primary job correct? job was orphaned under the files view? and then re mapped the backup repo to the job? could be that is ah bug.. because i would assume that there shouldn't be any change to the backupcopy if its done correctly... i would suggest to open ah case to just verify it.. not sure if its possible to investigate this afterwards...

maybe its needed to setup that again, normal SMB Job, backup 2 times, backup copyjob etc. then convert... ah little time consuming as i didn't had any copy jobs i can't test this..

[restore-helper]

Hi,

I tried it agine

Convert-VBRNASBackupSANFormat -Backup $nasBackup -Server $nasServer
is succesfull

Convert-VBRNASBackupSANFormat -Backup $nasBackupCopy -Server $nasServer
Convert-VBRNASBackupSANFormat : Unable to map a backup copy from a regular file share to a backup copy from a NAS filer
In Zeile:1 Zeichen:1
+ Convert-VBRNASBackupSANFormat -Backup $nasBackupCopy -Server $nasServ ...
+ ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~
+ CategoryInfo : NotSpecified: (:) [Convert-VBRNASBackupSANFormat], Exception
+ FullyQualifiedErrorId : System.Exception,Veeam.Backup.PowerShell.Cmdlets.ConvertVBRNASBackupSANFormat

converting the backup copy job fails.
Is converting a backup copy job suported at all?

[Dima P.]

Hello folks,

Unfortunately it's a bug - conversion scripts ignore backup copy backup and, as a result, next backup copy job starts full backup from scratch. We are discussing the possible fix.

[rpop]

Hello Dima,

did you resolve your problem ?

Indeed, i issued this conversion problem last week too.
I opened a case yesterday

My conversion command worked for all my "old standard" SMB share, these shares was transformed as expected in new "NetApp filer shares" (post-conversion), but !!! i lose each archives for my concerned nas backup jobs (i don't see archives restore points at "Disk" view in the console now), the good news is all data archives seems to be still present in the dedicated storage (HPE Storeonce for mid-long term retention)
and at "Disk" (Copy) View, "old" SMB Shares and "new" NetApp filer shares wasn't merged (i see both old and new shares )
and i confirm "Full" copy jobs started just after the conversion.

Regards

RPOP

[Dima P.]

Hello RPOP,

Can you please share the case ID with me? Worst case scenario - archive should be still available for recovery via Veeam B&R but could be 'detached' from the job. Thank you!