Veeam Backup and Netapp Snapshots/Snapmirror difficulties

[sandsturm]

Hi all

We use Veeam together with Netapp storage integration and have configured our backup jobs with a Netapp snapshot (on primary storage system) and a Netapp Snapvault (to a secondary storage system) as Secondary targets in the backup job. Beside that we use the Netapp scheduler to execute snapshots and snapmirrors on the same volumes, but the Netapp scheduler for snapmirrors runs only between 12:00am and 7:00pm (to not interfere with Veeam), all Veeam backup jobs are schedule between 7pm and 11:59pm.

Usually, the Netapp volume on primary storage system has on snapshot called something like snapmirror.2e903547-9902-00a87dice_2148723.2019-11-27_1023000, which is used as reference for the next snapmirror transfer to the secondary system. The secondary system has always two of these snapmirror called snapshots, the snapmirror knows which one the last transfered snapshot was and transfers all data changed after that reference point.

Here you'll the snapshots on primary system of such a volume from yesterday evening:

Code: Select all

Vserver  Volume   Snapshot                                  Size Total% Used%
-------- -------- ------------------------------------- -------- ------ -----
vserver01 volume01
                  
                  VeeamSourceSnapshot_BKP_LIVA-NSL3-VA-01.2019-11-24_2203 22.17GB 1% 3%
                  snp_hourly_business_01.2019-11-25_0717 148.3MB     0%    0%
                  snp_hourly_business_01.2019-11-25_0917 13.10GB     1%    2%
                  snp_hourly_business_01.2019-11-25_1117 95.16MB     0%    0%
                  snp_hourly_business_01.2019-11-25_1317 11.37GB     1%    2%
                  snp_hourly_business_01.2019-11-25_1517 102.5MB     0%    0%
                  snp_hourly_business_01.2019-11-25_1717  9.21GB     0%    1%
                  snp_hourly_business_01.2019-11-25_1917 64.27MB     0%    0%
                  snp_daily_evening_01.2019-11-25_2027    6.72GB     0%    1%
                  VeeamSourceSnapshot_BKP_LIVA-NSL3-VA-01.2019-11-25_2201 5.93GB 0% 1%
                  snp_hourly_business_01.2019-11-26_0717  8.74GB     0%    1%
                  snp_hourly_business_01.2019-11-26_0917  6.32GB     0%    1%
                  snp_hourly_business_01.2019-11-26_1117  6.86GB     0%    1%
                  snp_hourly_business_01.2019-11-26_1317 63.77MB     0%    0%
                  ppr_test_01                            27.52MB     0%    0%
                  snp_hourly_business_01.2019-11-26_1517  8.00GB     0%    1%
                  snp_hourly_business_01.2019-11-26_1717 68.79MB     0%    0%
                  snp_hourly_business_01.2019-11-26_1917  8.62GB     0%    1%
                  snp_daily_evening_01.2019-11-26_2027   49.43MB     0%    0%
                  snp_every20_01.2019-11-26_2101         36.66MB     0%    0%
                  snp_every20_01.2019-11-26_2121         33.63MB     0%    0%
                  snp_every20_01.2019-11-26_2141          7.96MB     0%    0%
                  snp_every20_01.2019-11-26_2201          4.29MB     0%    0%
                  VeeamSourceSnapshot_BKP_LIVA-NSL3-VA-01.2019-11-26_2202 1.09GB 0% 0%
                  snp_every20_01.2019-11-26_2221          4.62GB     0%    1%
                  snp_every20_01.2019-11-26_2241         18.62MB     0%    0%

snapshots on secondary system:

Code: Select all

Vserver  Volume   Snapshot                                  Size Total% Used%
-------- -------- ------------------------------------- -------- ------ -----
vserver02 bkp_volume
 
                  VeeamSnapVault_BKP_LIVA-NSL3-VA-01.2019-11-24_2203 13.59GB 0% 2%
                  snp_hourly_business_01.2019-11-25_0717 34.46MB     0%    0%
                  snp_hourly_business_01.2019-11-25_0917 36.14MB     0%    0%
                  snp_hourly_business_01.2019-11-25_1117 28.44MB     0%    0%
                  snp_hourly_business_01.2019-11-25_1317 34.40MB     0%    0%
                  snp_hourly_business_01.2019-11-25_1517 24.84MB     0%    0%
                  snp_hourly_business_01.2019-11-25_1717 39.11MB     0%    0%
                  snp_hourly_business_01.2019-11-25_1917 23.12MB     0%    0%
                  VeeamSnapVault_BKP_LIVA-NSL3-VA-01.2019-11-25_2201 4.94GB 0% 1%
                  snp_hourly_business_01.2019-11-26_0717 36.63MB     0%    0%
                  snp_hourly_business_01.2019-11-26_0917 38.71MB     0%    0%
                  snp_hourly_business_01.2019-11-26_1117 29.70MB     0%    0%
                  snp_hourly_business_01.2019-11-26_1317 31.24MB     0%    0%
                  ppr_test_01                            19.08MB     0%    0%
                  snp_hourly_business_01.2019-11-26_1517 25.21MB     0%    0%
                  snp_hourly_business_01.2019-11-26_1717 34.19MB     0%    0%
                  snp_hourly_business_01.2019-11-26_1917 27.91MB     0%    0%
                  snapmirror.2e903547-4ffd-11e7-9902-00a098b0d1ce_2149359463.2019-11-26_192300 8.92GB 0% 1%
                  VeeamSnapVault_BKP_LIVA-NSL3-VA-01.2019-11-26_2202 348KB 0% 0%

these snapshot were visible during the run of the Veeam backup job and there we can't see a snapshot called snapmirror-anything on primary system. Is Veeam removing this snapmirror snapshot? And the main thing we dont understand is the following: we can see the snapshot snp_daily_evening_01.2019-11-26_2027 on primary system, but this snapshot was not mirrored to the secondary system.. we don't understand why. In my opinion, Veeam just runs the existing snapmirror relationsship, but as we see on the above example, thats not right because this snp_daily_evening_01.2019-11-26_2027 was not mirrored to the secondary site, but if Veeam does not change anything at the defined snapvault policy this snapshot must also be mirrored to the secondary site.
After the veeam process has finished, we don't see the snapmirror-anything snapshot anymore, there is just the Veeam-triggered Snapshot visible on primary and secondary site which will be used as a reference for the next snapmirrors to run.
Due to this behavior, we lose some of our snapshots from the primary site. Is this a bug in Veeam or where does this behavior come from? (I hope I could explain the problem in a manner which is understandable...)

thx
sandsturm

[Andreas Neufert]

I guess this is a side effect from the point that we only transfer our snapshot to the snapmirror destination.
Will ask our NetApp SA to comment here.

[Adam.Bergh]

Hi Sandstrum,

I'd like to comment first on a best practice with Veeam and NetApp using our storage integrations. You should allow Veeam Backup & Replication to create and manage the schedule of all snapshots on a particular volume to avoid potential conflicts of two different snapshot management utilities conflicting with each other. There should not be a good reason to use a scheduled snapshot schedule in ONTAP directly along side Veeam created snapshots of the same volume. The best practice here would be to create a separate Veeam backup job that does snapshots/snapvault only tasks that are currently being managed in ONTAP. In addition to being able to create application consistent snapshots, you will get much more predictable results of the retention of snapshots at both the primary and secondary ONTAP systems.


Now to get back to your post, after reading it several times, it appears to me that you have a MirrorVault relationship created on the volume and you are using ONTAP to schedule the SnapMirror and Veeam to schedule the SnapVault? Is this correct?

Let's talk this over in more detail, but there are easier way to achieve your retention requirements with using just Veeam to manage all snaps and replication. If you could reply to this and state your retention requirements for snapshots on the primary and secondary system, I'll propose the proper Veeam backup job architecture to achieve this.

For example:
24 Hourly Snapshots on primary with mirror to secondary (identical retention).
30 Nightly Snapshots on primary, vault to secondary with 90 day retention.

The above would be achieved with two separate backup jobs in Veeam and give you the predictable results and single management tool for snapshots.

Thanks!
Adam Bergh
Veeam Global Alliance Architect - NetApp
adam.bergh@veeam.com

[sandsturm]

Hi Adam

thanks for your answer. Yes we use a MirrorVault relationship with ONTAP to schedule the "crash consistent" snapshots/snapvaults on these volumes. The idea of using ONTAP and not Veeam for the scheduling is that we are not dependent on a "third party" software for these snapshots.

Currently we create a crashconsistent "ONTAP" snapshot every two hours between 6am and 7pm and a daily ONTAP snapshot between 8pm and 11pm (we have about 5 different schedules for all NFS volumes to not run all daily snapshots at exactly the same time). Snapvault is running every 15 minutes between 12am and 7pm.
The veeam backup jobs are running once a day between 8pm and 12am, besides the Veeam backup, they create a snapshot and a snapvault with this backupjob.
The snapvault policy is configured to keep:
hourly 44
daily 22
weekly 13

Do I understand you right, that whenever you have a Netapp environment, you do the snapshot/snapvault scheduling for virtual machine volumes completey with Veeam?

thx
sandsturm

[Adam.Bergh]

"The idea of using ONTAP and not Veeam for the scheduling is that we are not dependent on a "third party" software for these snapshots."

Generally speaking, this is a bit like having two different backup software packages backing up the same data because you don't trust a single data protection tool. If you do this, you will inventively have to work around additional challenges or conflicts between the two protection tools.

I would recommend trusting Veeam as the single tool that manages/creates all snapshots/vaults of the volumes that contain VMware VMs. Of course it's true that if Veeam goes down you will not have data protection, so proper 24/7 monitoring of your Veeam environment is critical.

In addition, if there are any SQL/Exchange/Sharepoint/Active Directory/Oracle VMs, you're definitely going to want Application Consistent snapshots created of these applications, as crash consistent snaps may prove unreliable for recovery of those applications. More on that here: https://helpcenter.veeam.com/docs/backu ... l?ver=95u4 Ideally all snapshots would be application consistent if possible and creating these snapshots is not too disruptive to the applications.

Currently we create a crash consistent "ONTAP" snapshot every two hours between 6am and 7pm and a daily ONTAP snapshot between 8pm and 11pm (we have about 5 different schedules for all NFS volumes to not run all daily snapshots at exactly the same time). Snapvault is running every 15 minutes between 12am and 7pm.
The veeam backup jobs are running once a day between 8pm and 12am, besides the Veeam backup, they create a snapshot and a snapvault with this backupjob.
The snapvault policy is configured to keep:
hourly 44
daily 22
weekly 13

Can you clarify the retention on System 1 and System 2? Is it identical retention of the hourly/daily/weekly snapshots on both systems?

As I think I understand the retention requirements, you do have a fairly complicated requirements, which while doable in Veeam, it would require several jobs. One thing to remember about Veeam is that in any single snapshot job, we can't manage Hourly, Daily, and Weekly retention as a policy, so this would need a separate job for each retention type.

Thanks!

-Adam