Large number of snapshot attach/detach requests on EMC Unity array due to Veeam storage scanning

[BennyDC]

We would like some advice on a behavior we have been seeing ever since we integrated our SAN array with Veeam.

First of all, here is our configuration.

• In one of our DCs, we have a single EMC Unity 350F Fibre Channel storage array where all our production data sits on.
• Our backup server (running Veeam Build: 10.0.14854) is a standalone server, with its own dedicated storage - this server is connected to the SAN switches the array is connected to.
• The EMC array is integrated with Veeam (over management network), and we are carrying backups using SAN snapshots over the FC network.
• In addition to backups, we create array level snapshots - every 2 hours we take a snapshot of all volumes on the array and keep these snapshots for 5 days.

The snapshots created by the array are visible on the Veeam server, under Storage Infrastructure, as a result of the storage integration. However to keep the array snapshot info up to date, Veeam seems to be constantly scanning the storge array volumes. This in turn is creating high amounts of snapshot attach/detach requests on our EMC Unity 350F array, resulting in a lot of spikes on our array controller CPUs – which we can do without!

We have been in touch with Veeam support about this(Case # 04453937), this seems to be the default behavior of Veeam according to them. However we are interested to know if anyone else is seeing this behavior with EMC arrays. We do have other DCs with NetApp storage and we don’t see this behavior.

Now Veeam has provided a way to stop the scanning – using the following registry keys. But we would like to avoid this if possible, as it’s a non-standard fix.

SanMonitorDisabled
Type: REG_DWORD
Default value: 0 (change to 1 to disable automatic rescans)
Path: HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication\
Description: disable SanMonitor

Name: SanMonitorTimeout
Type: REG_DWORD
Default value: 600 (seconds)
Path: HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication\
Description: defines how frequently we should update SAN infrastructure and run incremental rescan for updated objects

[PetrM]

Hi Benny,

It seems strange to me that you don't see the same issue with CPU spikes during manual attach or detach of snapshots, I believe that more accurate way of issue reproduce might be needed to raise a support request with EMC as well. I guess you should connect Veeam support directly with EMC team in order to determine the root cause.

I've quickly reviewed the support case and I see that our engineer needs to have some time in order to find the best way of research, so let's wait for reply from our engineers.

Thanks!

[foggy]

We do have other DCs with NetApp storage and we don’t see this behavior.

Rescan works a bit differently on NetApp storage (using FlexClone).

As for the registry values, I wouldn't disable the entire rescan but either decrease its frequency (SanMonitorTimeout) or disable VMFS rescan (SanRescanSnapshotsParseFS).

[BennyDC]

On the 23rd of Nov, we implemented the registry to disable SAN monitor, to see if it has a positive effect on CPU utilization on array (approx. 11:10 AM GMT). But this made no difference, the attach/detach requests would not stop (requests were still coming in at 11:55 AM GMT).

To make sure requests were actually coming from the Veeam server, we powered it off (12:41 PM GMT) - and instantly requests stopped, and the CPU went down considerably (there were no backups running at the time). So, without a doubt Veeam is responsible for attach / detach requests and the increase in CPU utilization.

The registry key Veeam recommended to implement to stop scanning did not work, Veeam support scratching their heads at the moment.

SanMonitorDisabled
Type: REG_DWORD
Default value: 0 (change to 1 to disable automatic rescans)
Path: HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication\
Description: disable SanMonitor

[foggy]

Have you restarted the Veeam Backup Service after setting the registry value?

[BennyDC]

Hi Foggy,

Of course we did, we even rebooted the whole backup server and monitor the aray for while.
No impact from the reg key however

[foggy]

I will check regarding that registry value, have you checked the two other ones mentioned?

[foggy]

As an update - we've checked the registry key and it worked fine. The OP was able to isolate the issue from Veeam B&R and closed the case.