V9 EMC Snapshot Integration Issues

[dustinn3]

I loaded V9 last week and ran into major issues this weekend. I'm not sure if the issue is with Veeam or my VNX, but when I enabled storage integrated snapshots my backup jobs started failing, Recoverpoint started failing to connect to my SAN, and my SAN was so slow my VM's wouldn't function. I stopped the jobs, turned off storage integration, and everything returned to normal.

Also, snapshots created via a backup job do not display in the Storage Infrastructure screen unless they are manually created. I tried rescanning several times on several Luns.

I have tickets open with Veeam and EMC.

[Gostev]

Hi, please add Veeam support case ID, as requested when you click New Topic. Thanks!

[dustinn3]

Case # 01508488

[elapshov]

Also, snapshots created via a backup job do not display in the Storage Infrastructure screen unless they are manually created. I tried rescanning several times on several Luns.

It's by design - Veeam create temporary snapshot only for backup job and delete them after that

[dustinn3]

Ok, I just assumed that any snapshot currently created on the VNX by VEEAM would show up there during the backup operation. I realize it deletes the snapshot after the job completes so it wouldn't be shown, but I assumed that rescanning the LUN while a snapshot exists would show the snapshot without having to go to my VNX to see it.

[foggy]

Temporary snapshots created during backup, restore, and rescan operations have the 'Veeam_' prefix and are not displayed in Veeam B&R console (though you can see them in the storage management console). This is actually mentioned in the release notes:

Storage snapshots and volumes with name starting with “Veeam_” are automatically excluded from processing.

[dustinn3]

I think we finally found our issues and have them fixed for the most part, none of which was really caused by Veeam V9.

The issue seems to have been entirely performance related to my VNX and VMware. It seems that implementing the storage snapshots just exasperated already existing issues and caused large latency spikes which caused our issues with both VMware and Veeam. I enabled storage IO control in Veeam, which fixed our VMware issues, but it took my transfer speeds down to about 30MB/s per job and about 6-7 MB/s on a VM max.

Primarily we fixed an ISCSI issue on all of our hosts and are re-configuring our SAN Pools for better performance. Fixing the ISCSI issue improved my backup performance to about 120 MB's. I re-enabled storage integrated snapshots last night and and enabled per-vm backup files and one job was running at 600 MB/s with no performance issues.

I appreciate the Veeam team jumping right on my issue and escalating it to make sure it wasn't a product defect.

[Gostev]

Fantastic! Thanks very much for sharing the root cause, and especially for including some real-world performance numbers - we are often asked about those by the users. I assume you have 10 Gb Ethernet for SAN fabric?

[dustinn3]

Yes, I'm running 20 GB to each host in my Cisco UCS 5108 with B200M4 Blades to my FI6296's, then to a pair of Nexus 5548UP's with 4 10 GB Twinax cables to my VNX 5300. I also have my DD2500 connected to my Nexus 5548's with 2 10 GB Twinax cables and I'm using DDBoost. Looking at my proxies during a backup I'm seeing between 2-3 Gbps sustained read rates on each.