Hyper-V Cluster 2016 HPE 2052 SAN

[HSC TSA]

Hi,

We have Hyper-V Failover Cluster 2016 that contains 4 nodes, and HPE MSA 2052 SAN storage connected to it.
HPE MSA 2052 SAN storage is 20TB total (8 TB for Cluster Disk CSV, 12 TB for Backup Storage).
The Backup Storage (12 TB) is connected to Node-1 in the cluster.

I have installed Veeam B&R 9.5U3 on the Node-1 of the cluster nodes. Now upon performing Veeam backup, I noticed heavy traffic on the network that reached over 300 Mbps during the backup process period, although both of LUN storages are connected to the same Node-1!

The Summary showed the following information:
Duration: 07:14:19
Processing rate: 111 MB/s
Bottleneck: Source

Processed: 3.0 TB (100%)
Read: 2.7 TB
Transferred: 1.2 TB (2.3x)

Sucess: 34
Warnings: 0
Errors: 0

My question: Is this backup time frame for this size of data acceptable? And why does Veeam use the Network during backup process?


TIA

[Vitaliy S.]

Hello,

Do you see traffic coming from the backup proxy to the repository or from source hosts to the proxy? Based on what you've said above you're using on-host proxy with software VSS provider, is it correct?

Thank you!

[HSC TSA]

Hi,

There is no traffic coming (Receive), only outgoing (Send) traffic!
Veeam B&R installed on Node-1 server, witch is the on-host proxy and repository as well as the Cluster CSV Disk.
Yes, we're using on-host proxy with software VSS provider.

TIA

[Vitaliy S.]

How did you connect the backup target to the proxy? Data flow in your case would be the following: Source -> On-host proxy -> the way how you mounted SAN volume to the proxy -> SAN. So in any case data traffic would "leave" SAN device and then would be written via proxy server back to the storage.

[HSC TSA]

Node-1 server is the proxy.
Both the backup source LUN-0 (C:\ClusterStorage\Volume-1\Hyper-V) and destination LUN-1 (F:\Backup) are mapped to Node-1 server.

There is no network traffic in the data flow that you mentioned! In my case there is!

TIA

[Vitaliy S.]

How did you mount LUN 1 (F:\Backup) to the proxy? Can you please elaborate on this?

[HSC TSA]

I think I'm miss understanding the terminology of veeam proxy server!
Actually, I set the backup job "Storage" tab, "Backup proxy:" On-host backup. And as I mentioned LUN 0 & LUN 1 are mapped (mounted) to the same Failover Cluster node server where Veeam B&R software is installed!

TIA

[foggy]

In your setup, proxy is the node itself, so data is being retrieved from the storage by the proxy component running on the node and then sent to the storage to be stored in the backup.

[Vitaliy S.]

Thanks Foggy. My question was basically about how do you write data from Windows host to the F:\ drive? Do you also see the same network traffic if you copy any file from Windows server to the F drive manually?

[HSC TSA]

In your setup, proxy is the node itself, so data is being retrieved from the storage by the proxy component running on the node and then sent to the storage to be stored in the backup.

Perfect, now what about the or why is the network traffic? and why more the 7 hours to backup 3 TB?

TIA

[HSC TSA]

Thanks Foggy. My question was basically about how do you write data from Windows host to the F:\ drive? Do you also see the same network traffic if you copy any file from Windows server to the F drive manually?

No at all!

[HSC TSA]

I found this weried thing...!!!
VM #21: ACTION: Hard disk 2 (3.0 TB) 369.2 GB read at 23 MB/s [CBT] DURATION: 04:32:39
VM #32: ACTION: Hard disk 1 (1024.0 GB) 641.8 GB read at 826 MB/s [CBT] DURATION: 13:24

This is the backup time of the biggest VHDX!

Why the hug deference between the two in reading speed?!!

TIA

[PTide]

Hi,

Given that you use on-host proxy and direct connection (is it FC/iSCSI by the way?) from your proxy-repo-vbr host to the storage, there should be no traffic, indeed.

First of all, it would be great if you could clarify where exactly you observe the traffic, between which points?
I get it as the network interface of Node-1 shows network spikes, however it is not clear yet what is the destination of that traffic (you've mentioned that there is no (Receive), only outgoing (Send).

From the top of my head, one method of figuring that out would be to create a test backup job that will start outside of any other backup activity. The backup job should include that one VM that has the biggest VHDX (VM #21).
While the job is running, please check network activity on the other hosts of the cluster.

The other method would be to sniff the outgoing traffic on Node-1 with some tool like Wireshark.

Looking forward to hear from you as without that info we can hardly tell you what could be possibly going on.

Thank you

[HSC TSA]

Our Microsoft Hyper-V Failover Cluster 2016 Build from the following:
Four HPE Proliant DL560 Gen 10 (Two Intel Xeon Platinum CPU 2.70GHz (24 Core) - 256 GB RAM) Servers. (each server has four 1-Gb NICs each - 1 for heartbeat (on 100-Mb CISCO Switch), 1 for Hyper-V Host, 2 teamed for guest VMs)
Two HPE SAN Switches (24 16 Gb FC Ports).
One HPE MSA 2052 SAN Storage 20 TB (52 400 GB SSD - 200,000 IOPS, 512 Snapshots).
One HP MSA P2000 G3 FC 12 TB (very old).

SAN-2052 has two LUNs (8 TB & 12 TB).
LUN-0 (8 TB) is mapped to all four servers, and been reserved (in Disk Management) by the Failover Cluster, and reintroduced as the path "C:\ClusterStorage\Volume1\".
LUN-1 (12 TB) is mapped to all four servers, Online on server Node-1 as F:\ drive and Off line on the other three servers.

SAN-P2000 has two LUNs (8 TB & 4 TB - cannot have one LUN of 12 TB because of the lack of virtual volume technology!)
LUN-0 (8 TB) is mapped to Node-1 server only, Online as G: drive.
LUN-0 (4 TB) is mapped to Node-1 server only, Online as H: drive.

Veeam Backup & Replication 9.5 update 3, installed on Node-1 server. After finishing installation wizard, Veeam B&R by default added server Node-1 as "Backup server" and drive F: as "Backup Repository".
(we know it is not recommended by Veeam, but we distributed the VMs (Preferred Owner with Allow failback Immediately), the first four low CPU access on Node-1 and all other VMs on Node-2 & 3 & 4, the peruse was to get the best performance and efficiency in backup and restore time!) Not sure if this is the best practice for this environment!

I added Microsoft Hyper-V Failover Cluster 2016 as a Hyper-v cluster server, and Veeam picked up the four nodes servers automatically.
I added a Scale-out Repository, extents drive G: and drive H: to have one 12 TB repository!
I added one offsite server (has 2.62 TB HDD), as a Windows servers and as well as a repository.

Created a backup job-1 (local SAN-2052), selecting the Failover Cluster to be Backed up and the default Backup Repository (drive F:) as backup repository with "On-host backup" as a backup proxy.
Created a backup job-2 (old SAN-P200), selecting the Failover Cluster to be Backed up and the Scale-out Repository as backup repository with "On-host backup" as a backup proxy.
Created a backup job-3 for offsite backup, setting up the backup proxy as "On-host backup".

When the backup job (1 or 2 or 3) runs as a full backup it takes more than 7 hours (in case of offsite almost 47 hours), and a heavy network traffic on Node-1 Hyper-V Host NIC (more than 300 Mbps! & in case of offsite +-100 Mbps!) during the backup time period! After using "Wireshark" (network monitor app, suggested by PTide) it turns out that Node-2 & 3 & 4 are sending all their hosted VMs' data to Node-1 (On-host proxy) through Hyper-V Host (Management) NIC!


Q: In backup jobs 1 & 2 & 3 why Veeam B&R not reading all VMs' data from Node-1 path "C:\ClusterStorage\Volume1\", in other words why Veeam B&R dealing with Cluster nodes as a stand-alone Hyper-V servers? Why Veeam is not benefitting from Microsoft Hyper-V Failover Cluster and HPE NAS Storage VSS snapshot feature?

TIA

[HSC TSA]

BTW, the biggest VHDX (VM #32) hosted by Node-1 and the nest biggest VHDX (VM #21) hosted by Node-2. Which make sense!

TIA

[PTide]

Right. It totally slipped from my mind that Node-1 is the only node that holds a repository role. Since all other nodes cannot access LUN-1 directly, they have to send backup data via Node-1 which serves as a gateway since LUN-1 is mounted there. Even if you assign a repository role to each host you will need to split the list of VMs between the jobs correspondingly.

Thanks

[HSC TSA]

Thanx PTide,

Would you please advise me the best practice for our system. If there is one other than split jobs!

TIA

[HSC TSA]

Even if you assign a repository role to each host you will need to split the list of VMs between the jobs correspondingly.

I mounted (Mapped) LUN-1 on each host and assign a repository role to each host and created backup jobs to each host and split the list of VMs between the jobs correspondingly!

Now, the first run of each jobs (in row) was perfect and run under 10 minutes each. But, the second run was disastrous! All jobs "Failed" with almost different error each time, one of them:
"7/17/2018 4:49:13 AM :: Processing <VM> Error: The file or directory is corrupted and unreadable.
Failed to open file [<Path>.9e735701-07d9-4784-8df8-681f42d60c1D2018-07-16T073830.vbk] in readonly mode.
Failed to download disk.
Shared memory connection was closed.
Failed to upload disk.
Agent failed to process method {DataTransfer.SyncDisk}."

I think setting the jobs the way you suggested will overrid each other! And maybe, the only way Veeam B&R deal with MS Hyper-V Failover Cluster is what I did in the first place and complain about it! and we have to live with it!! Please, correct me if I'm wrong!

TIA

[PTide]

I mounted (Mapped) LUN-1 on each host <..>

Which should not be done unless you have a cluster file system on LUN-1. My proposal assumed that there would be a separate LUN for each host acting as a repo, as you cannot mount a single LUN with a non-cluster file system on it to multiple hosts. You should either have a separate LUN for each host, or revert back to your initial design.

Thanks

[HSC TSA]

My proposal assumed...

!!!

One more thing, when creating backup job on VM level! The user will have to add every and each newly created VM to its corresponding job!!

TIA

[PTide]

In order to make all backup data to bypass the network, you need to have a dedicated physical machine serving as an off-host proxy. That machine must meet the requirements listed under the "Requirements for Off-Host Backup Proxy and SMB Shared Storage" section of this page.

Thanks

[HSC TSA]

"Requirements for Off-Host Backup Proxy and SMB Shared Storage" section of this page.

I'll check if this is feasible in our environment and test it and get back with the results!

TIA

[HSC TSA]

"Requirements for Off-Host Backup Proxy and SMB Shared Storage" section of this page.Thanks

Q. We'r using CSV (SAN) Storage! So, which one is suitable for our environment to setup "Off-Host Backup Proxy and CSV (SAN) Storage" OR "Off-Host Backup Proxy and SMB Shared Storage"?

TIA

[PTide]

Right, you should follow the "Off-Host Backup Proxy and CSV (SAN) Storage" section, sorry for the typo.

Thanks