veeam configuration for lefthand multi-site cluster question

[cmdrriker]

We just purchased an HP StoreVirtual 4330 a/k/a Lefthand san. It is configured as a six node, multi site cluster. There are two sites, with 3 nodes at each site. The san cluster is configured
with network raid 10. It is our understanding that this configuration can survive a site failure, as all of the data from the nodes at one site are replicated to the nodes at the other site. There
is one vmware host (esxi 5.1) at each site connected to the san via iscsi.

We previously had an iscsi san at each site and replicated the virtual machines from one site to the other. The lefthand solution would seem to obviate the need to cross replicate between sites (if it works as advertised), so it would seem then the primary purpose of replication would be to protect against os corruption only, as the Lefthand network raid will protect against both disk/san and site failures.

In light of this, we were thinking of creating six luns of various sizes, with each lun sized to accommodate both virtual machines and their respective Veeam replicas. Is this a good idea? Should
the replicas be on different luns from the virtual machines they are mirrors of? Will replicating to the same lun that the vm resides on incur a performance/speed hit? Would anyone configure this
differently?

Please ask questions if you need more information to reply.

[foggy]

Will, as I see it, if the goal of Veeam replication is to protect from corruptions inside guest OS only, then you can replicate to the same datastore. This however will put double impact on the storage itself, of course.

[dellock6]

Uhm, Alexander is right on thinking about the double I/O on storage, and I would also add the doubled network traffic to replicate all the storage nodes, since you are replicating both the live VMs and their Replicas. It's not bad per se, but something I would check with few tests to see if the storage is not overloaded.
Otherwise, supposing the replica is reliable enouth to the point replicas are not so much needed, I would go with backups and replicate them between sites, for example using the upcoming Veeam BackupCopy feature.

Luca.

[cmdrriker]

Okay, Luca and Alexander, this question is for both of you, (and everyone else) so would you then create luns dedicated to replication, or would you just insure that replications did not go to the same lun as the vm that was being replicated? For instance, if we have production vm lun A and production vm lun B, then we would replicate lun A vm's to lun B and lun B vm's to lun A?

But then if there are concurrent replication jobs occurring on both luns, wouldn't the performance hit be the same as simply replicating to the same lun? Would dedicated replication luns make more sense from a performance standpoint then?

On the other hand no matter what we do, all data will always be synchronized to all nodes. So maybe then with the Lefthand solution there is no way to architect around i/o
impact from replication? Or is there an option I am overlooking?

[dellock6]

Uhm, wait, on LeftHand every LUN is replicated on every node of the cluster, even if at a given time only one node is the "gateway" to access the lun itself. This is way we said that there would be an additional impact on the storage I/O.
And yes, the design would be to have different LUNs created into the management group, at least one for production and one for replica.

But, since LeftHand is supported by Veeam since 6.5, I'm thinking about another option: why don't you use storage snapshots? They are replicated too among the nodes, are thin provisioned, and are hardware-based so they do not even impact vSphere. You can configure for example a snpshot to be created every 15 minutes, and keep only the last 8 (so you can go back in time 2 hours).

Luca.

[cmdrriker]

Luca,

I haven't considered san snapshots because I'm not familiar with them and I'm not sure there is time to test before we put the san into production. I just found your granular recovery from san snapshots article and am starting to read it, but if you could list a quick summary of the pros and cons of san snapshots vs. Veeam replicas I would appreciate it.

[dellock6]

Uhm, let's see: the best pro of san-based snapshots is independence from the "infamous" VMware snapshots problems, so you can have quicker snap operations, thus having a more frequent snap if needed. On the other side, san snapshots works per LUN, so each snapshots takes a copy of all the VM inside that lun. Instead, with Veeam you select individual VMs and replicate only them. Also, VSS support is better via Veeam, so if you need a consistent point in time is better to use Veeam.

Maybe you can place for example all VMs that you can replicate with only crash-consistent replica in a dedicated LUN, and replicate them via storage.

Luca.

[cmdrriker]

Luca,

Thank you for the succinct answer, the vss support pretty much makes the decision for us, I think we will stay with Veeam replication.

[yizhar]

Hi.
I think that you are overlooking and misunderstanding some hp lefthand concepts, and your plans seems bad to me.
HP Lefthand is great product, but needs careful planning and experience.
On the other hand, the whole system (all nodes) can go down by human error or misconfiguration and you should have protection against such problems as well.

Main problem in your design:
* You are planning to put both production and backup/replica on same system.
This is bad.
It is bad for performance - backup/replication activity will noticeably affect production as well.
It is bad for recovery - It won't help you if the whole system (management group) is down.
It is also bad from the finance point of view - storing backup/replicas on expensive production system.

So my answer -
* Do not put replicas on the Lefthand at all.
* Or: take out 1 or 2 P4330 systems out of the production management group, and use them as backup/replica in a different management group (Ask your HP experts to explain further).
So in that case - 4 P4330 systems for production data only, in multisite configuration (2 systems at each site)
+ 2 other systems for backup/replication.
This will of course provide less capacity then 6 systems - so you should check and plan carefully.


Some other things you should check or consider, which are more related to storage, networking, and VMware design. Not directly related to Veeam backup:

Check out the following documents:

HP LeftHand Storage Multi-Site Configuration Guide
http://h10032.www1.hp.com/ctg/Manual/c03562494.pdf

VMware KB Implementing vSphere Metro Storage Cluster using HP LeftHand Multi-Site
http://kb.vmware.com/selfservice/micros ... Id=2020097

Network link(s) between sites - can you provide more details?
A lefthand multisite design must use fast links with low latency, because it uses network raid (similar to synchronous replication).
If each site is simply different floor in the same building with 1gbps links, this is OK. If the sites are connected by WAN or other high latency link, no good.
The maximum round trip latency on the storage network between sites should not exceed 2 milliseconds (ms) RTT.

Number of links between sites:
How many links do you have?
Did you plan for all needed networks - vmotion, management, iscsi + MPIO, etc...

Raid level:
When configuring lefthand multisite with 2 sites, you should choose between RAID10 (2 copies, one at each site) or RAID10+2 (4 copies, 2 at each site). Each has pros and cons, so make sure to ask your HP experts in advanced about the differences.
You can choose network RAID level per volume, so for example accounting DB or ERP system can be on RAID10+2 volume, while other might be on network RAID10 .

Proportion of esxi hosts:
To me, it looks a bit strange that you plan to have 6 lefthand systems, but only 1 esxi host per site. I suggest considering 3-4 hosts.

FOM - Fail Over Manager:
Did you plan for FOM (read above document)?

I know that I'm quite off topic here, so going back to Veeam design. The bottom line is:

Do not use the same production system to store backups/replica. Use different system (your old SAN, server with local disks, or a dedicated independent SAN - in different management group).

Bye
Yizhar

[cmdrriker]

Hi Yizhar

Thanks for the detailed response.


re: replicating to the same system - we only replicate once a day, after hours, so no one will be using the virtual machines
during replication to notice any degredation. In addition to replication we will be running Veeam backups to a different storage
system.


The two sites are across a small parking lot from each other, a distance short enough to be within the limits of the multi mode
fiber that connects the two buildings, so effectively it's the same as if the sites were in different floors of the same building.
The current link speed is 1 gig, and we will be adding another strand in order to bond two - one gig links before going into
production.

The configured lefthand raid level is raid 10.

We have set up a failover manager at a third site.

The number or virtual machines we are running doesn't require more then two hosts, both of which have sufficient capacity to run
all of the virtual machines in the event one host should fail.

A VAR had special pricing on a six node package that was too good to overlook, which is how we ended up with a six node lefthand
cluster - it otherwise would have been out of our price range. The lefthand network raid solution seemed to offer a simpler way to
protect against disk or san failure then the cross site san replication system we are now migrating from. (I.E vm's in site one
were replicated to site two, and site two vm's to site one - care had to be taken when scheduling jobs to insure vm's were
replicated to the opposite san and host - the network raid seems to eliminate the need to do this).

Out of curiousity, how is your organization's metro lefthand cluster and veeam jobs configured?

[yizhar]

re: replicating to the same system - we only replicate once a day, after hours, so no one will be using the virtual machines
during replication to notice any degredation. In addition to replication we will be running Veeam backups to a different storage
system.

I still think that this is bad design for your case - replicating to the same system.
The possible problems are not only related to degration during replication and snapshot commit activity.
Main issue is - if the whole system is down, replicas aren't available.
Other issue is waste of production resources (storage space, performance, but also the network link between sites for example, and more).

If you want to use part of the total lefthand capacity for replicas, I suggest the following:

1. Use only 4 LH boxes for the multisite cluster, instead of 6.

2. Configure the other 2 boxes as an independent single site management group.

3. Replicate VMs from the multisite production group to the backup/replica group using either of the following:
3a. Veeam replication.
3b. Lefthand SAN replication (remote snapshot feature).

This will be a more robust and stable design.

Yizhar

[yizhar]

The two sites are across a small parking lot from each other, a distance short enough to be within the limits of the multi mode
fiber that connects the two buildings, so effectively it's the same as if the sites were in different floors of the same building.
The current link speed is 1 gig, and we will be adding another strand in order to bond two - one gig links before going into
production.

The configured lefthand raid level is raid 10.

We have set up a failover manager at a third site.

Hi.
You should plan network connectivity very carefully, before implementing multisite configuration.
Some catches to avoid:
If you have a single device (network link, switch, cable, or any other SPOF) connecting from any site to the other, this might bring down the whole site and maybe also the whole system.
For instance, please check how is the FOM going to be connected to other sites?
If there is a single "networking room" that all sites connect to, or if the FOM is connected to site A and from there to site B,
networking/power problem at site A might take down the whole cluster.

You should also understand all the traffic that would be going over intersite links - this will not be only storage replication, but much more in your design.
clients accessing server VMs.
VMs in host A to VMs in host B.
VMware hosts management traffic.
Backups, replication and restores traffic.
vmotion.
lefthand traffic between systems during normal operations, and when something fails.

You should plan and understand network activity during partial downtime of either device - weather planned maintenance, power, networking problem, hardware malfunction or human error.

An alternative approach is a non-balanced design, such as:

Define one site as main or production, and the other site as "active standby".
Lets call them site A (main) and site B (standby).
2 lefthand systems at each site (as mentioned above).
2 VMware esxi hosts at site A.
The above hosts will benefit with multiple (and separated) network links for iscsi, management, production, vmotion networks.
1 VMware esxi host as site B.
(total 3 esxi hosts).
All (or most) production VMs running on the 2 hosts in site A.
Site B as mentioned is used as "hot active standby" which means that it can take over in 0 downtime, but only when needed.

Network links between sites are used primarly for lefthand san iscsi network raid between sites (btw - please note that every write operation is saved to both sites so this link will be active all the time).

In above situation, some operations like patching esxi host in site A, will still be handled "locally" inside the site, all VMs vmotioned to neighbor host in the same site over several dedicated links, most activity remains in the site.

I must mention that I have no field experience at all with Lefthand multisite, and what I'm writing here is based on my experience with smaller single site setups, and theoretical knowledge + common sense.
So I suggest that you further consult with other HP and VMware experts + yourself after reading and understanding Lefthand operation.

Again - HP Lefthand with network raid is a great system with many advantages, but is more complex then other storage systems especially with multisite, and needs careful planning and implementation.
I'm not saying that your design is wrong - just trying to pinpoint potential problems and possible improvements.

And again - sorry if we're going too off topic here, but I think that it is OK as we are here to help each other, and also a major step for good backup and recovery system is robust infrastructure which we discuss...

Yizhar

[yizhar]

protect against disk or san failure then the cross site san replication system we are now migrating from. (I.E vm's in site one
were replicated to site two, and site two vm's to site one - care had to be taken when scheduling jobs to insure vm's were
replicated to the opposite san and host - the network raid seems to eliminate the need to do this).

This is correct - network raid can eliminate this and provide 0 RPO + 0 RTO in case of several potential san problems.
But - it adds complexity - for example:
In your current design, most network connections and activity are internal to each site, and inter-site link is used only for replication. For example Host in site A communicates only with SAN in site A currently (am I correct?).
In the new multisite design - inter-site links have much more responsibilities that should be considered and taken care for.

Out of curiousity, how is your organization's metro lefthand cluster and veeam jobs configured?

None...
I have several clients with minimal LH implementation (mostly 2 devices each + FOM), all of them in simple single site.
I have 1 client were we also use LH remote snapshot (async replication over slow WAN) to remote site.
All my previous tips are based on theoretical knowledge about LH, and (hopefully) common sense.

Yizhar