Concurrent Triple Backup Questions

[ctchang]

For example, we have three kind of backup running at 6AM daily AT THE SAME TIME.

VMs are all on Equallogic SAN VMFS volume.

1. Acronis True Image backup inside each VM. (ie, File Level, backup time is 5-10 mins per VM)
2. Veeam Backup SAN (ie, Block Level, backup time is 1-5 mins per VM)
3. Equallogic Snapshot (ie, Block Level, backup time is 1-5 seconds for the whole array)

Will this actually create any problem? I mean LOCKING problem due to concurrent access to the same volume?

But Beginning with Equallogic version 5.0, the PS Series Array Firmware supports VMware vStorage APIs for Array Integration (VAAI) for VMware vSphere 4.1 and later. The following new ESX functions are supported:

• Harddware Assisted Locking – Provides an alternative meanns of protecting VMFS cluster file system metadata, improving the scalability of large ESX environments sharing datastores.

So this shouldn't be a problem any more.

Any hints of recommendation?

[Gostev]

Hello, I don't see any possible issues with this with or without VAAI, since all 3 solutions operate on very different levels. Thanks!

[joergr]

As so many times the answer is: it depends. You can easily go with multiple operations, it depends on your san power. Please get you SAN HQ from DELL-EQL Homepage and check out the latency of your luns during the backup operation. If it stays in range, let´s say it won´t go over 15ms permanently for many minutes you are quite good to go. If it stays above let´s say 30ms for a longer time range, i would suggest to adjust your config.

Best regards,
Joerg

[ctchang]

Thanks, I am going to give the above a try, plus Veeam's Backup to DR site, that's 4 level of protection.

Bt,w, Gostev, aren't 2 and 3 the same? They both work on Block level? My question is if they do, wouldn't they compete with each other at 6AM ?

2. Veeam Backup SAN (ie, Block Level, backup time is 1-5 mins per VM)
3. Equallogic Array Snapshot (ie, Block Level, backup time is 1-5 seconds for the whole array)

[joergr]

Not at all. The EQL Snapshot is done on the EQL Hardware Level. The vmware snapshot is done at vmware level, with vaai hardware assisted but not done on the hardware level. (BUT if it is exactly that what you want DELL provides a cool tool named auto snapshot manager vmware edition, this will allow you to trigger a hardware snapshot exactly the same time a vcneter snapshot is triggered - good for volume shadow copy consistency).

To conclude: The VEEAM Snapshot is done with vCenter Server or ESX/i directly using VAAI or not (dependes on your firmware and esx/i version). Don´t mix around Hardware Snapshots and hardware assisted snapshots, is is NOT the same.

Best regards,
Joerg

[ctchang]

Not at all. The EQL Snapshot is done on the EQL Hardware Level. The vmware snapshot is done at vmware level, with vaai hardware assisted but not done on the hardware level. (BUT if it is exactly that what you want DELL provides a cool tool named auto snapshot manager vmware edition, this will allow you to trigger a hardware snapshot exactly the same time a vcneter snapshot is triggered - good for volume shadow copy consistency).

To conclude: The VEEAM Snapshot is done with vCenter Server or ESX/i directly using VAAI or not (dependes on your firmware and esx/i version). Don´t mix around Hardware Snapshots and hardware assisted snapshots, is is NOT the same.

Best regards,
Joerg

That's very insightful, thanks!

Btw, I was told Veeam backup or snapshot whatever you call it, say using SAN Method, the data still has to go out of EQL box, then Veeam Backup server will do all the de-dupe and compression and then store on local storage on the backup server. I think VAAI doesn't even kick-in in this situation.

Of course, it would be great if Veeam v5 can tell EQL to take a snapshot locally first using VAAI (ie, super fast), then send that Completed snapshot to Veeam Backup server for de-dupe and compression and then finally save to local storage on the backup server. or does it actually work that way already if I am using EQL FW5.0.2+ESX 4.1+EQL MEM Plugin?

Thanks again,
Jack

[Gostev]

Yes, this is how it works today. Any snapshot you create via VMware API (whether with Veeam or vSphere Client) is hardware-assisted, so all snapshot operations should have noticeably better performance.

[joergr]

That is correct, but at a very small time window, VAAI kicks in, when the snaphot is triggered, at this time VAAI kicks in with the locking mechanism. But then again, don´t mix it up with a hardware snapshot, VAAI can NOT trigger SAN-Vendor-specific hardware snapshots.

You can quite do a lot with Equallogic SAN, if you want it. There are many ways allowing you to trigger hardware snapshots if you want them triggered. So i suggest you take a look at

a) Auto Snapshot Manager VMware Edition
b) latest Host Integration Tools (H.I.T.)

best regards,
Joerg

[ctchang]

That is correct, but at a very small time window, VAAI kicks in, when the snaphot is triggered, at this time VAAI kicks in with the locking mechanism. But then again, don´t mix it up with a hardware snapshot, VAAI can NOT trigger SAN-Vendor-specific hardware snapshots.

You can quite do a lot with Equallogic SAN, if you want it. There are many ways allowing you to trigger hardware snapshots if you want them triggered. So i suggest you take a look at

a) Auto Snapshot Manager VMware Edition
b) latest Host Integration Tools (H.I.T.)

best regards,
Joerg

Fantastic confirmation from both of you! Thanks gain!

I feel very happy now as finally VAAI can be used to greatly increase the snapshot performance and shorten the backup window, together with Veeam's vStorage API, I am pretty sure the backup time is going to break the records shortly, I will report later after I installed B&R V5 tomorrow.

One more thing, I did try to use ASM/VE to backup and restore VM once for testing, I forgot if it's with VAAI or not or if the FW is 4.3.7, but it's quite slow and one thing I also don't like is to backup snapshot onto the Same Array!

1. Array space is too expensive to place snapshot on it, at least for VM, I am ok with taking array snapshot for lun/volume only though to avoid array failure kind of disaster.

2. It's not safe to place VM snapshot on the same array, what if the array crashes?

3. As an EQL user, everyone know that "Dirty Bit" problem (ie, once the block is written, there is no way to get it back.) In other words, it's a great waste of deleted/empty space. Not Until EQL releases the Thin/Thick Space Reclaim feature in the coming FW5.x version, I think the technology is still not mature or ready yet to have VM snapshot to be placed on EQL volume. FYI, 3PAR is the only vendor having true Thin Space Reclaim, HDS reclaim is a "fake " one, search google about a 3PAR guy commenting about HDS's similar technology you will see what I mean.

4. Remember EQL use 16MB stripe? It means even there is 1K movement in block, EQL array will take the whole 16MB, so your snapshot is going to grow very very very very huge, what a waste! I really don't understand why EQL designed their stripe size to 16MB instead of say 1MB, is it because 16MB can give you much better performance?

5. Another bad thing is even you buy a PS6500E with lots of cheap space, but you still can't use ASM/VM to backup a VM on PS6000XV volume and place the snapshots on PS6500E, it HAS TO be the same array or pool and stay at PS6000XV, so seem there is no solution.

That's why we finally selected Veeam B&R v5 Enterprise Edition and that's how I arrived here and encounter all the great storage geeks!

[Gostev]

Here is a very good Equallogic analysis, also covers 16MB stripe size issue:
http://www.tuxyturvy.com/blog/index.php ... logic.html

[joergr]

You are welcome.

1) That is a vmware related issue, not EQL related
2) Correct, but then again: Snapshots are NOT made for backup only for helping the backup process.
3,4) Correct, but VEEAM/vmware-triggered snapshots, when doing backups at night, would not really grow that much because the time window for doing the backup is extremely small (when using high speed lan and cbt) - therfor me personally i have no problem with it - and even if it gets a problem: VEEAM has sophisticated mechanisms which allow you to safely break the backup operation if the snapshot grows too huge.
5) Are you talking about vmware based or eql based? If vmware based this is vmware related. EQL based you are right but for that kind of problem replicas are made. Create another EQL Group with your SATA Arrays and use it only as replication target.

best regards
Joerg

[ctchang]

Here is a very good Equallogic analysis, also covers 16MB stripe size issue:
http://www.tuxyturvy.com/blog/index.php ... logic.html

I've seen that before we purchased EQL, so it's ok for us now as we are going to use Veeam Backup anyway. (but would be perfect if EQL reduced the 16MB page size
)
Another good one I found on his latest blog update.
http://www.tuxyturvy.com/blog/index.php ... ation.html

[ctchang]

You are welcome.

5) Are you talking about vmware based or eql based? If vmware based this is vmware related. EQL based you are right but for that kind of problem replicas are made. Create another EQL Group with your SATA Arrays and use it only as replication target.

best regards
Joerg

I was referring to EQL's ASM/VE snapshot? is that EQL or VMware? I think it's EQL based as it placed a snap on the volume, also a snap on the VMFS, so is it atually both? (ie, vmware based and eql based ?)

Btw, Replication between groups works fine as the page size is 50KB or something, again, I don't understand why can't EQL make normal snapshot page size to be 50KB, why as hugh as 16MB? There must be a reason I think.

Oh...has anyone compared ASM/VM Backup vs Veeam Backup yet?

Let me start some points.
- ASM/VM Backup cannot do file restore, it's a whole VM
- ASM/VM Backup must store snapshots on the same SAN disk, can't backup to external disks like Veeam does (aren't there any jailbreak problem for EQL? )
- ASM/VM Backup has way less features

In fact I couldn't think of any pros of ASM/VM Backup, except it's the FASTEST may be?

Anyone care to commment?

[Gostev]

Well, to start SAN snapshot cannot really be considered real backup at all... you need to take the data off the storage to consider it real backup. Because you know, SAN corruption will take both your data, and your snapshots. I guess I could mention yet another post on the very same blog

Also, there are many other reasons why you really want to additionally sync your backups offsite... people like to talk about planes crashing into buildings but hey, these kind of things are much more common (if you get bored initially, scroll to about 5:00). Also, 7:00 and above, do you recognize your server room hehe? Watch those SAN snapshots snorkeling!

[joergr]

Holy shit Anton. This video might cause nightmares

[ctchang]

Update: Official Answer from Equallogic

Good morning,

So, the question is does VMware’s ESX v4.1 VAAI API allow you to have one huge volume vs. the standard recommendation for more smaller volumes while still maintaining the same performance?

The answer is NO.

Reason: The same reasons that made it a good idea before, still remain. You are still bound by how SCSI works. Each volume has a negotiated command tag queue depth (CTQ). VAAI does nothing to mitigate this. Also, until every ESX server accessing that mega volume is upgraded to ESX v4.1, SCSI reservations will still be in effect. So periodically, one node will lock that one volume and ALL other nodes will have to wait their turn. Multiple volumes also allows you to be more flexible with our storage tiering capabilities. VMFS volumes, RDMs and storage direct volumes can be moved to the most appropriate RAID member.

i.e. you could storage pools with SAS, SATA or SSD drives, then place the volumes in their appropriate pool based on I/O requirements for that VM.



So do you mean if we are running ESX version 4.1 on all ESX hosts, then we can safely to use one big volume instead of several smaller ones from now on?

Re: 4.1. No. The same overall issue remains. When all ESX servers accessing a volume are at 4.1, then one previous bottleneck of SCSI reservation and only that issue is removed. All the other issues I mentioned still remain. Running one mega volume will not produce the best performance and long term will be the least flexible option possible. It would similar in concept to taking an eight lane highway down to one lane.