Don't Store Backups on VMFS...But why not?

[taylorbox]

Environment: Veeam B&R 6 (w/patch 3) virtual machine, VMware vSphere 5 hosts using direct-attached storage SAN.

Veeam support folks have repeatedly said they strongly recommend NOT using a VMFS volume for storing the backup data on the Veeam repository virtual server, because it's "bad practice," it's the "last place you want your backup data to," etc.

I would like a more definitive explanation on this.

I've read all of these posts:

http://forums.veeam.com/viewtopic.php?f ... bad#p22603
http://forums.veeam.com/viewtopic.php?f ... ory#p41776
http://forums.veeam.com/viewtopic.php?f ... ory#p41089
http://forums.veeam.com/viewtopic.php?f ... ory#p45876

But I'm just not convinced.

The arguments against this seem to be the following:

1) Corruption of the VMFS volume would result in the backups being unusable.
But NTFS can become corrupted just like VMFS can become corrupted. So if the backups are stored on an NTFS volume that's mounted as an iSCSI LUN, that volume can become corrupted just like any volume can become corrupted. (Of course the VMFS volume for the backups should not use the same RAID array as the VMFS volume for the protected VM's).

2) Storing backups directly on an iSCSI or FC LUN on the SAN provides better performance than having the backups stored in a VMDK running on VMFS.
Okay, I understand that removing the VMFS layer from the storage stack and going directly to the SAN is more efficient. Yet, isn't the more important issue here the underlying disk drive performance (IOPS)? Consider the following setups:

Setup A: Veeam B&R VM (proxy & repository) with backups stored in a VMDK on VMFS, and that VMFS is using six SAS 15K RPM drives in RAID 10.
Setup B: Veeam B&R VM (proxy & repository) with backups stored on an iSCSI LUN, and that LUN is using two 2TB SATA 7200 RPM drives in RAID 1.

Would not Setup A have much better performance than setup B due to the superior IOPS of Setup A's drives, even though setup B is using a LUN and not VMFS? So my argument is that the performance of underlying drive array is the bigger factor than whether or not the backup data is living on VMFS or an iSCSI LUN. Am I wrong? (Obviously Setup A is more expensive than Setup B).

3) If the ESXi hosts die, then you won't have access to the backup data.
Well, if one ESXi host dies, then of course the remaining hosts should be able to run the VM's. If some major bug in vSphere somehow killed all hosts simultaneously, then reinstalling ESXi is actually pretty quick and easy, and thus access to the VM's and backup data on VMFS would be restored (since this data is on the SAN and not the ESXi hosts).

So I just don't get the stay-away-from-storing-backup-data-on-VMFS stance. Why is this such a terrible idea? Does this impact the performance of Instant Recovery of VM's? Please advise.

-Taylorbox

[dellock6]

You are talking about backups, not replicas, so your statement:

Well, if one ESXi host dies, then of course the remaining hosts should be able to run the VM's

is wrong.
If you do backups in a vmfs datastore, you will have in it a bunch of files .vbk, .vib and .vrb, but none of the ESXi reading from that datastore will know what to do with those files. You will always need a Veeam Backup Server to read those files, and since Veeam is a windows based software, the best choice you have to save files is a NTFS drive. Choosing between a local disk, a lun mounted on the Veeam backup, or maybe a SMB2 nas server (please, do not call it CIFS anymore...), it a minor choice compared to using NTFS.

My 2 cents.

Luca.

[Gostev]

1) How many tools you know which can do as simple as undelete, or actually recover data from corrupted NTFS partition? Now, ask yourself the same question about VMFS. No comparison there!

2) Well, we give generic recommendation for comparing apples to apples. Of course, any generic recommendation can be proved wrong in specially crafted scenario, such as above

3) Yes, but this scenario requires SAN. If you note from the above topics, most people trying to store data on VMFS on local disks simply because do not have SAN, and so cannot backup to iSCSI LUN (which is the primary reason why they are trying to resort to VMFS in the first place). Moreover, they typically only have only 1 host - so as soon as it is dies, you indeed cannot access your backup data.

Also, another 3 from the top of my head, just for a start:

4) 2 TB size limit for VMDK, obviously too little for a backup repository, big showstopper for most customers.

5) Imagine your virtual infrastructure is down, all hosts won't start after latest patch (happened before!). How long will it take for you to get the data out from VMFS? While with NTFS, it takes less than 1 min to reconnect iSCSI LUN to any Windows notebook, and get the data out. Or even simply run a part of your production in VMware Workstation to start with (one of our customers actually had to do this!)

6) Makes tape backup impossible in many cases (tape libraries typically require physical computer with backup files LUN mounted). Another big showstopper.

May be more, I really never needed to think that deep - just the size limitation and recoverability concerns alone were enough for me. But, if all of that does not matter much for you, then just do what you think is best. Again, we only provide generic, universal recommendations - and every rule has exceptions.

[taylorbox]

Luca:

In a cluster of multiple ESXi hosts, should one host fail, vSphere HA will restart on the remaining hosts the VM's that had been running on the failed host. That's the simple point I was making in this sentence: "If one ESXi host dies, then of course the remaining hosts should be able to run the VM's."

If you were actually referring to everything I said in point #3 (not just the first sentence), then, just to be sure we're on the same page, in our environment we have Veeam B&R running as a VMware VM using VMDK's, and in turn the VMDK's are of course living on a VMFS datastore volume (on our SAN). So the VMFS datastore does not (directly) have the Veeam .vbk, .vib, and .vrb files. The VMFS datastore has the typical VM files for the Veeam VM: .vmx, .vmxf, .vmdk, etc. The Veeam backup files are inside the VMDK that's used as a virtual hard drive by the Veeam VM. That VMDK drive is of course formatted as an NTFS volume in the Veeam server.

Actually, I don't think it's even possible for a Windows server running as a VM to write directly to a VMFS datastore. Windows doesn't know anything about VMFS. Windows VM's can use either VMDK files, or raw device mappings, or iSCSI LUN's, but in all cases those volumes are formatted as NTFS.

Also, I stated that "If some major bug in vSphere somehow killed all hosts simultaneously, then reinstalling ESXi is actually pretty quick and easy, and thus access to the VM's and backup data on VMFS would be restored (since this data is on the SAN and not the ESXi hosts)." Losing all ESXi hosts does not necessarily mean losing all VM's and data on the VMFS datastores. If the SAN is fine, and the SAN is where the VM's and data live in the VMFS datastores, then the reinstalled ESXi hosts would be able to connect to those datastores and run their VM's.

-Taylorbox

[taylorbox]

Gostev:

1) Like I stated in my response to Luca, the Veeam B&R VM is storing its backup data on an NTFS-formatted volume, which is on a VMDK drive in the VM. Thus, I could make use of the NTFS recovery tools. In the rare event that the underlying VMDK file or its VMFS datastore were to become corrupted, well, that's why we make two copies of our backup data and keep them in remote sites. "Corruption" could happen at any level in the computing stack (files, file system, hard drive cache, hard drive spindles, RAID controller, etc.), so the most comprehensive protection against corruption is a separate copy of the backup data.

2) My point is that whatever overhead in disk I/O that comes with using a VMFS volume in the storage stack is negligible compared to the performance of the hard drives in the RAID array. If I'm concerned about the performance of my Veeam backup volume, I should focus on the hard drives and RAID array configuration much more so than whether the volume is in a VMDK or on an iSCSI LUN.

3) Okay. We have a SAN and multiple ESXi hosts in our environment.

4) The 2TB VMDK limit is valid, but it really depends on the size of the environment. Two or three VMDK's could provide sufficient storage for many shops.

5) If our virtual infrastructure goes down, I see no need to try to "get the data out from VMFS." A better approach is to restore access to the VMFS datastores. I can rebuild two or three ESXi servers (using a stable version of ESXi, of course) in under 30 minutes, and thus restore access to the VMFS datastore. Thus, no need to get the data out. And just imagine how long it would take to get the data out if that's what I had to do!

6) Tape backups are not impossible when Veeam B&R is running as a VM with the backup data on VMDK's. In fact, we're actually doing this today. We have a physical backup server with a tape drive. That server's backup software backs up (via an agent) Veeam's data to tape. This may not be a very efficient setup, but it's by no mean impossible.

So, honestly, I'd really like to hear some more objections to using VMDK's for storing Veeam backups, because so far to me only point #4 provides a reasonable argument against this in our environment. (And the 2TB VMDK limit may perhaps be increased in the next version of vSphere).

NOTE: I appreciate your responses on this topic...This is helpful for me to think through all of these points.

-Taylorbox

[chrisdearden]

the way I view it is that if your backups were on a directly NTFS formatted LUN - you could ( at a pinch ) present that lun to another physical server , even in the event of catastrophic failure of the Virtual Environment , you could still get to your backups. If your backup volume was inside a VMDK file then you'd have an extra step ( of building a new host ) to try and get at those backups.

With all of these things I dont see the rules as absolute - but backups are all about risk management. If you accept the risk that you would be encapsulating your backup files in a format that would require a couple of extra steps to get at your backups , then go right ahead and put your backups inside a VMFS datastore.

[Gostev]

With all of these things I dont see the rules as absolute - but backups are all about risk management. If you accept the risk that you would be encapsulating your backup files in a format that would require a couple of extra steps to get at your backups , then go right ahead and put your backups inside a VMFS datastore.

Well said

[dellock6]

Taylorbox,
you asked for objections, can it be another one that, using NTFS inside a VMDK, you are saving VMs inside the technology whose faults you are trying to proctect from?

[Gostev]

That is also a very good point, Luca

[taylorbox]

Taylorbox,
you asked for objections, can it be another one that, using NTFS inside a VMDK, you are saving VMs inside the technology whose faults you are trying to proctect from?

Luca,

If I follow your reasoning here, then we should likewise avoid saving backups of Windows servers onto NTFS volumes, just because the source volumes are also NTFS.

Is not the key issue separation of the backup data from the source data, much more so than the type of file system on which the backup data lives?

We trust these file system technologies enough to have confidence that even if our primary systems were to fail, the backups/offsite systems can be used. Otherwise, we probably wouldn't use either VMware or Windows.

-Taylorbox

[Gostev]

If I follow your reasoning here, then we should likewise avoid saving backups of Windows servers onto NTFS volumes, just because the source volumes are also NTFS.

Well, I think I will disagree with this one. NTFS is quite different story - it has not been changing for so many years, and is truly proven... VMFS is very "volatile" on the other hand, constantly changing. Moreover, just had its major release (VMFS5). You know what I am talking about? For example, look how careful Microsoft wording is regarding implementing ReFS in production right after Windows Server 8 release

[taylorbox]

Well, I think I will disagree with this one. NTFS is quite different story - it has not been changing for so many years, and is truly proven... VMFS is very "volatile" on the other hand, constantly changing. Moreover, just had its major release (VMFS5). You know what I am talking about? For example, look how careful Microsoft wording is regarding implementing ReFS in production right after Windows Server 8 release

Gostev:

I understand your point, yet if VMFS is so shaky of a file system, then why would Veeam have built its business around a product (VMware) who's file system we shouldn't even trust?

...Oh, wait, that is why Veeam B&R was created!

Seriously, I think we're now making rather subjective comments. In my career I've dealt with corrupt NTFS volumes more than a few times, but not once have I seen VMFS even hickup, let alone blow up (including 1000's of VM's on dozens of hosts). But ask another tech and they'll give you their own unique story.

-Taylorbox

[dellock6]

I agree VMFS is a really stable filesystem, imho more than ntfs; my objection has appeared maybe more philosophical than technical.
I meant, to use two layers made by different filesystem "for me" is a non-sense for backups, since in this way I need to restore an ESXi first, than a Windows machine with Veeam Backup Server, than I can finally restore a VM or its files.
With a native NTFS storage, I do not need the first step, and since recovery has to deal with RTO, faster is better.

[taylorbox]

I agree VMFS is a really stable filesystem, imho more than ntfs; my objection has appeared maybe more philosophical than technical.
I meant, to use two layers made by different filesystem "for me" is a non-sense for backups, since in this way I need to restore an ESXi first, than a Windows machine with Veeam Backup Server, than I can finally restore a VM or its files.
With a native NTFS storage, I do not need the first step, and since recovery has to deal with RTO, faster is better.

Luca,

I understand your explanation. Thank you for your comments.

In a 100% virtual environment (including the Veeam server), it would seem unavoidable to have to rebuild an ESXi server first.

-Taylorbox

[Heiko]

Hi

It has been mentioned above that 1st thing is the risk behind everything. How far does a customer wish to go to protect their data? It is NOT good practice to keep all your eggs in one basket.
Take note that the chances are very slim that all datastores get corrupted at the same time. It is so or so NOT got practice to have huge datastores (extends) to store production VM's on. It is a cost relationship as well. The more you wish to protect the bigger your budget.

Why not use both scenarios? Please comment and give your ideas?
1st Backup - to local Datastore - This is being used for immediate day to day restores. I keep as many copies as possible what the assigned storage can hold.
2nd Backup/replication is to outside of the production VMFS file system. Either remote storage (DR) or CIFS share. Most SAN's support CIFS shares these days and the performance is great. This will result if a worst case scenario the production VMFS gets corrupted you can recover. The second good part CIFS you can backup very easy or replicate.
3rd - If need/do a tape backup. - Last resort of restore and only being used when complete production data is gone

[Gostev]

Heiko - actually, what you explained is the ONLY right way to do things. Backup is not a backup until you have 3 copies of it. However, I see too many customers neglecting this rule of thumb, often because "doing this right" requires up to 3x of backup storage, and they simply cannot afford doing it right...

[bhwong]

I was having the same thought as taylorbox regarding using VMFS for storing backup.

1. Installing an ESXi to retrieve backup data thru VMFS is much faster than installing Windows to retrieve backup data thru NTFS.

2. If all ESXi hosts are down, you will still need to setup a new ESXi host anyway in order to run the backup VMs.

3. Chances of VMFS corruption - never experienced before. Chances of NTFS corruption? Quite often. At least an virus attack to delete files on an VMFS volume is highly unlikely.

4. Simulaneous connection (write) in a SAN volume is supported by VMFS but not NTFS which will lead to corruption. (Having 2 backup app writing to the same volume at the same time etc?)

Having said the above, someone reminded me about the big difference between using the storage for backup and replication. For replication, it will be VMFS instead of NTFS.

However, since backup will be compressed and dedup, we will need Veeam to restore it, or use extract utility provided. Thus, NTFS will provide direct access to restore the backup from any connected Windows computer or even execute the backup VMs to boot up immediately thru vPower. With VMFS, we will add an additional step of mapping the VMDK files into a Windows VM etc to access it. That's the only disadvantages I can agreed upon, apart from the 2TG limits.

[sergeadam]

Additional consideration. My VEEAM proxies are VMs. The backup targets are Dell MD1200 SAS attached arrays. ESXi will nor see any 'locally attached' arays as RDM. I have to create a vmdk to point to the storage. Sometimes you can't get away from vmdks.

[Gostev]

Thank you for your opinions, guys. This will let future readers make the right choice, having all pros and cons in hands.

[pcrebe]

Hi,
With the new v7 version it will be supported a vmdk disk for store the backups?
I think now you can create a linked backup copy job to "save" your backup to a CIFS share...

Thanks,
Carlo

[Gostev]

Hi Carlo - it was always supported, just not recommended... but if you have a copies of your backups sitting somewhere else, then it is certainly fine to use VMDK as a primary backup repository. Thanks!

[pcrebe]

Hi Gostev,
Back from holiday i will install the new v7 using, as defaul repository, a vmdk disk attacched to the B&R server and creating the new backup copy job for GFS rotation on my qnap.

Some suggestion for compression?

Being now the B&R vm on the destination host of replicas i'm thinking to store replicas in the production SAN, another raid and controller, for best RTO and the backups on the local esxi datastore because if i'll lost that esxi host all my replicas and backups would be lost too...

Thanks!

[Vitaliy S.]

I believe the default compression settings should work best.

[karlochacon]

hey guys what about VMDK size?

now that 2TB limit is gone, can I just use 1 single 15TB repository in my Veeam Server stored in datastore as VMDK? or should create instead 3 VMDKs separated in 3 SCSI controllers? any ideas?

thanks

[veremin]

You can do that, but it's still not considered to be the best practice due to the complexity it brings to restoration procedure. Thanks.

[karlochacon]

which procedure brings complexity? one big disk VMDK or multiple VMDK disks (small ones)?

[foggy]

Vladimir means complexity caused by the fact that you'd need to setup a new VI prior to starting the actual restore (while with NTFS repository you could just start the restore right away).