Using Veeam to backup VMs on a NFS Datastore

[nmace]

We've been happily using Veeam B&R (version 7.0) to backup our VMs from a traditional iSCSI SAN based setup. However we're moving away from iSCSI to NFS, but I can't seem to find much of anything about how I can backup my VMs stored on a NFS share. Does such a thing exist for it as the "SAN mode" backup scheme that I've used and enjoyed (at 10 gigabit)? Pulling the VMs into Veeam via the hyper-visor is doable....but that seems so kludgy after getting used to Veeam having direct access to the VM datastores. Is there any other option with NFS? All of my searches turns up stuff about the Power NFS service, which isn't the type of NFS I'm looking for.

[Gostev]

There is no such thing as direct SAN access, best you can do is hot add (technically this is also direct SAN access, except the data goes through ESXi storage stack).

Also, be prepared to face NFS-specific issues you've never had to deal with before. NFS storage in VMware has really bad track record as it comes to backup... a number of ugly issues in the past, like this one for example. Even though most of those issues have been fixed by VMware, I personally would not move from block storage to NFS. Based on my experience observing all the issues our customers had with NFS storage in the past 5 years, to me it's clear NFS is a step child for VMware, and is not being tested as well as block storage. Also, no NFS v4 support in 13 years since release speaks a lot for whether NFS storage is a focus for VMware.

That said, I've also met customers who have been reasonably happy with NFS storage... just not enough to change my perception, especially considering they all sounded like hardcore NFS fans to me, and the talk was mostly focused on why block storage sucks

[nmace]

Thanks Gostev.

So if I am understanding correctly, I just need to add the new vCenter to B&R and let it pull the data it needs through the hypervisor stack? Nothing extra needs to be done (like what was needed to enable SAN mode) since none of that would work with NFS?

[foggy]

You need at least a proxy server running on the host you're backing up VMs on. Here's a bit more on hotadd configuration.

[nmace]

Thanks.

What OSes are supported by Veeam for the proxy server? Server 2012?

[nmace]

Question. Doesn't this indicate that NFS Storage isn't supported direct san access, virtual appliance, or network mode? Is the chart out of date? Or am I mis-understanding something?

http://helpcenter.veeam.com/backup/70/v ... modes.html

[Vitaliy S.]

This chart says that NFS storage can be used with Virtual Appliance and Network modes only. Not sure I follow you here...

[nmace]

LOL. I'm sorry, I had to look closely but you are correct. I thought the "Not Supported" box spanned all 3 columns instead of just the first. The lines indicating the edge of the boxes in that chart are hard (for me at least) to see. Sorry about that, nevermind!

[Gostev]

What OSes are supported by Veeam for the proxy server? Server 2012?

A lot. You can find the complete list in the System Requirements.

[dellock6]

I'm not an NFS hardcore fans, but I see why people want to implement it in VMware:
- visibility at the VM level instead of LUN level (look at Tintri or Nutanix for example and what they can do with NFS)
- no problem with scsi locking, all lockings happen at the single file level. This means a large NFS storage can hold many many VMs without having to span VMs between multiple LUNs
- many new storage systems are coming out with NFS-only frontend connection, so there is no real choice for customers

That said, many of these new systems are also running with 10 GBps connections, and this alone makes network mode backups really efficient, since network mode by the way is the less impacting mode on vSphere itself.
Probably, many of these advantages of NFS against block storage are going to be reduced once vVols will be available, and even with block storage we will be able to see everything at the VM level and not at LUN level.

Luca.

[Gostev]

no problem with scsi locking, all lockings happen at the single file level. This means a large NFS storage can hold many many VMs without having to span VMs between multiple LUNs

Hardware-assisted locking through VAAI has rendered this benefit obsolete though with that, locks are atomic and happen on the block level vs. LUN level.

many new storage systems are coming out with NFS-only frontend connection, so there is no real choice for customer

I can also see why all the storage startups are using NFS these day... I think this because NFS is so much easier to implement. This does not necessarily make it better choice for customers though

Probably, many of these advantages of NFS against block storage are going to be reduced once vVols will be available, and even with block storage we will be able to see everything at the VM level and not at LUN level.

Yep. This is the future.

[dellock6]

Hardware-assisted locking through VAAI has rendered this benefit obsolete though with that, locks are atomic and happen on the block level vs. LUN level.

Uhm, not 100% agree Anton, mainly because of two reasons:
- vaai is not available on every storage, and not in every VMware license, only enterprise and enterprise plus
- vaai is still less efficient than NFS locking, since it still uses scsi T10 commands and has some limits, even if higher than usual scsi locking (see for example http://www.boche.net/blog/index.php/201 ... rban-myth/)

NFS instead is available at every vSphere edition, even the old one without VAAI, and is simple to implement as you said. And again, this is coming from an iSCSI fan like me.
Probably vVols are going to make these discussions useless, at least it's what I hope (together with a better VM snapshot management...)

Luca.

[pcrebe]

Hi,
I've changed from iSCSI to NFS and everyday i've CBT warnings in replica and backup jobs

Now, reading this post, i'll think if come back to iSCSI

Thanks

[kalebksp]

I'd say the NFS vs block decision comes down to your storage vendor and the features they support. I primarily work with NetApp and the majority of my customers use NFS.

Pros:
Easier to configure - Don't have to deal with LUNs, creating igroups and adding IQNs, and with the vSphere Web Client you can add the NFS export to all your hosts at once.
Thin provisioning - No worrying about a LUN going offline due to running out of space. Still not great if a NFS store runs out of space, but it's not nearly as bad.
Shrinking Datastores - You can increase a VMFS datastore but there is no support for shrinking.
Space Utilization* - Controller knows when a VMDK is deleted and dedup is more efficient.
Cloning* - It's possible to near-instantly clone a VM at the controller level. Although with VAAI sub LUN cloning now allows the same functionality.
SMB Access* - Can access your datastore through NFS and SMB simultaneously. Useful for loading ISOs, copying VM files, etc.
File Restore* - A VM file can be directly restored out of a snapshot from the controller, no need to make a R/W copy of the LUN, then restore.
*These may be NetApp specific features.

Cons:
Backup - As mentioned, if using VADP for backup there is no SAN mode. With hot-add you have to deal with the snapshot removal issue Gostev mentioned. All my environments use network mode reliably.
Multipathing - There is no MPIO for NFSv3. Even if you use link aggregation to your host it will only use one link due to the source-destination hashing. You can get some semblance of load balancing by mounting separate datastores with different IPs. You can not however mount the same datastore with different IPs if you want vMotion and HA to work. Another option is DNS round robin, though I typically don't like my storage connections to rely on DNS. (We can continue to hope that VMware will someday support pNFS to get around this.)
Maximum Datastore - No more than 256 datastores per host.
Exchange support - Technically Microsoft does not support Exchange on NAS protocols. It works fine as far as I've seen, but if you run into storage problems Microsoft probably isn't going to help you out. (SQL is supported.)

[ITP-Stan]

With HP SAN's you can use the HP VAAI plugin on all vsphere editions.

[dellock6]

Really? Where can I find informations about this? Because as far as I know, VAAI is a VMware feature only available on some vsphere editions.

Also, by the way, the "VAAI plugin" is not needed anymore since vSphere 5.0 on "any" block storage, they were added to the official T10 commands right to remove the need for a dedicated plugin. Plugin is only needed for NAS storages. Which HP storage are you talking about?

Luca.

[tinto1970]

I'd say the NFS vs block decision comes down to your storage vendor and the features they support. I primarily work with NetApp and the majority of my customers use NFS.
....
Backup - As mentioned, if using VADP for backup there is no SAN mode. With hot-add you have to deal with the snapshot removal issue Gostev mentioned. All my environments use network mode reliably.

hi all, i'm totally a newbie with Veeam, I've got my first license last week and scheduling the first backups on an handful of Linux VM running on NFS Datastores.
As every newbie does, i've started with the "all default" options, and i see "hot add" is being used (guess so because the VM where Veeam is running has disks attached when performing backups).

At the moment all seems to be working very fine: i've tried to launch backups on VMs running in production hours without recieving calls from angry people. Jobs scheduled during nights seems to be fast and reported as successful until now. As suggested, in case of troubles i will try switching to Network Mode, but i hope not to run into troubles

P.S.: i'm using veeam b&r and vsphere 5.1u1, both at lastest build, and the nas used is a netapp 2020, which is the "entry level" of a few years ago, already carring a significant load.

[CaptainFred]

I have a few questions about this because we have recently purchased some NFS storage to go along side our iSCSI storage:

1. So to obtain the best performance the best way to backup the VMs on the NFS storage is to deploy a Veeam Proxy to a VM on the NFS storage and set it to Virtual Appliance mode (Hotadd)?

2. Does the VM that's the Veeam Proxy need to be on the NFS storage or could it in fact be on the iSCSI storage?

3. Does the VM that's the Veeam Proxy need to be on the same ESXi host as the VMs on the NFS storage?

4. We have 10GBe Storage Network (for the NFS and iSCSI SANs) separate from the 1Gbe VM/Management Network. Regarding the Virtual Appliance mode, I understand that this adds to the VMDK of the VMs that it's backing up to itself and copies the data directly out of them, BUT doesn't this data then travel across the VM/Management Network to the our physical Veeam Backup Server? Meaning that, the fastest the backup can run at is 1Gb/s whereas the backups from the iSCSI storage are much faster because our physical Veeam Backup Server can use Direct SAN Access? It has a dedicated 10Gbe connection to the 10GBe iSCSI Storage Network and the VMFS Datastores are presented to it.

[Vitaliy S.]

1. Correct.
2. It has to be on the host that has access to both storages.
3. Yes, see the VMware KB article Anton has referenced in earlier posts.
4. Why do you think this data should travel to your physical backup server? If you have a virtual proxy, then data will go directly to the repository you have specified.

[dellock6]

About point 4, Vitaliy maybe the physical Veeam server is also the repository, so even if he deploys a virtual proxy, backups will flow from this to the physical repository via the VM network. If this is the case, a good solution could be to create (if possible) a dedicated BAN (backup area network) not via vlans but proper separated ethernet channels, to be used for all the backup traffic. If the 10G allows for virtual segmentation (for example flexfabric on HP) it can use a part of the storage network.
As an addition, on 10G ethernet I would also consider Network mode, directly from the ESXi server to the physical Veeam. In order to do this, you only need to create e secondary vmkernel on the ESXi facing the 10G network.

Luca.

[Vitaliy S.]

Hi Luca, yes, it could be. Good tips, btw!

[CaptainFred]

Yes dellock6 is correct (thanks), our physical Veeam Backup Server is also the repository. So the data DOES across the 1Gb/s management/VM network.

Ok so how can I create a BAN on our 10G infrastructure? Should I create a VM port Group on the iSCSI vSwitch on each host and put a NIC of the Veeam Proxy VM into that and give it an IP in the same subnet as my physical backup server has in the 10G network? Although how can I force/configure the backup process to use this network instead of the management/VM network?

[foggy]

Although how can I force/configure the backup process to use this network instead of the management/VM network?

Just add the proxy to Veeam B&R console via the corresponding IP address.

[CaptainFred]

Ok thanks

Regarding my above question "3. Does the VM that's the Veeam Proxy need to be on the same ESXi host as the VMs on the NFS storage?" which Vitaliy S. answered "3. Yes, see the VMware KB article Anton has referenced in earlier posts." Are you sure about this because this would mean that if I spread my VMs across my 6 hosts that are on the same NFS storage I need to deploy 6 Proxies as VMs, one on each host?!

[Vitaliy S.]

Well.... My answer is based on the VMware KB article that recommends this kind of setup to avoid VM stuns during snapshot commit operations.

[CaptainFred]

Hmmm interesting. Ok to use the alternative, NBD backup transport option and create a BAN on my 10G network, do I simply create another VMkernal on my iSCSI vSwitch on all hosts and tick the Management Traffic box? Then point the physical Veeam Backup Server at the 10G Management IPs?

[Vitaliy S.]

Yes, I believe so:

As an addition, on 10G ethernet I would also consider Network mode, directly from the ESXi server to the physical Veeam. In order to do this, you only need to create e secondary vmkernel on the ESXi facing the 10G network.

[dellock6]

You don't need to tick the "management traffic" option on that port, only create a vmkernel type portgroup. The management traffic option in reality only allows HA heartbeat traffic to use that portgroup. All other options (ssh and even connection via vclient or join to vcenter) works even if the management traffic option is not selected.

Luca.

[CaptainFred]

Thanks, this works although performance doesn't seem that much better. Will test some more and feedback again soon.

[ITP-Stan]

I was talking about a MSA P2000 G3. I have used the HP VAAI plugin on Standard & Enterprise editions of VSphere.
I know it's no longer necessary, but when some hosts are already using it, you have no other option then to use it on new hosts aswell without downtime scheduled.