Backup Target ESXi Not Supported (Yet?) In v4.1.1

[bproven]

I'm reading through the documentation and have attempted a backup in my lab environment and it appears that ESXi cannot serve as a target. I searched a bit and found some posts from about 6 months ago confirming this for backup and replication, but it looks like replication at least is now supported in 4.1.1. If this is true (no ESXi backup target), when is this feature expected? v5 possibly? We are fully ESXi 4U1 environment, so at this time it seems replication would be our only option with this software.

[Gostev]

ESXi do not make good backup targets because of bad I/O performance (enough to handle replication of selected VMs, but really lacking to be a target for backup jobs with multiple VMs). Do you have a file server VM you could use as backup target? Or, I suppose you can just create new VM to serve for this purpose?

Most typically, our customers use cheaper NAS devices as backup targets (this way, you don't loose your backups together with ESX). Alternatively, even cheaper way is to use USB hard drive connected to some desktop (just share it on the network).

[bproven]

OK - sounds good. I'll give that a try since we do have Linux (physical) servers to backup to.

Another question - would this I/O issue with ESXi also present a problem if the ESXi was a target, BUT the datastore was an NFS share on the same ESXi host? In other words ESXi A has NFS datastore attached and you were to backup a VM on ESXi A' local datastore to an NFS share attached to ESXi A. Recovery could then be (maybe) having ESXi B mount this NFS share and run those backed up VMs live from NFS in DR situation. MAybe a bit crazy as most people might just go SAN all the way, but for cheap this might work for us Essentials folks running in DAS environments only

Anyway, just curious. It will be great when VMware does address this issue with slow file transfer I/O on ESXi hosts -at least I hope they do. Especially since they are pushing ESXi as a total replacement for ESX.

Again, thanks for your help.

[Gostev]

Actually, Linux will make even better backup target. We support backing up to Linux servers natively, just add the server to Veeam Backup via Add Server wizard, and it will become available for selection in the Backup Destination step of the job wizard.

Yes, I/O issue will be present in any scenario when downloading or uploading files from/to ESXi storage through the host.

[MadDog2K]

Gostev,

ESXi do not make good backup targets because of bad I/O performance
(enough to handle replication of selected VMs, but really lacking to be a target for backup jobs with multiple VMs).

do you know if VMware will address this issue, like with the bad performance for ESX ?

[Gostev]

I am not aware about any fixes coming... the problem of slow download/upload with VIC datastore browser is as old as ESX itself, VMware users have been complaining about this for years, and since it is still not resolved, I doubt it will ever be... I think it is simply technical limitation of some sort.

And while we are able to workaround this issue by using service console agent on "fat" ESX, this is not an option for ESXi.

May be VMware simply does not want us to use ESX(i) hosts as file servers, and throttles network VMFS I/O intentionally not to affect VMs running on the host.

[jroot]

Anton,
Please don't view this as a complaint against your fine product but additional user input.
VMWare has publicly stated that they want their user environment to move to ESXi. We built our infrastructure around that direction so all our hosts are ESXi. We have also invested in our first SAN components but have loads of local storage capacity. We want to utilize that capacity for backups but cant's access it through Veeam. Despite the VMFS I/O throttling, wouldn't this scenario be faster than VMware vStorage API in Network mode and suffer less performance degradation? Why not support the capability or at least give us the choice so we could test actual results and make the decision for ourselves?

[tsightler]

Actually, there's no reason you can't use your local VMFS storage for backup, you just can't use it directly via ESXi. You can easily install a Linux or Windows VM with it's local disks on the local VMFS storage and then use that as a backup target. Actually, if you use Windows, you can leverage virtual appliance mode to backup other VM's to the local VMFS storage and not be affected by the ESXi I/O throttling at all.

[jroot]

Tom,
I'll give that a shot. For whatever reason, I didn't view that as a performance option that would beat Network mode but it really should be close to the performance limits of the local RAID adapter.
Thx and I'll report back soon.

[tsightler]

Actually, reading data from a VM that's on VMFS storage is typically faster than reading directly from VMFS as VMFS is heavily optimized for the storage of virtual disks. Reading and especially writing via the ESXi agent, or even the ESX service console, causes significant overhead because of the extra locking that has to happen in this scenario due to the frequency of metadata updates. Reading and writing to a fat VMDK file on an VMFS volume has no such requirement (VMware suspends updates of atime and mtime attributes and writes directly to the reserved blocks). For thin mode VMDK files writes still have some additional overhead as blocks are allocation, but VMware typically allocates blocks in 16MB increments to keep the amount of metadata from increasing significantly.

We run several linux VM's that are used as backup targets with local storage and this method works very well.

[jroot]

Tom,
Based on our test from just a few minutes ago, the results back up your ascertions. We created a 256GB drive on the same local storage and connected it directly to the Veeam 4.1.2 VM. We just backed up a 100GB thick disk vCenter Server using virtual appliance mode and obtained 73MB/s with a 0:23:21 duration (full backup). The same backup using the storage api in network mode was processed at 18MB/s and 1:34:21 duration. Why did I not try that before today!? Thanks for weighing in on this Tom! I appreciate it. I like the idea of using the Linux VM's since our network backup targets are Lime unRAID(linux) servers. Did you use someone's template or a favorite distro?

[tsightler]

Did you use someone's template or a favorite distro?

I didn't use any special template. It's basically our in-house, stripped down RHEL5 install. We're a Redhat Enterprise Linux shop, but for those systems we actually just used CentOS 5 with the same kickstart template, it's basically a default RHEL5 install with every option unchecked, even "Base", and then a few options cherry picked back in. The kickstart is very similar to the one available at http://blog.vodkamelone.de/archives/151 ... ation.html.

[arsprod]

Actually, there's no reason you can't use your local VMFS storage for backup, you just can't use it directly via ESXi. You can easily install a Linux or Windows VM with it's local disks on the local VMFS storage and then use that as a backup target. Actually, if you use Windows, you can leverage virtual appliance mode to backup other VM's to the local VMFS storage and not be affected by the ESXi I/O throttling at all.

How do you get a windows vm to show up in virtual appliance mode?

[Gostev]

Sorry Aaron, I do not understand what you are asking. "Virtual appliance" is one of the processing mode option for Veeam jobs (defines how source data is retrieved).

[arsprod]

tsightler wrote "Actually, if you use Windows, you can leverage virtual appliance mode to backup other VM's to the local VMFS storage and not be affected by the ESXi I/O throttling at all" as an option to creating a linux machine as a backup destination. I'm not clear how to make the windows vm show up as a virtual appliance.

aaron

[tsightler]

"Virtual Appliance mode" refers to installing and running Veeam inside a VM. When Veeam is running as a VM you can use "Virtual Appliance mode" as a processing mode, which basically uses SCSI Hotadd to present LUN's from other VM's to the Veeam server so that they can be backed up. What I was referring to above is that, if you have to backup to a VMFS volume (not my preference but perhaps sometimes required), the best performance is usually achieved by creating the VMDK and presenting it to the Veeam VM rather than backing up via the ESX console, that's all.

[arsprod]

ahh, thanks for clarification