vStorage API in Virtual Appliance mode

[TrevorBell]

[UPDATE 1/11/2010] As of ESX4 Update1, vStorage API Virtual Appliance processing mode no longer requires that you have VMware Advanced or later license. In Update 1, VMware had fixed what appeared to be the SCSI Hot Add feature licensing glitch, so now you can enjoy all benefits for Virtual Appliance mode with any VMware vSphere license level, including VMware Essentials and Standard.

What sort of setup is required for this just a VM runnign VB4 with iscsi san lun for the images ?

[Gostev]

Trevor, the beauty is - no setup is needed. You just install Veeam Backup in a VM, and select this backup mode - that's it. And Veeam will be able to do direct-from-SAN backups through ESX I/O stack (instead of network stack). Speed should be quite similar to SAN mode (per VMware), and you will be able to backup all VMs located on the same shared storage as Veeam Backup VM (but not from other storages which are not connected to ESX on which Veeam Backup VM runs).

But please don't bother with this right now, because we have confirmed that this mode is indeed not working properly in BETA1 - we will try to fix this by BETA2.

[TrevorBell]

oki keep me posted when you need a test doing... ill gt a vm ready with alot of space to do a good test when needed..

[Gostev]

I will really appreciate if someone can do some testing in Virtual Appliance mode with fast SAN storage, and compare performance with direct SAN access mode. Due to our SAN currently having performance issues, it is really hard to compare performance between different modes, as they all produce equally low speed

Thanks in advance!

[tsightler]

I plan to test this, but due to time constraints with real work getting in the way I probably won't be able to get to this until Friday afternoon.

[dkeathii]

I'll do some local testing, but my san isn't really a speed demon.

[dkeathii]

I fired it off first from the vc server(physical box outside the VMs) using san mode, it switched to san/nbd and from the network usage looked like it was pulling one through the san interface and the other through the lan interface. I was seeing about 40MB/s. Both source and destination were on the same SAN, different LUNs.

I installed veeam rep. on a guest VM and tried the same job. Same SAN as the first test, same situation (Guests on same SAN, different LUNS)
I got some funkyness inside task manager, and the job failed. See screenshots below...or not, I don't see how to attach anything. I'll email it to you.

This won't cause me any issues as I intend to continue using the VC server, but since you asked...

[TrevorBell]

Ill have a play with this today not much else to do

[Gostev]

Daniel, the screenshots look really weird indeed... could you please send me the logs, we would like to investigate.

[TrevorBell]

First impressions :

I made a New vm win 2008 32bit, 40 gig c drive, picked one VM and backed it up to the 40g drive was getting 41Mb/s not too bad....
Now i have made a 100gig LUN off the SAN and doing some more testing i have to say the speeds are pretty good. First set of figures are normal san backup mode followed by the Virtual appliance mode.


San Mode Backups
CSDOM01
6 of 6 files processed
Total VM size: 24.00 GB
Processed size: 24.00 GB
Processing rate: 74 MB/s
Backup mode: SAN/NBD with changed block tracking
Start time: 10/14/2009 7:50:50 PM
End time: 10/14/2009 7:56:20 PM
Duration: 00:05:29

CSPRT01
6 of 6 files processed
Total VM size: 24.00 GB
Processed size: 24.00 GB
Processing rate: 62 MB/s
Backup mode: SAN/NBD with changed block tracking
Start time: 10/14/2009 7:44:14 PM
End time: 10/14/2009 7:50:50 PM
Duration: 00:06:35

CSPDL01
6 of 6 files processed
Total VM size: 24.00 GB
Processed size: 24.00 GB
Processing rate: 36 MB/s
Backup mode: SAN/NBD with changed block tracking
Start time: 10/14/2009 7:32:51 PM
End time: 10/14/2009 7:44:14 PM
Duration: 00:11:22

Now for New Virtual Mode
CSPRT01
6 of 6 files processed
Total VM size: 24.00 GB
Processed size: 24.00 GB
Processing rate: 79 MB/s
Backup mode: HOTADD with changed block tracking
Start time: 15/10/2009 11:14:41
End time: 15/10/2009 11:19:52
Duration: 00:05:10

CSPDL01
6 of 6 files processed
Total VM size: 24.00 GB
Processed size: 24.00 GB
Processing rate: 75 MB/s
Backup mode: HOTADD with changed block tracking
Start time: 15/10/2009 11:09:14
End time: 15/10/2009 11:14:41
Duration: 00:05:27

CSDOM01
6 of 6 files processed
Total VM size: 24.00 GB
Processed size: 24.00 GB
Processing rate: 95 MB/s
Backup mode: HOTADD with changed block tracking
Start time: 15/10/2009 11:19:52
End time: 15/10/2009 11:24:10
Duration: 00:04:17

[TrevorBell]

All figures are LIVE PRODUCTION network not a test envoriment

Here is our Terminal Server VM only about 2-3 people using it but here are the figures -:

6 of 6 files processed
Total VM size: 24.00 GB
Processed size: 24.00 GB
Processing rate: 23 MB/s
Backup mode: SAN/NBD with changed block tracking
Start time: 10/14/2009 7:56:21 PM
End time: 10/14/2009 8:14:01 PM
Duration: 00:17:40

6 of 6 files processed
Total VM size: 24.00 GB
Processed size: 24.00 GB
Processing rate: 31 MB/s
Backup mode: HOTADD with changed block tracking
Start time: 15/10/2009 11:24:11
End time: 15/10/2009 11:37:24
Duration: 00:13:13

[Gostev]

Trevor, thank you for testing this. Most important thing for me is that speed is comparable with SAN mode, and even a bit faster. If there was an issue I suspected, speed would be 5-10 times smaller than with SAN mode (just like with "original" BETA2 performance drop). This looks good, thanks again for checking this!

[TrevorBell]

Anton,

The VM i made was 2 gig memory and 4 CPU when the operation is running it is intense with 78% guest memory used too....so proves you need to set aside some hefty Ram and Cpu

Im thinking now of moving to this mode for all my backups and then i can remove the Poweredge 860 which is the physical backup server.....another server out the server room!! very good feature this

[Gostev]

Im thinking now of moving to this mode for all my backups and then i can remove the Poweredge 860 which is the physical backup server.....another server out the server room!! very good feature this

Ditto! You should go 100% virtual, like our production.

[TrevorBell]

I can also report very good restore times using this method and also very good replication too, better than SAN mode in my enviroment anyways

Thanks

Trev

[Gostev]

So I was thinking more on why Virtual Appliance mode works faster for you, and I think I have one idea. This could due to the VM having faster memory I/O than your physical server, because VM is running on modern server hardware (very fast memory bus). On the other hand, physical host you use for Veeam Backup is probably based on some older hardware?

Thing is, unless CPU is 100% loaded, memory I/O performace is what mostly affects backup performance with "ideal" storage access (faster memory improves on-the-fly dedupe and compression speed).

[Gostev]

[UPDATE 1/11/2010] As of ESX4 Update1, vStorage API Virtual Appliance processing mode no longer requires that you have VMware Advanced or later license. In Update 1, VMware had fixed what appeared to be the SCSI Hot Add feature licensing glitch, so now you can enjoy all benefits for Virtual Appliance mode with any VMware vSphere license level, including VMware Essentials and Standard.

We are still researching Daniel's issue reported early in this thread, but it looks like Virtual Appliance mode requires that you have VMware license with the following feature (see below in red):

[floss]

Hi I am running Backup via The Vstorage API in VA mode and its works superb i am seeing backup speeds of on 125-150mb per sec , one question i can see when the backup runs four extra disks are created then deleted on my Virtual backup sever can someone expalin in detail how the mode work in terms of the scanning and data transfer througn the ESX i/0 stack

[Gostev]

Hello. In this mode, virtual disks of the backed up VM are attached directly to the Veeam Backup server by leveraging SCSI Hot Add functionality (this is exactly what you are observing), so their content becomes available for Veeam Backup server "directly" (just like Veeam Backup VM's own disks). So all data transfer is performed directly from storage via ESX storage I/O stack, instead of network I/O stack as in case of the network backup. All the rest is still the same as with other backup modes (including changes scanning).

[TrevorBell]

Guys HOT ADD is the real way forward for VB if you get chance to try it out and have enterprise Vmware license it really is the fastest option for backups in my enviroment getting 10% faster than traditional API backups. Hopefully you all will see the same results in your enviroments.

[Gostev]

Actually it should work with "Advanced" license and up. Simplest check is this: if you have VMotion, you should be able to use it.

Biggest reason for such a great performance is due to the fact that VMware had completely re-written ESX iSCSI initiator in ESX4, and sounds like they did great job with this.

I believe that for powerful modern physical backup server and HBA with latest drivers, performance should be a few percent faster than for Virtual Appliance mode (ESX still has some overhead comparing to physical). But in all other cases, Virtual Appliance mode will be winner. ESX servers typically run on high-end server hardware which beats regular physical servers on memory speed, while this is defining factor for Veeam Backup performance if CPU and source/target storage are not bottlenecks.

I think the biggest benefit this mode brings is not even performance, but the fact that you can remove one more physical server from environment. However, you have to have ESX host with some good spare CPU capacity - Veeam Backup can chew all CPU you throw at him when it pumps the raw data from storage at 100MB/s. So while you are removing one physical server, you are taking big chunk of power from another one (although this may not be a problem for off-hours backups).

Trevor, could you please send me the logs from this job, I would like the devs to review and find out what have caused the error above (just in case).

[tsightler]

Actually it should work with "Advanced" license and up. Simplest check is this: if you have VMotion, you should be able to use it.

Don't think this is quite true. We have vMotion, but it was purchased as an addon to our "Standard" edition back in the early vMotion days (ESX 2.x). Since we've always been under maintenance, we still have vMotion (along with Storage vMotion) with our Standard edition ESX 4 servers, but Standard edition does not include Hot Add support. Disappointing because we were hoping to use that mode but we'll get over it.

[Gostev]

Yep, my rule of thumb does not account for custom feature sets I just thought it would be easiest question for us to ask the customer to determine the vSphere license level.

On the other note, I wonder if it is possible to upgrade only 1 ESX host to "Advanced"?

[tsightler]

Yep, my rule of thumb does not account for custom feature sets I just thought it would be easiest question for us to ask the customer to determine the vSphere license level.

Perhaps, but in my case the answer would be misleading. It seems you could be 100% sure by simply asking them if they list "Hot-Pluggable Vitrual HW" in the licensed features on the Configuration tab. Perhaps your way covers enough of the users that's it's not a big deal, but I'm a stickler for the details, it's a personality flaw.

On the other note, I wonder if it is possible to upgrade only 1 ESX host to "Advanced"?

I don't see why not. We're seriously considering upgrading our core datacenter hosts to Enterprise Plus because there are currently some big discounts and we think it may pay off once the core count in server processors goes above 6, but I doubt we'd upgrade the servers at our remote sites.

[Gostev]

On the other hand, remote sites probably do not even need this mode to start with? Because most typically they are very small (2-3 ESX with few VMs to backup) and all ESXes are with local storage (so only "Network" mode can be used)? At least this is what we are seeing in huge accounts like retail chains.

[tsightler]

Maybe, maybe not. Our remote sites generally only have 2 servers, and 5-7 VM's, typically 500-1TB of data or more, but they do have small, iSCSI SAN's because with think vMotion is worth it, giving us much flexibility during hardware/OS maintenance. I suppose with Veeam 4/block change tracking we could probably use two ESX servers with local storage, and Veeam replication running every few minutes and perhaps actually be more robust without the SAN.

Most of our remote sites have no physical servers at all, or if they do have a physical server it's a Linux server, so hot add mode would be great for these because it's ideal for running Veeam within a VM and still does direct SAN backups. The network mode would be almost as good, but is much slower with ESX 4, although perhaps not a big deal with block change tracking. We plan to do a lot of testing with our remote sites with Veeam once v4 finally hit "release" status. We've never really run Veeam at the remote sites, but we're thinking it might be able to cover pretty much all of our backups now.

[Gostev]

Agree - with 2 separate local storages all eggs are no longer in one basket.

[Gostev]

This thread was moved here from the closed beta forum.

[schaffeb]

I've been testing replication in Virtual Applicance mode, which works great. However, even after the replication job is done, I still see the disk of the vm I replicated in the Veeam VM's explorer. Shouldn't it be disconnecting after the replica is complete?

[Gostev]

Ben, probably the explorer just does not autorefresh itself properly... disks are actually disconnected right after backup is done (otherwise, snapshot removal would be impossible).