Very little success in VA mode

[scottyyyc]

After a very impressive and successful trial of 4.1.2, our company has decided to purchase and implement Veeam B&R for our vSphere environment. Our environment is fairly simple - 3 ESXi 4.0u2 hosts, vSphere Advanced, all running on a Dell EqualLogic iSCSI SAN.

When demoing the product, I installed Veeam on a spare physical server I had lying around (Server 08R2). I added my vCenter system, and everything worked very well and as expected (not to mention - FAST!). In production, I've decided to install Veeam on a VM (Server 08R2). I can't really seem to get a simple test backup of 1 XPSP3 VM running at all.

Ive tried using the vStorage API mode with both network backup mode and VA mode. In network backup mode, it's INCREDIBLY slow (a couple MB/s), and the CPU is pegged at 100%. In VA mode, it never works at all, I get the following message:

Code: Select all

4 of 5 files processed 

Total VM size: 50.00 GB
Processed size: 12.26 KB
Processing rate: 0 KB/s
Backup mode: HOTADD with changed block tracking
Start time: 09/29/2010 8:39:25 AM
End time: 09/29/2010 8:42:29 AM
Duration: 0:03:04

Backing up file "[Alpha Datastore] XPVM/XPVM-flat.vmdk"
BackupDisk failed
Client error: Failed to open VDDK disk [[Alpha Datastore] XPVM/XPVM.vmdk] ( is read-only mode - [true] )
Failed to open VMDK.
Logon attempt with parameters [VC/ESX: [MWNS-VC];Port: 443;Login: [contoso\administrator];VMX Spec: [moref=vm-550];Snapshot mor: [snapshot-562];Transports: [hotadd]] failed because of the following errors:

Server error: End of file

Everything worked incredibly easily and well when running Veeam from my physical server, there must be something I'm missing as to why I'm having to many difficulties running it from a VM... Any basic thoughts before I contact support and create an official support ticket?

[Gostev]

Hi Scott, does your virtual Veeam Backup server meet the System Requirements? Specifically, is it configured with 4 vCPU?

For reasons why hotadd may fail on specific VM, you can refer to this post if you want to troubleshoot this yourself. Contacting our support might be simpler alternative

[scottyyyc]

I only have the VM configured for 1vCPU. Didn't think 4 would be necessary. Either way, not a big deal. I'll reconfigure after hours this evening and report back.

Regarding the HotAdd failing, I've read over that post you linked several times. Nothing stands out as being an issue from that list.

[Gostev]

For your evening test, try backing up any other VM and it will likely work.

[scottyyyc]

Big time problems now. So I go to shut down the server 08 r2 VM (that veeam is installed on) so I can add some more vcpus. Everything is fine there, add the vcpus with no problems. I go to fire up the VM, and it starts where it is asking if you want to 'start windows normally, safe mode, etc'. OK...

Boots fine otherwise, login screen comes up. Log into domain (as domain admin), and now suddenly it's telling me it's lost its trust relationship to the domain...? WTF? So I go into vSphere to check things out and there's a Veeam snapshot present... Hmmm... WTF?

I guess I'll be spending my evening trying to figure out WTF is going on...

[scottyyyc]

So I have the machine back on the domain (good old netdom). That being said, it appears as though the root cause of this issue is that there was an orphaned Veeam snapshot hanging around, and for some reason the machine after my reboot/vCPU reconfiguration, caused it to revert to the snapshot (roughly 24 hours ago). That of course borked the domain trust, etc etc.

So the machine seems fine now (albeit lost the last 24 hours of logging in a particular program, not the end of the world), but this isn't confident inspiring. Are there widespread issues with orphaned snapshots handing around?

That said, now the VM has 4 vCPU's, so back to testing...

[scottyyyc]

Alright - update... SUCCESS!! Mostly.

With 4 vCPU's, network backup works like I expect. Decent speeds, no complaints. VA mode now works (for other servers, like you said Gostev), but not the original XP VM that I'm trying to backup. Any 08R2 VM that I back up is fine in VA mode... Something to do with XP? Same conditions otherwise (same datastore, host, vmware tools version, etc etc). Or is this a one-off error I should be contacting support directly for...?

That leaves me with a few last "best practice" questions:

1. My main file/print VM is over 1TB, so perhaps VA mode wont be the best bet after all...? Documentation says vmdk files over a 1TB are a concern for VA mode...
2. Any worries about backing up the VM that veeam is installed on?

[Vitaliy S.]

Hello Scott,

1. Yes, if you have disks larger than 1 then TB Virtual Appliance mode should be replaced either with SAN or Network job mode due to VMware limitation.

2. When using Veeam Backup to backup itself, you must use any processing mode other than Virtual appliance (or backup server will crash attaching its own disks twice). Also, you should have Veeam VSS disabled as otherwise the backup job will be timing out when VM freeze happens (because Veeam server is unable to process API calls being "frozen").

Release note on issues with using Virtual Appliance mode to backup Veeam server is described in our release notes:
http://www.veeam.com/files/release_note ... _notes.pdf

On top of that, you may want to try Veeam Monitor 5 which will be generally available next week to monitor abandoned snapshots. If those keep growing, they will obviously fill the datastore and thus affect other VMs residing on that datastore.

Thank you!

[Gostev]

Is it possible that failing Windows XP has IDE disks?

Are there widespread issues with orphaned snapshots handing around?

Not wide spread, plus version 4.1.2 added a few enhancements to prevent those from appearing.

[scottyyyc]

Ah ha! That all explains why the one problematic VM (the one that crashed and had orphaned snapshots) is the VM running Veeam. Every other VM except the XPSP3 VM was backing up fine.

Is it possible that failing Windows XP has IDE disks?

Just checked, and yes, it does indeed use IDE for its main disk. At least now I know what causes that issue. Luckily, the XP VM is just a test VM that doesn't need to be backed up regularly (albeit it good for testing Veeam).

I suppose then VA isn't the best choice for me, based on the fact I have a few large VMs, and need to back up the VM that Veeam is running on (as it hosts a few other important tools I use). I'll stick with [vStorage API] Network mode. Seemed to work really well in my tests anyways.

Thanks for your help guys. Being able to jump on the forums like this and get little annoyances solved quickly is a big help!

[Frosty]

Maybe you could just convert v2v and give the XP VM a SCSI disk?