Backup Fails when disks processed in different modes

[kjc3303]

Hi All,

got an issue here when I try to back up a VM with multiple Disks in different processing modes

Disk 1 40gb processed in SAN mode processing fine
Disk 2 40gb processed in SAN mode processing fine
Disk 3 400gb processed in hotadd error below

29/09/2014 09:37:08 :: Processing '**********)' Error: Client error: Failed to open VDDK disk [[lun2********o] ********)/*********] ( is read-only mode - [true] )
Failed to open VMDK.
Logon attempt with parameters [VC/ESX: [********];Port: 443;Login: [****];VMX Spec: [moref=vm-50284];Snapshot mor: [snapshot-51233];Transports: [hotadd];Read Only: [true]] failed because of the following errors:

It does not seem to be disk specific as in another attempt disk 2 was SAN mode and disk 1 was hotadd mode and the same error againts disk 1 was given

Troubleshooting so far

Added the disk to the hotadd proxy manually - OK
Backed up the VM excluding the 2 x 40 gb HD using Hotadd - OK
Backed up using single Processing Mode (hotadd and SAN)
Recreate backup with just this single VM (was initially in a backup with other VMs)

Seems to be only when all 3 disks are processed in parallel using multiple modes

Will add support number soon

Kev

[kjc3303]

More info

I have migrated the VM to the same host as the proxy and it will process all 3 disks at the same same time OK, 2x SAN & 1x hotadd

When I migrate it back I get the same error and fails

Kev

[Vitaliy S.]

Hi Kev,

What version of Veeam B&R are you at? If proxy server is located on the same storage/host as the backed up VM, it should not fail when using hotadd mode for the 3rd disk. Let's see what our support team finds out after reviewing the debug logs.

Thanks!

[kjc3303]

Hi Vitaliy,

Version 7.0.0.87.1 - Latest version I belive? This is a new install as I am moving my Veeam installation to a physical machine.
Yes the Proxy is on the same Storage (even has disks on the same datastore) but a different host than the backed up VM.
As mention above when I move to the same Host it backs up fine.

I have tried another proxy for troubleshooting purposes but show the same errors.

Support Case number #00644138

Thanks Kev

[Vitaliy S.]

I have just talked to our QC team regarding the behavior you observe, and it doesn't seem to be a known issue, so logs deep dive is required here. On a side note, how many hosts do you have? Does it matter which host your proxy server resides when it fails to use hotadd mode? Can you try to migrate to another host (not local to backed up VMs and not the one it is currently located at) and try to run the job?

[kjc3303]

Only two in this cluster I will swap the vm's over now and try, I'll feed back here shortly

I will test the following

Host 1 = vm
Host2 = Proxy

Host1 = proxy
Host2 = vm

Host1 = proxy
Host1 = vm

Host2 = proxy
Host2 = vm

Thanks

Kev

[kjc3303]

Results of test

Host 1 = vm
Host2 = Proxy FAIL

Host1 = proxy
Host2 = vm FAIL

Host1 = proxy
Host1 = vm SUCCESS

Host2 = proxy
Host2 = vm SUCCESS

So it seems the issue lies when the vm and proxy are on different hosts

Thanks

Kev

[Vitaliy S.]

Thanks for the running these tests, I have run out of ideas on what might be causing this. Let's see what our support team finds out by looking through the log files.

[kjc3303]

Cheers Vitaliy will they see this info? or need it?

Kev

[Vitaliy S.]

Send a link to this post in order to save time when describing your issue and please update this thread with the case ID. Thanks!

[kjc3303]

#00644138

[kjc3303]

Update, had a webex with Sergey from Veeam - looks like the issue is something to do with hot add trying to mount the disk which is already being processed by SAN mode.

Logs been sent to Veeam for further investigation.

Not really a big issue for me as I can split the jobs as a workaround, still nice to see Veeam taking this seriously - will post an update when I receive one

Kev

[kjc3303]

Hooray this has resolved!!!! Thanks Sergey @ Veeam support.

The resolution was to reset CBT on the VM. Not entirly sure what the underlying issue was but resetting CBT did the trick

Kev