Replication: snapshot issues

[rb51]

Case ID: 5204652
Hi all
Sorry for the lengthy post…

We are working with a cloud provider for our DR environment. Basically they setup the latest veeam 6.1 on their end (with a proxy VM) and a proxy VM on our end. I have no control of Veeam. We were replicating on a 15min interval.

One of the VM we were replicating got problems with snapshotting and I was a bit slow to detect it, even though I run a snapshot report daily on our VMware infrastructure.

Basically the VM now has 40 delta files, so I am not sure whether I have to give V2V a go and hope for the best. Last successful Symantec backup Tue 10Jul

Environment
vSphere 4.1U2
VM: Win2003R2 x64 with 2 vmdks attached (same datastore)
Our backup is done with Symantec Backup Exec 2010 R3

Steps tried so far:
1) vMotion VM to another host and back original host;
2) Immediately tried to take manual snapshot (failed)
3) Shutdown all Veeam VMs on both sides then tried to take manual snapshot (succeeded); however when trying to ‘delete all’ it fails
4) By checking vmware.log of VM it seems that problem is with 00004-delta.vmdk and 00040-delta.vmdk files which seem to be locked
5) Following VMware KB10051, the owner is all zeros which means that it is not the host locking files and possibly Veeam or Symantec?
6) Following VMware KB10051, lsof | grep <locked file name> does not return anything
7) Tried to clone VM just in case but it failed

Resources used so far:
VMware KB10051 - http://kb.vmware.com/selfservice/micros ... alId=10051
http://www.robvanhamersveld.nl/2012/04/ ... -snapshot/

Our cloud provider logged a case (5204652) yesterday with Veeam and let my details for direct contact, but still haven’t heard from them.

Case logged with VMware which advised me to deal with Veeam first and if not resolved to come back to start troubleshooting.

I wonder whether you guys could suggest something for me to have a look at.

Is there any conflict using Symantec backup exec on our infrastructure and Veeam for replication????

Thanks in advance

[Vitaliy S.]

Hi Ricardo,

Restarting ESX(i) host management agents might release the locks which are preventing you from deleting "00004-delta.vmdk" and "00040-delta.vmdk" files. I would give it a try, after that the "Delete All" operation should complete successfully.

Is there any conflict using Symantec backup exec on our infrastructure and Veeam for replication????

Do you use both solutions to backup your VMs? I guess there might be some issues if you attempt to backup the same VM at the same time, in all other cases there should be no conflict.

Thanks!

[rb51]

Hi Vitaliy,
thanks for your reply.
restarting the agents did NOT solve the issue. So still need help to troubleshoot this problem.

We use only Symantec BE 2010 R3 for backing up, but I want to try Veeam and if happy make a business case to replace Symantec.

Could you kindly provide further assistance?

Thanks, ric

[Vitaliy S.]

We use only Symantec BE 2010 R3 for backing up

Could Symantec BE be the reason for the locked snapshot files? Have you tried restarting BE services and committing the snapshot tree again?

Could you kindly provide further assistance?

I would love to do that, but, unfortunately it is impossible to troubleshoot this issue via forum, without webex and full debug logs review. Please continue working with our and VMware support team on this issue.

[rb51]

Could Symantec BE be the reason for the locked snapshot files? Have you tried restarting BE services and committing the snapshot tree again?

Tried that but to no avail

I would love to do that, but, unfortunately it is impossible to troubleshoot this issue via forum, without webex and full debug logs review. Please continue working with our and VMware support team on this issue.

I have already uploaded logs to VMware and I am waiting them to contact back so as Veeam.

Thanks, ric

[rb51]

Just a quick update to say that issue has been fixed with VMware (maybe it is still worth getting Veeam logs investigated).

snapshopt file was being locked by the host (strangely on an used vnic). Host was rebooted and kept into maitenance mode until I got the snapshot files committed.

basically this procedure was followed:

1. SSH session onto the host or VMA
2. change directory to VM path
3. grep -i scsi.*filename *.vmx to find out current active vmdk(s) -- or just grab vmx file on a notepad
4. vmkfstools -D <vmdk(s) name> from step 3 (if the file is locked it should return mode1, mode2 or mode3 (see KB1003397 for explanation)
5. once you identify the owner by its MAC just confirm whether it belongs to the current host: ifconfig | grep <last digits of MAC>
6. put host into maintenance mode, make sure all VMs have been migrated, reboot and leave host in maintenance mode
7. SSH onto the host which the affected VM has been migrated to
8. re-check lock status: vmkfstools -D <vmdk(s) name> from step 3
9. if lock is mode0, then you might be able to commit snapshots
10. change directory to VM dir and vmware-cmd *.vmx removesnapshots (the time to complete this will depend on how much data needs to be committed). You do not see its progress via VIC, so you need to wait until the output removesnapshots()=1 appears
11. check the VM file structure by ls -lthr *.vmdk It should be nice and clean

Just wanted to update this in case someone experiences the same.

HTH

Thanks for the help.
ricardo

[Vitaliy S.]

Thank you for coming back and posting your resolution.