Discussions specific to the VMware vSphere hypervisor
Post Reply
VladV
Expert
Posts: 223
Liked: 25 times
Joined: Apr 30, 2013 7:38 am
Full Name: Vlad Valeriu Velciu
Contact:

Replication failed Operation is not allowed in the currstate

Post by VladV » Oct 22, 2013 10:27 am

Case ID:00464246

The replication job for one of our VMs failed with the following error: Processing configuration error: The operation is not allowed in the current state.

The major problem is that the replication job corrupted the VMX file of the replica and so it went into an unkown state. Then the auto retry job started creating a new replica which I cancelled. I need to recover the replica so I can continue the replication process and maintain the restore points. We also need to restore important files from the server which coincided with the outage.

Our virtual environment is configured as follows: 1 vcenter, 2 esxi servers in a HA cluster and 1 esxi as target for the replication jobs. The replication target esxi is added in Veeam as standalone esxi although it is managed by the same vcenter.

Has anyone encountered this issue? How can I recover or repair the vmx for the replica?

foggy
Veeam Software
Posts: 17931
Liked: 1512 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Replication failed Operation is not allowed in the currs

Post by foggy » Oct 22, 2013 10:48 am

Vlad, basically, job retry should not create new replication but rather overwrite VMX file of the existing replica. Have you performed any actions over the replica VM prior to auto retry (failover, for example)?

VladV
Expert
Posts: 223
Liked: 25 times
Joined: Apr 30, 2013 7:38 am
Full Name: Vlad Valeriu Velciu
Contact:

Re: Replication failed Operation is not allowed in the currs

Post by VladV » Oct 22, 2013 11:12 am

No, no actions after the failed job.

It created another replica because it was not able to find the other one. The failed replica having had its vmx file corrupted dropped from inventory and leaving in place a vm named "Unknown". I cannot readd the replica in inventory because the vmx isn't recognized as being a configuration file.

I believe something is wrong in the file.

VladV
Expert
Posts: 223
Liked: 25 times
Joined: Apr 30, 2013 7:38 am
Full Name: Vlad Valeriu Velciu
Contact:

Re: Replication failed Operation is not allowed in the currs

Post by VladV » Oct 22, 2013 1:18 pm

I have found what made the vmx unable to to be imported:

First:

Code: Select all

                                                                                   d><OrigObjId>
instead of

Code: Select all

VeeamReplicaSummary = "<Summary><CreationDate>5246866293620377185</CreationDate><BackupId>ffc14772-6c34-44fe-b032-7cab8cab63e6</BackupId><JobName>Replication ServerET</JobName><DigestsRepositoryId>5b50b0b9-0177-4eaa-8d72-698403c4b61f</DigestsRepositoryI
The blank space was filled with <null> characters

Support has recovered the correct data from the veeam task log file.

Second:

Code: Select all

usb.autoConnect.device0 = "path:1/0/3 host:esx1.electrotel.local hostId:38\ b8\ d8\ f6\ 3c\ 8c\ e1\ 11-bd\ 1d\ 00\ 1e\ 67\ 45\ c9\ 5e autoclean:1 deviceType:remote-host"
was remaining in the vmx and this was making the VM unimportable.

Some questions still remain: 1. Why did this happen? 2. Why did Veeam leave the vmx file in an inconsistent state?

For question number 2. i think this is because the original vm has an USB hardlock connected to it and veeam didn't update the replica vm to exclude it when it failed.

LE: I have not yet imported the VM. I am waiting for support to get back to me to see if this is the way to go and to see if they need any logs to fix the problem so this does not happen again.

Vitaliy S.
Product Manager
Posts: 22697
Liked: 1498 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Replication failed Operation is not allowed in the currs

Post by Vitaliy S. » Oct 22, 2013 3:48 pm

1, 2 should be investigated by our support, I believe they will be able to answer it after reviewing your log files.
VladV wrote:i think this is because the original vm has an USB hardlock connected to it and veeam didn't update the replica vm to exclude it when it failed.
So your job failed two times, did I get it right? And the second time was with the error you have mentioned above, correct? I would appreciate if you could update this topic, once our support reviews all the details.

VladV
Expert
Posts: 223
Liked: 25 times
Joined: Apr 30, 2013 7:38 am
Full Name: Vlad Valeriu Velciu
Contact:

Re: Replication failed Operation is not allowed in the currs

Post by VladV » Oct 22, 2013 5:35 pm

The job only failed once. After failing the replica "transformed" into an Unknown(Invalid) VM because the replica VMX got corrupted. Then the auto-retry kicked in which created another replica because the other one wasn't accessible. When I opened the vmx file there where two inconsistencies:
1. the null characters in veeamreplicasummary line
2. the line with the usb

With support we were able to correct and fill in the blanks in the veeamreplicasummary line from the log file in veeam backup and replication but still we couldn't import the vm back into the infrastructure because the vmx wasn't recognized.

Only after finding and deleting the usb.autoconnect line the vmx permitted importing. I then reran the job and now I am waiting for the job to complete. It will take some time because it started "digesting" the disks which are very large.

It seems that veeam failed to update the replica vmx file (I don't know why) and left it in an inconsistent state. The fact that the original VM had an USB complicated things and made the replica invalid because Veeam hadn't finished its process (I think).

I will update the thread as information becomes available.

foggy
Veeam Software
Posts: 17931
Liked: 1512 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Replication failed Operation is not allowed in the currs

Post by foggy » Oct 23, 2013 5:26 am

Vlad, thanks for the detailed response, now the picture of what had actually happened is more clear. I believe support will be able to locate the reason of the job initial failure, since it is not expected, indeed.

VladV
Expert
Posts: 223
Liked: 25 times
Joined: Apr 30, 2013 7:38 am
Full Name: Vlad Valeriu Velciu
Contact:

Re: Replication failed Operation is not allowed in the currs

Post by VladV » Oct 25, 2013 6:02 am

I received a conclusion from support regarding this issue:
"I have found out that we have a bug when we need to cut out of VMX devices which couldn't be backed up and restored from the backup like USB disks, SCSI pass-through disks and etc. "

I cannot say I am happy with the resolution because I have be replicating the VM for about 5 months now without any problems using VBR 6.5 and 7, 10 times a day. I also powered on manually or ran surebackup for replicas on it without any issues. Only earlier this week the job failed at the point when it was processing the configuration.

Later this week after my last post, comparing the vmx files, I have come to the conclusion that the reason we could not import back the replica vm in vpshere was that this line :

Code: Select all

usb.autoConnect.device0 = "path:1/0/3 host:esx1.electrotel.local hostId:38\ b8\ d8\ f6\ 3c\ 8c\ e1\ 11-bd\ 1d\ 00\ 1e\ 67\ 45\ c9\ 5e autoclean:1 deviceType:remote-host"
was two times in the replica vmx. So it was not a problem that it existed (as I thought in the beginning) but because it existed on two lines.

My conclusion is that VBR, failing to process the config, was unable to correctly edit the replica vmx files, leaving 2 autoconnect lines and a lot of <null> charecters on VeeamReplicaSummary line. I don't believe that there is a bug in cutting out VMX devices because we managed to fix the replica vmx by looking at the job log file (task.vm-name.vm-180.log) and copy pasting what VBR tried to write in the replica vmx. VBR intentions where good but it was unable to write them.

I have sent my opinion to support and I hope they will continue investigating it. We rely on replicas because of the short RPO and coincidentally we needed to recover some files from 2 hours old restore point which I couldn't access because the vmx got corrupted.

Regards,
Vlad

foggy
Veeam Software
Posts: 17931
Liked: 1512 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Replication failed Operation is not allowed in the currs

Post by foggy » Oct 25, 2013 6:47 am

The good thing is that at least some adjacent bug was discovered within this case. ;) Thanks for taking your time to investigate that!

VladV
Expert
Posts: 223
Liked: 25 times
Joined: Apr 30, 2013 7:38 am
Full Name: Vlad Valeriu Velciu
Contact:

Re: Replication failed Operation is not allowed in the currs

Post by VladV » Oct 30, 2013 10:12 am

After having a web session with a support technician we have come the conclusion that there is no evidence of an network or software problem and we will monitor the job closely in the next few weeks.

I have a rather big problem though. The support technician edited the VM settings and changed the measurement unit of the 2TB VMDK from GB to TB. The VMDK was about 1.96TB and when he changed to TB vmware rounded to 2TB. The issue that I am facing now is that I am unable to create a snapshot on the VM because of this error: " Create virtual machine snapshot VIRTUALMACHINE File <unspecified filename> is larger than the maximum size supported by datastore '<unspecified datastore>'"

After reading http://kb.vmware.com/selfservice/micros ... Id=1012384 it seems that although the VMDK is OK on that datastore, the maximum possible size of the resulting snapshot would go over 2TB having a larger overhead.

I am now in the situation in which I cannot backup or replicate the VM.

I need help in finding a solution for reducing the provisioned space of the thin provisioned VMDK with about 100GB. Unfortunately punching hole in zeroed blocks or migrating to thick and then back to thin on another datastore only affects the occupied space not the provisioned space.

foggy
Veeam Software
Posts: 17931
Liked: 1512 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Replication failed Operation is not allowed in the currs

Post by foggy » Oct 30, 2013 10:32 am

Might be a silly question, but can you just change TB back to GB or, at least, decrease the disk size slightly so that it does not exceed the allowed size?

VladV
Expert
Posts: 223
Liked: 25 times
Joined: Apr 30, 2013 7:38 am
Full Name: Vlad Valeriu Velciu
Contact:

Re: Replication failed Operation is not allowed in the currs

Post by VladV » Oct 30, 2013 10:35 am

Unfortunately you cannot reduce the size of the VMDK. You can only increase it. In my case you cannot increase it either because it is already at max size for vSphere 5.1

foggy
Veeam Software
Posts: 17931
Liked: 1512 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Replication failed Operation is not allowed in the currs

Post by foggy » Oct 30, 2013 10:41 am

How about creating a new smaller virtual disk and copy the data over to it? This however will require some additional actions for the replication to continue and will remove all existing replica restore points.

VladV
Expert
Posts: 223
Liked: 25 times
Joined: Apr 30, 2013 7:38 am
Full Name: Vlad Valeriu Velciu
Contact:

Re: Replication failed Operation is not allowed in the currs

Post by VladV » Oct 30, 2013 10:48 am

In this moment I don't care about the replica. The thing is that this is a file server that hosts 1.8TB of files with folders and permissions which have to be carried and reconfigured on another partiton. It will be also a question of time taken to realize this movement and then the time to reconfigure the shares.

Vitaliy S.
Product Manager
Posts: 22697
Liked: 1498 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Replication failed Operation is not allowed in the currs

Post by Vitaliy S. » Oct 30, 2013 11:00 am

Let me chime in to your discussion.

I don't think there is any other way of fixing everything rather than the way proposed by foggy. On the other hand, I might be off base, but is replica VM still has the disk that can be snapshotted? Can you replicate it back?

VladV
Expert
Posts: 223
Liked: 25 times
Joined: Apr 30, 2013 7:38 am
Full Name: Vlad Valeriu Velciu
Contact:

Re: Replication failed Operation is not allowed in the currs

Post by VladV » Oct 30, 2013 11:04 am

Yes the replica has the disk that can be replicated but it is already 4 hours old.

Vitaliy S.
Product Manager
Posts: 22697
Liked: 1498 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Replication failed Operation is not allowed in the currs

Post by Vitaliy S. » Oct 30, 2013 11:08 am

But it still can bring you back to a "working" VM, right? ;) I'm just trying to help here. Let me search if it is possible to safely merge the changed data on the Guest OS level via some 3rd party tools.

VladV
Expert
Posts: 223
Liked: 25 times
Joined: Apr 30, 2013 7:38 am
Full Name: Vlad Valeriu Velciu
Contact:

Re: Replication failed Operation is not allowed in the currs

Post by VladV » Oct 30, 2013 11:23 am

The source VM is online and is working fine. I just can't back it up or replicate it. So I would prefer not revert to the beginning of the day only for this issue and try to fix it during off hours without losing data.

Vitaliy S.
Product Manager
Posts: 22697
Liked: 1498 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Replication failed Operation is not allowed in the currs

Post by Vitaliy S. » Oct 30, 2013 9:49 pm

Ok, makes sense. BTW, I have just googled and it seems like one of the ways of fixing it is to use this KB article that explains how to shrink your virtual disk a bit and allow snapshot operation. Hope this helps!

VladV
Expert
Posts: 223
Liked: 25 times
Joined: Apr 30, 2013 7:38 am
Full Name: Vlad Valeriu Velciu
Contact:

Re: Replication failed Operation is not allowed in the currs

Post by VladV » Oct 31, 2013 1:22 am

First of all thank you Vitaliy and Alexander for your help. Finally we went with the disk partition copy method using a partition manager software.

The reason for not using the Vmware Converter was simple. Due to the fact the the converter clones the disk at the file level when shrinking we would have obtained an extremely low transfer rate (3mil files). Also, when shrinking disks, the converter cannot sync the changes after cloning, so this would have forced us to clone it offline. A partition manager, although offline, copies the partition at the cluster level thus we managed to have a transfer rate of 270MB/s (read+write) and finished the task in 5 hours.

After cloning we booted the machine, changed the drive letter, restored the reg file that contains the shares (because they are lost once you remove the old disk [the permissions still remain though] )and we had our VM back in shape after a total of 7 hours (3:00AM here). Now we need to run a full active backup after resetting the CBT and then recreate the replication job. This stinks so much when you have a large VM.

Now, I don’t want to exaggerate or overreact because of this issue, but the support team needs to be better trained when interacting with production VMs. First of all NEVER press OK when you have no intention of making any changes and there is a Cancel button. Second, given that Veeam works only with a virtual environment they need to know that a 2TB (2048GB) disk cannot be snapshot-ed. Third, be cautious around production VMs and inform the client of your intentions.

The issue can be easily replicated if desired: just create a 1960GB thin VMDK (this was the vmdk sized before the incident), or a similar sized one, save the configuration, reopen the vm config and change the measurement unit from TB to GB and back again and you will see that it is rounded to 2TB. Once it’s 2TB you can say goodbye to snapshots.

Hope this helps other users if they incur this vmdk behavior.

Vitaliy S.
Product Manager
Posts: 22697
Liked: 1498 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Replication failed Operation is not allowed in the currs

Post by Vitaliy S. » Oct 31, 2013 8:28 am

Vlad, good to know that you're up and running after all these troubles you've run into. I have passed your feedback to our support director, he will look through your support case and this situation. Sorry for inconvenience.

bdufour
Expert
Posts: 198
Liked: 29 times
Joined: Nov 01, 2017 8:52 pm
Full Name: blake dufour
Contact:

Re: Replication failed Operation is not allowed in the currs

Post by bdufour » Sep 06, 2018 3:27 pm 3 people like this post

i know this old, but i received this error today while trying to replicate a vm while taking the defaults in the new vm machine creation task. the veeam kb and vmware kb didnt solve the issue in my case. the issue for me was related to the the default compatibility option of 6.5 (on vcenter 6.5), but the host im replicating to is 5.5. the error is very generic and there was a bit of over site on my part when creating the new vm. once i created the vm with compatibility 5.5, everything worked as expected and replicated. hope this helps someone, as this error is very common.

Post Reply

Who is online

Users browsing this forum: JHarman and 29 guests