V11 - Install CDP I/O filter - Operation is not allowed in the current state

[Stabz]

Hello Guys,

I m trying to install Veeam CDP on my lab.

Unfortunately I get this errors:

Code: Select all

17/02/2021 12:15:55 Error    Installing VeeamCDP components on cluster: Labs-Cluster-DATA-DRP... Error: the operation is not allowed in the current state
17/02/2021 12:15:55 Error    Failed to install VeeamCDP I/O filters: Failed to install VeeamCDP I/O filter on cluster: Labs-Cluster-DATA-DRP

The acceptance levels of hosts is set to CommunitySupported.

Is there any other prerequisites ?

Thanks!

[Gostev]

Hello, Philippe

Setting VIB acceptance level of the hosts to Community Supported is not required, since our I/O filter is certified by VMware and signed.

This error is coming from the only non-Veeam component in the picture: the VIB deployment framework provided by VMware which we are forced to use. And it is really bad in terms of error handling: the above is basically the "universal" error that the framework returns for any environmental issue it runs into. Every unhandled exception falls through to this one handler, which in turn has only one thing to say

Having said that, literally 9 out of 10 times this error is caused by the DNS issues in the environment. So focus on this part really hard to ensure everything can resolve everything, as it's almost for sure the issue given you're deploying into a lab, and not into production environment. Another (much more rare) issue to check for would be if vCenter EAM certificate has expired.

But in any case, it will always be due to an infrastructure issue in your vSphere deployment, and VMware support would be better equipped to troubleshoot those. This does not mean you cannot open a support case with Veeam, but rather that it will take longer for our support engineers to understand an issue with your vSphere infrastructure.

Thanks!

[Stabz]

Hello Gostev,

Thanks for you quick answer.
Good to know for the Vib acceptance level !

Yes this error don't really help for debug.
I ll take a look to the DNS part but everything works correctly (Backup, Surebackup, Replication...) so I m a bit worried .
The DNS entries for my vCenter and ESX are defined in the local hosts file. My VBR can ping without problems this elements.

Lets see.

[veremin]

Check whether:

* ESXi hosts, vCenter server, CDP proxies and backup server are able to resolve DNS names of each other
* vCenter EAM certificate has not expired

If this does not help, open a ticket and provide debug logs, including logs from vCenter server.

Thanks!

[Skyview]

Having this error also.

Stood up a v6.7u3 environment. 2 hosts, no shared storage, just independent hosts. Stood up a 6.7 VCenter Server and added both.

vcenter server is using just an ip for FQDN. Hosts added in by ip.

Brand new win2019 server vm stood up and v11 RTM installed. All clean fresh installs everywhere.

[Skyview]

Update-

Just as Gostev said, it was DNS.

It's always DNS.

Because the lab was setup without FQDNs and just using IPs, DNS resolution of the FQDN inherent in the host (ex: testhost1.bogusdomain.com) didn't work.

I was able to get it working in the lab by making host file entries in the hosts as well as the Vcenter server, as detailed here:
https://kb.vmware.com/s/article/1020302

[Stabz]

Hello Guys,

Sorry for the delay, unfortunately I did'nt work on this problem for the moment.

Skyview thank you for your feedback, my lab environnement is a bit messy so I ll check asap the DNS configuration for the vCenter and the Hosts.

[liorme]

when trying to install i/o filters with vcsa 6.7 and esx6.7, i get this error : the operation is not allowed in the current state

i tried both access by ip and by dns (with resolve by hosts file), no change.

any ideas?

[tsightler]

Remember that the ESXi hosts themselves also need to be able to resolve hostnames, and reach VBR server via the resolved IP. I see a lot of people where it works the other way, Veeam and vCenter can resolve ESXi hosts, but ESXi hosts cannot resolve Veeam/vCenter server IP which is a critical for successful deployment.

[nunciate]

I am getting this error also. I have a dev backup server but it is pointed at my DR and Prod Vmware environments. Both vCenters report the same error. All systems involved are correctly registered in DNS however my vCenters have 2 names. The base server name and a CNAME called VCENTER-TX and VCENTER-CO. I have tried registering both names in my dev server but still the same errors.

I realized we had upgraded our vCenter to 7.0 but our esxi hosts are all still running 6.5. Not sure if that could be the problem. We can't upgrade our hosts in our DR site to 7.0 because they are older DL380 G8 hardware.

Any ideas? How would I know if my vCenter EAM certificate has expired?

[veremin]

Check the KB article that Anton has referenced above. Also, it might be time to open a support ticket. Thanks!

[netscaper71]

Hello, we also have such issue. But it strange because I have tried with vCenter that have correct DNS and FQDN for vCenter and ESXi hosts then I think that problems is in Cluster (HA, DRS or EVC) no without or with selected options error is same. Then I tried with 7.0.2 - officially unsupported version of VMware. Then I go to the 3rd infrastructure correct working fresh installed (less that 6 months ago) recently upgraded to 7.0.1 - supported by Veeam 11 and cannot install I/O filters for CDP. It is strange because almost same infrastructure in our VCC environment I/O filters was installed without problems.
Veeam Case # 04712275

[maol]

Hello,
same error here, no dns issue... just a cluster with a single esxi node 6.7.0, 17098360
vcenter server is 7.0... anyone resolved this already?
L

Updated the ESXi node to 7.0 nothing changed, opened Veeam Support - Case # 04722129

[veremin]

Aleksandr, Lorenzo, thanks for the tickets provided, I've asked QA team to follow the investigation. So, if I have more information about the issue, I will update the topic. Thanks!

[veremin]

Guys, we've checked the case and failed to find the logs from the vCenter server. Please, attach those to the ticket. Thanks!

[maol]

Tks @veremin, attached vcenter logs to the case, also opened a parallel case in vmware.
l

[netscaper71]

Hello, Thank You.I have already uploaded logs to the case related to logs from Veeam. I will wait with case to VMware because as I mentioned earlier in our Veeam Cloud Connect vCenter it works but I cannot find out why it is not working in our customers side.

[veremin]

Aleksandr, logs from vCenter server are typically required to identify the source of "Operation is not allowed" issue, so, without them neither support nor QA team can be of any help here. Thanks!

[maol]

Hello Veremin,
let me know if the vcenter logs I've uploaded are good.
in the meanwhile I can add that the VIB installation itself is working well on the non working hosts:

Code: Select all

The ESXi Shell can be disabled by an administrative user. See the
vSphere Security documentation for more information.
[root@JITT**************10:~] cd /tmp/
[root@JITT**************10:/tmp] esxcli software vib install -d /tmp/veecdp-offline-bundle.zip


Installation Result
   Message: Operation finished successfully.
   Reboot Required: false
   VIBs Installed: VEE_bootbank_veecdp_11.0.358-1OEM.650.0.0.4598673
   VIBs Removed:
   VIBs Skipped:

but also if I install manually the VIB the veeam wizard fails to configure it (with the same error), so i think is probably vcenter related.
Lorenzo

[veremin]

Logs seems good, I will let you know, if we need anything else from your side. Thanks!

[netscaper71]

Aleksandr, logs from vCenter server are typically required to identify the source of "Operation is not allowed" issue, so, without them neither support nor QA team can be of any help here. Thanks!

Sorry for my bad English, I want to write that logs from vCenter are collected and uploaded from same infrastructure where from was earlier collected Veeam logs to have full picture of situation and for easier investigation because of action timeline, because I have tried install I/O filters on 3 different customers infrastructure.

[veremin]

Got it, I've passed the information to QA team, will see what they say after the logs investigation. Thanks!

[netscaper71]

Thanks to Veeam support. Problem was that EAM want to reach Veeam server by hostname but DNS is not pointing to it, modified VCSA hosts file (added IP/hostname) and I/O filters are installed. Resolution correct DNS records for Veeam server.

[Gostev]

It's always DNS

[veremin]

Case # 04722129

The logs suggest that there might be a network issue:

Code: Select all

java.net.SocketTimeoutException: connect timed out

So, might be worth checking it out, unless it's not a DNS issue (like it always is) or support team have already reached a conclusion.

Thanks!

[maol]

Ok, solved this one, it was a routing problem, that specific veeam server have two NIC, and the VCSA missed the route to one of them.
thanks to all for your help!

[veremin]

This explains the connection timeout - thanks for coming back and updating the topic with the solution, appreciated!

[jeroen_st]

Can confirm it's DNS:
On backup server:

Code: Select all

nslookup esxi -> 10.0.0.1

From ESXi host console:

Code: Select all

nslookup veeam -> NXdomain

Install failed


after fixing DNS (either use the above mentioned hostfile, or fix it with a static DNS record in local DNS server)
On backup server:

Code: Select all

nslookup esxi -> 10.0.0.1

From ESXi host console:

Code: Select all

nslookup veeam -> 10.0.0.100

Installation succes

[nunciate]

Well, crud. I have this issue after upgrading to v11 today. I was hoping it would work but it didn't.

I noticed I had 2 DNS names for my VCenter. One is the base server name we run the virtual appliance on. The other was a DNS CName we set up and that is what I have registered in the Veeam console. It is just called VCENTER-TX.

I went ahead and removed the CName and added it back as just an A record.
NSlookup vcenter-tx returns the correct IP from the Backup Server.
NSLookup of the Backup server from the hosts returns the correct IP for the backup server.

I still have the error though. I tried registering my vCenter server by its server name in the Veeam console and it still didn't like that either.

Can't think of anything else to try.

Case #04768120 submitted

[nunciate]

I figured it out Finally. No need for support.
It turns out my DNS was fine. In fact, the issue was that we have our Backup VLAN locked down and the vCenter was being blocked from coming into the Veeam Server on port 33034.
Once we opened that port up it was fine.

I found this information in the log file C:\Users\All Users\Veeam\Backup\Utils\I_O_filter_deployment.log

Code: Select all

[21.04.2021 14:14:04] <19> Info         [Soap] InstallIoFilter, vibUrl 'https://veamserver.domain.com:33034/dapi/bundle/6.5.0/11.0.358', clusterComputeResourceRef 'domain-c237803'
[21.04.2021 14:14:04] <19> Info         [Soap] QueryClusterIoFilterInfo, clusterComputeResourceRef 'domain-c237803', vendorName 'VEE'
[21.04.2021 14:14:04] <19> Error        [IOFilter] Failed to log filter issues: filter not installed
[21.04.2021 14:14:04] <19> Error        [IOFilter] 
[21.04.2021 14:14:04] <19> Error        Failed to install CDP components via cluster API for EVC (System.Exception)