V11 - Install CDP I/O filter - Operation is not allowed in the current state

[veremin]

Thanks, Alan, for updating the thread with the resolution found; appreciated!

[rweis]

Real life tip: IO Filter installation failed due to DNS resolution, but the issue wasn't immediately apparent. The vCenter Appliance was named at time of deployment with only a short host name, not a FQDN. There is no way to add a domain suffix search order or point the appliance to a lookup zone (secondary, forwarding) as near as could be determined. The appliance was renamed with a FQDN. The resolution of the short names to FQDN in the cluster then worked correctly, and installation was successful.

[nunciate]

I ended up fixing this as it was a network block between my primary backup server and VCenter. We fully isolate our backup network and only allow very specific ports through.
I can't tell you what port was opened but my network security team did identify something being blocked and after they allowed that it worked fine for me.

[gtelnet]

Had the same exact issue and nunciate's first post helped find the issue immediately. Our BNR is also on a locked down network so we had to open port 33034 from the Veeam BNR server to the vCenter and hosts and all is working. Thank you for posting your fix!

I figured it out Finally. No need for support.
It turns out my DNS was fine. In fact, the issue was that we have our Backup VLAN locked down and the vCenter was being blocked from coming into the Veeam Server on port 33034.
Once we opened that port up it was fine.

I found this information in the log file C:\Users\All Users\Veeam\Backup\Utils\I_O_filter_deployment.log

Code: Select all

[21.04.2021 14:14:04] <19> Info         [Soap] InstallIoFilter, vibUrl 'https://veamserver.domain.com:33034/dapi/bundle/6.5.0/11.0.358', clusterComputeResourceRef 'domain-c237803'
[21.04.2021 14:14:04] <19> Info         [Soap] QueryClusterIoFilterInfo, clusterComputeResourceRef 'domain-c237803', vendorName 'VEE'
[21.04.2021 14:14:04] <19> Error        [IOFilter] Failed to log filter issues: filter not installed
[21.04.2021 14:14:04] <19> Error        [IOFilter] 
[21.04.2021 14:14:04] <19> Error        Failed to install CDP components via cluster API for EVC (System.Exception)

[RubinCompServ]

Having said that, literally 9 out of 10 times this error is caused by the DNS issues in the environment. So focus on this part really hard to ensure everything can resolve everything

Count me as another "it was a DNS issue". The Veeam server didn't have an entry in the DNS server that the VMware environment was using for resolution; as soon as I added, I was able to install the filters. Thanks Gostev and veremin!

[veremin]

You are welcome, David, thank for coming back and confirming that if there is an exception to "it's always DNS" rule, it is a rare one

[isan]

I have been trying to install I/O filters for the CDP feature that veeam offers, but it always throws out an error.

Error Installing CDP components on cluster: Office Cluster... Error: The operation is not allowed in the current state. The operation is not allowed in the current state.
Error Failed to install I/O filters: Failed to install I/O filter on cluster: Office Cluster
Error Failed to perform CDP components deployment Error: The operation is not allowed in the current state.

I do have a case open with veeam (Case #04914632) as well and haven't got a solution. Has anyone faced the same or a similar error or perhaps knows how to solve it?

[Natalia Lupacheva]

Hi Ishaan,

moved your post to the existing thread, please take a look at the discussions here.
As for your Support case, I see you have quite an active discussion and the latest message was sent yesrday. Anyway, if you are unhappe with your case, you can always escalate it.

Thanks!

[RubinCompServ]

I have been trying to install I/O filters for the CDP feature that veeam offers, but it always throws out an error.

Error Installing CDP components on cluster: Office Cluster... Error: The operation is not allowed in the current state. The operation is not allowed in the current state.
Error Failed to install I/O filters: Failed to install I/O filter on cluster: Office Cluster
Error Failed to perform CDP components deployment Error: The operation is not allowed in the current state.

I do have a case open with veeam (Case #04914632) as well and haven't got a solution. Has anyone faced the same or a similar error or perhaps knows how to solve it?

I assume that "Office Cluster" is the name of your VMware cluster? Are any of the hosts in Maintenance Mode? Make sure that your vCenter AND hosts can reach your VBR by FQDN, not just IP (obviously VBR has to be able to access the vCenter and hosts by name as well).

[Ciso_2021]

i remember me having this issue with Time different between the hosts.
check that too.

[mcz]

Just wannted to mention my experiences with a failed CDP-driver installation: In our case we had no active firewall-policy that would allow the connection FROM the vcenter and the hosts to the veeam components. Just realized that later when I checked the "used ports" section in the helpcenter.

It would be nice if that situation would be mentioned somewhere in the logs or the console - the current errormessage is very misleading and doesn't point to communication issues between vmware- and veeam-components. I know, I should have read the articles first, but even the support engineer first wannted to know about DNS and vcenter certificate allthough I have explicitly asked about the tcp-communication between the components. You see it's even misleading for them...

Thanks!

[Gostev]

That's the main problem: I/O filter deployment is a black box for us as well. And all we get back from this black box is that misleading error message which gives us no idea whatsoever as to what was the reason for the failure.

[mcz]

Oh I see, you don't get more from the API... Maybe some pre-execution steps/checks could be done by veeam itself to give the user a hint - for instance when the communication is not successful (due to firewall restrictions, whatever) or if there's a dns misconfiguration... Probably it would decrease the support cases as well...

Thanks!

[Gostev]

It would be nice indeed but unfortunately we can't really do things on behalf of ESXi e.g. have it check outbound communication to some IP:port, have it resolve some DNS name etc... it is a closed system with very limited API and these capabilities are just not there (and unlikely to be added for security reasons).

Naturally, the only logical solution is for VMware to start returning meaningful errors instead of a single error for any situation. We will keep providing them this feedback.

[mcz]

Makes sense, thanks Anton!

[jtupeck]

Have been working through this with my former employer, now a customer and ran into some port issues as well, as their backup network is segmented off from normal production. A previous post (last on page 1 of this thread) stated that vCenter to Veeam Server communication also needed port 33034, so we added that allowance in the ACL for the backup network, from both vCenters in play (prod and DR) to the VBR server. CDP filter installs were still unsuccessful and the vCenter logs showed

Code: Select all

faultCause = com.vmware.vim.binding.vmodl.MethodFault: "com.vmware.eam.EamIOException: Failure loading VIB package" caused by "java.net.NoRouteToHostException: No route to host (Host unreachable)"

We had already made provisions for ICMP (so DNS could function properly in all directions) and tested successfully, so this led us back to the ACL side of things. In checking the logs there, we saw a random high port (I do not know which, as a network team member was relaying the info) that was being denied. So we opened the ACL entry to allow vCenter_IP to VBR_IP without port restrictions and retried the install. It was successful after this change. Veeam's documentation (https://helpcenter.veeam.com/docs/backu ... onnections) about required CDP ports does not call out any of the vCenter ports needed, so this thread was instrumental in helping us figure this out. It might be good to get the document updated to include vCenter connection port requirements.

Additionally, we had to work through some vCenter EAM certificate issues, which I was able to complete from this VMware KB: https://kb.vmware.com/s/article/2112577 ... _appliance

Wanted to update this thread with my findings. Hopefully it's helpful to someone out there!

[veremin]

CDP activity does not require any specific port open on vCenter server, only on the specific hosts where daemons reside.

May I ask you whether this issue has been investigated within the support ticket? We would like to check debug logs and see what the real root cause was.

Or may you confirm what exact port was opened in your case?

Thanks!

[jtupeck]

Not a port on vCenter, a port on the VBR server being accessed BY vCenter. When vCenter was only allowed ICMP and 33034 access to VBR, it did not work. We observed a high port that was being blocked at the ACL level. Once we opened IP>IP access without port restriction from vCenter TO VBR, the install worked fine.

[veremin]

Understood, so we are back to the question what exact port it was, without this information we cannot say whether it's already mentioned in the documentation or not. May you grasp this information from your network team? Wasn't it 10443, by any chance? Thanks!

[jtupeck]

I see what you’re saying now… we were looking at the ports in the documentation for CDP (linked above) and vCenter wasn’t listed there. I don’t think 10443 was ever allowed prior to this work, as it’s called out for use from VBR to vCenter and not the other way around. Because the port listing for 10443 is listed as Backup Server source and vCenter as a target and there there is no ACL blocking access to the vCenter VLAN, we wouldn’t even have thought to add that in the reverse direction. If 10443 is needed for access with vCenter source>VBR target, then that should be called out, no? Maybe I missing that on the “used ports” page? I did a search and didn’t see anything, so please correct me if I’m wrong.

I can ask the network team at my customer’s office what port was observed being blocked, but they referenced ‘high port’ which generally refers to 49152-65535 dynamic ports. I’ll see what I can find out from them and update as necessary.

[veremin]

It would be really appreciated, Jason, if you could collect this information from the customer. Having this information would allow us to correct the documentation, should need be.

[jtupeck]

Will do. We have another environment where we need to deploy this in the future and I will either get the added info then, or we will fiddle with the ACL and monitoring when one of their team has a bit more time available. Thanks, veremin!

[AlexandreD]

Hello,

I'm totally agree that it's always a DNS issue.
But how can we resolve this if VBR is not part of a domain ? And more generally, the DR site and the production site is not in the same DNS zone.

For example, I have install a fresh new VBR server (version 12.2), vbr server is not part of a domain, so I create a A entry on the DNS, for example "vbr.dr-region.local".

All Veeam and VMware components have the same DNS with two configured zones, dr-region.local and prod-region.local.

When i tried to install I/O filter on the disaster recovery vcenter called vcsa.dr-region.local, everything is ok. But when i try to install I/O filter on the production site, vcsa.prod-region.local, I have the error "Operation is not allowed".

According to logs, i think source esxi and source vcenter try in return to resolve http://vbr:33035 and not http://vbr.dr-region.local:33035
So i have to modify the etc/hosts on vcenter and ESXi... not fan of this, especially if we have to modify hosts files with new CDP proxy, or anything else later, some mistakes are possible, we have around 70 esxi host file to modify, and these are just two clusters among many others.

Do you think there are others solutions ?

thank you

Alexandre

[AlexandreD]

Hello again,

I opened a case # 07424000

As expected, It's confirm that it is a DNS issue. But as i said, ESXi and vCenter try to join name "vbr" and not "vbr.domain.local"

Modify the hosts file is the only solution when vbr is not part of a domain ?

thank you

Alexandre

[veremin]

It seems there is a small misunderstanding. In general, the backup server itself can be outside the domain where the vCenter server, hosts, and other components are located. However, it is important to note that the DNS server must be able to resolve the name of the backup server.

One way to achieve this (besides correcting hosts file) is to access the backup server, obtain its name, access the DNS server, create a separate zone there, and then within that zone, add an A record where you explicitly specify the name of the backup server and its IP address. After doing so, the DNS server will be able to resolve the name of the backup server to the specified IP address, even if the backup server is located outside the domain.

Thanks!