Backups fail. The specified transports are not available

[SeanD]

Hi,

I am running B&R v7.0.0.871.

All m backups are now failing with a: Client error: The specified transports are not available. Available transports: [nbd]. Specified transports: [san]. VMDK path:[san path to vmdk]

Any ideas what to look at?

Thanks,

Sean

[foggy]

Sean, looks like you've originally configured your jobs to run in direct SAN mode and disabled failover to network mode, but for some reason direct SAN transport became unavailable (were there probably any changes in your environment that could cause this?). I'd check whether your storage is still correctly presented to the proxy server and contact support for further investigation, if required.

[SeanD]

I have transport mode set to Automatic selection and have never changed it. Our SAN admin has verified that nothing has changed in regards to lun mappings. I went into the disk mamager on the server and I see all the disks there. Is there a compability problem with v7.0.0.871 and the latest vCenter 5.5? I now there were some changes I think to SSL to fix a serious SSL vulnerability, so just wondering if this broke Veeam?

I can't talk to support, as we no longer have an active support contract. That is why we are still on v7

[dellock6]

vSphere 5.5 is supported on Veeam 7.0.0.771 (R2a), so your version should be good, but with the general 5.5 release, I don't have additional informations about latest vSphere patches as I moved my labs to v8 when it came out.
The SSL changes in vSphere 5.5 U3 indeed affected Veeam, but in order to rule out this problem you can re-enable SSL and see if the error is still there or not. We have a KB article for it that you can try (https://www.veeam.com/kb2063) however the error should be related to NFC connections if the problem is SSL...

[SeanD]

In disk management on the Veeam server. Should the SAN drives be offline? I can't remember the normal state. I ran a test backup allowing the proxy to drop to network backup and it worked, so it has to do with the SAN connectivity.

[dellock6]

Yes when installed Veeam sets the SAN policy to remote-offline so that VMFS volumes are not mounted by mistake by an admin once they are visible in disk management.

[SeanD]

So does anyone have an idea on how to fix this problem?

[emachabert]

I suppose you did it but I ask :
- Did you try to rescan the disks in the dis management console ?
- Did you try a reboot of the Windows host ?
- did you check that everything is ok regarding the multipathing ?

[Vitaliy S.]

In addition to Eric's advice, I would try to dismount all the LUNs and then connect them again. BTW, check if ACLs were changed on the SAN device or not, seen this as a reason in a couple of similar cases.

[SeanD]

My SAN administrator has assured me nothing has changed at all. We have 2 VEEAM implementations backing up VM's residing on 2 different IBM XIV SAN's. The only common point between them is vCenter. I have one vCenter shared between two sites. vCenter got upgraded to v5.5 Update 3b and I think the backups stopped around then, but not 100% sure. The Veeam host is in the same server group on the SAN as the VMware hosts, so they all see the same luns.

[Vitaliy S.]

If logs review is not possible with the help of our support team, then try to re-configure direct SAN access and let us know how it goes.

[SeanD]

Are there certain logs files I can scan over to see if there are any errors? Not sure which ones to look at.

[Vitaliy S.]

Start with the backup job log (get it through Help-Support Information), there should be information regarding the reason of switching to NBD mode, right in the beginning of the job session.

[SeanD]

So I am looking in the Agent source vmdk log and I am seeing an ERR The specified transports are not available. I see it trying to connect to the vCenter over port 443. Is Veeam 7 using SSLV3 to connect to the vCenter? It is when I is trying a vddkConSpec command. I see a VDDK logon has failed message.

[Vitaliy S.]

There are indeed a couple of existing topics about SSLv3 issues, but I'm not that it fully relates to your case. Might be worth checking.

[SeanD]

I fixed the problem by re-enabling SSLv3 on vcenter. I followed kb.vmware.com/kb/2139396 and enabled SSLv3 for the Security Token Service (STS) and VMware vSphere Web Client Service (vspherewebclientsvc). Not sure if doing both or just the Web Client Service would of fixed it.

[Vitaliy S.]

Sean, good to know that we've fixed it! Thanks for the update!

[boatmn810]

Try splitting your jobs based on the disk type - IDE on one pool, SCSI another pool as this has bitten me in the a$$ a couple times - undocumented feature up till 8 release 3.

Proxy gets lost trying to schedule and can't mix the types drives

[PTide]

Hi,

Try splitting your jobs based on the disk type - IDE on one pool

IDE disk processing in Direct-SAN mode requires B&R v8.0 Update 2 whereas OP has v7.0.0.871.

Thank you.

[SeanD]

Well it looks like the problem has come back. It started about a week or so ago, but I didn't have time to work on it. I now get a: Client err: NFC storage connection is unavailable. Failed to create NFC download stream

Funny thing is, I have 2 Veeam 7.0 installs connecting to the same vCenter and one Veeam installation works and the other doesn't. I have rebooted the one that doesn't work and that didn't help.

[foggy]

Please review this thread and this KB article, should give you some hints.

[SeanD]

I fixed the problem. The issue now was with each ESXi host. I had to re-enable SSLV3 on each host. Once I did that, all the backup jobs started to work again.