3Par Storage Integration Not working

[stb74]

Case # 04995965

Hi

I have recently upgraded to v11 and since then the Storage integrated snapshots are not working jobs are failing with Error: FC initiators list is empty.

When a job starts everything appears to be running as expected, in the Job you see Creating storage snapshots for backup and in vCenter the snapshot being taken.
Then in the job
10/09/2021 18:23:44 :: Using backup proxy PROXY1 for retrieving Hard disk 1 data from storage snapshot on SSTOR

Eventually this times out
Error: FC initiators list is empty

I have checked all the sections of the configuration and just cannot figure out why this is happening.

[foggy]

Sounds like a proxy detection issue during export. Please continue investigating with support.

[chris.childerhose]

Based on the messages you have posted it seems like there is still some configuration that needs to be checked. If you only have the one Proxy then it needs to have the correct access to the FC storage otherwise it will not work. Hopefully support comes back with an answer for you.

[foggy]

Veeam B&R checks proxy availability before attempting backup from storage snapshots - for some reason, the proxy that was properly matched at the beginning of the job is not available upon the actual export.

[stb74]

When the job get to Using Backup Proxy for retrieving Hard Disk you can see on the Storage that a volume has been exported VeeamAUX_3TVfbmxYwr80iP7BcLrW7D to the proxy

[foggy]

A deeper logs review is required, please continue working with our engineers. Thanks!

[stb74]

So after some further testing we have found that Sophos is the problem, we have all the exceptions in place but it doesn't like the disks being mounted on the Proxy for the backup to take place. Is there any other exceptions that people think are required when mounting the disks.

I have checked Sophos logs and there isn't much information or any error messages.

[foggy]

Thanks for the update. I'd contact Sophos directly with the exceptions-related question.

[johannesk]

So after some further testing we have found that Sophos is the problem, we have all the exceptions in place but it doesn't like the disks being mounted on the Proxy for the backup to take place. Is there any other exceptions that people think are required when mounting the disks.

I have checked Sophos logs and there isn't much information or any error messages.

Hi,

How did you resolve those "FC initiators list is empty " errors? We are seeing those when backing up with one of our proxy doing storage snapshot of Pure. Pure plugin in VBR is up-to-date. Zoning is ok, and SAN transfer works if storage integration is disabled. We have removed the pure arrays from the storage infrastructure config and re-added them. Disabled antivirus completely to rule that out. We've done alot of troubleshooting (case #05259111 ). No success yet.

So Im curious, how did you resolve this issue?

Regards,
Jóhannes

[rennerstefan]

Hi Johannes,

can you provide a little more details:
- Did it never work or just stopped working?
- Is Pure MPIO installed and configured the right way?
- Do you have only one proxy or multiple and is the error only on one proxy if you have multiple?

I see in the ticket that 3rd level is already involved.
Did you also create a ticket at Pure to let them have a look?

Thanks

[johannesk]

Hi,

We have two identical proxy servers, each in it's datacenter. Only one of them has this issue. Storage integartion backups have been working fine for about 15 months on those proxies.

recent changes before the issue is that 'Windows Defender Advanced Threat Protection Service' (ATP) was enabled on the server, but also on the non-problem proxy. We have run the promlem proxy with ATP completely disabled for a few days and still have the same issue.

The Pure MPIO DSM has been reconfigured and reviewed. Pure support has been involved. they show this issue little interest since there seems to be no problem with the storage array. They have reviewed the DSM config in MPIO, zoning and host groups on the Pure storage array.

We have removed the Pure storage integration connection from the storage infrastructure in VBR done a reboot, and add it again.

Storage discovery is sucessful.

The Pure-plugin is the most recent version 1.3.18 and has been in use on this system since last year.

SAN transfer in backups work ok between the problem proxy and the storage array.

If we run storage integration backups a snapshot is created on the Pure array and there is an error in the backup process that results in that the snapshot is left there without being cleaned up.

That resulted in that the devicemanager on the problem proxy would see those snapshots (VEEAM-StorageLUNSnap* snapshots) , and have those as ! remarked disks in device manager. If those snapshots are cleaned up in Pure array, then device manager gets clean.

We are on the verge of just reimage the proxy server. But I think this problem might just be with a fresh OS also.

Regards,
Jóhannes

[rennerstefan]

Thanks for the details.
The interesting thing is that you say: "Only one of them has this issue."
So it is not a general issue but more a local issue with that server. That is good in a way that it limits the problem to one server and bad as you already tried lots of stuff with the problem one.
I had a similar issue with a HPE Primera recently where one proxy worked and the other didn't. At the end it was a setting in the MPIO driver which HPE found after investigating
Not saying it can't be our issue but as one works and the other doesn't it sounds unlikely.

To be very honest, I'd go ahead and re-image that proxy (I guess it only holds the proxy role and that is not really a big deal to re-install).
I would also remove all existing host-groups for that proxy etc. on the pure so that you have a "fresh" environment when it comes to that proxy.
After you re-imaged the proxy please don't install anything else than the minimum required tools such as the proxy role and MPIO and see if it works again.
In my eyes it looks like something has changed on the proxy (most likely the OS) and no one know what the change was.
If it still doesn't work a deeper investigation would be needed.

But please check back with our support first BEFORE you re-image the system to make sure they are ok with it and don't need any logs.

Please let us know here how you proceed and also how it was solved as this may help others as well.

Thanks

[Eric_Cartman]

Any news on this matter?
We've got the same problem since the beginning of this year but it only appears in irregular intervals

[foggy]

In the latter case mentioned above, reinstalling OS on the proxy helped.

What storage do you have in this setup? Do you have an open support case? As you can see, the two cases above had different prerequisites for the issue so working with support is required to identify the reasons in your case.

[Eric_Cartman]

Thank you foggy but the issue seems to be resolved by the application himself.
I opened a call as well and executed the recommended steps by the support staff