MPIO Issue

[gpomanti]

We had an issue earlier this month where we could not backup anything using 3PAR storage snapshots. We have all of our 3PAR volumes presented to our physical Windows 2012 R2 proxy server and we were successfully backing up our environment with storage snapshots for about a year. After upgrading to v9, everything worked fine for a few days but then backups began to fail. Veeam would initiate the storage snapshot on the 3par, but the storage snapshot was never presented to Windows so the backup couldn't begin. We attempted to launch Device Manager but it would hang and never open. We also noticed that the boot time for this windows proxy server takes an abnormally long time to boot up. Using process monitor, we saw that the proxy was constantly adding regkeys for 3PAR volumes. The regkey for MPIO and the 3PAR volumes was 350 MB. Based on all of this, it seems that when the proxy attempted to get data from the storage snapshot, windows could not see it because it was taking too long to present the snapshot.

We uninstalled MPIO and deleted the regkeys. After a reboot, the system started up in a normal time and device manager launched without an issue. We reinstalled MPIO and tested some backups successfully. It appears that every time Veeam takes a storage snapshot for a backup, it presents that snapshot to the proxy and adds it as a device. Over time the number of devices seemed to have added up and caused system performance issues.

Everything is currently working fine, however, over the last few weeks, the regkey for MPIO has continued to grow to an unusually large size and I'm afraid that the same issue will occur again.

I have a support case open, #01688144, but have not made much progress. Has anyone else experienced an issue like this?

[allankjaer]

May have the same problem.

Can you comment what regkey or where to look for them?

[gpomanti]

Hi Allan,

On the Veeam Proxy server, the regkey is under "HKEY_LOCAL_MACHINE\SYSTEM\CurrentControlSet\Enum\MPIO". I'm not sure how your specific registry will look, but in ours we have a few subfolders below MPIO for different versions of our 3PAR. The most current version is where it creates a new entry every time a storage snapshot is presented and you can see the thousands of entries that have been created.

Unfortunately I still haven't heard anything from support on this so I don't have any sort of update or resolution to provide.

[m.antonio]

Hi Greg,

did you receive any update on your case?

I'm observing the same behavior in a customer deployment that recently upgraded to version 9u1 where those MPIO/SCSI registry keys keeps growing with phantom devices.

Since nothing is changed in the customer environment and that the issue has never shown with version 8.x, I'm inclined to think that there is something in the new proxy server logic that triggers the problem.

In the meantime as a workaround I've configured a weekly scheduled execution of DevNodeClean Microsoft utility to purge phantom storage devices.

Regards,

Antonio

[gpomanti]

Hi Antonio,

No real progress made. Veeam support did testing and determined that it was not an issue on Veeam's end. We opened a case with HP and they determined that it was a Microsoft issue related to MPIO.

I'm not sure if it related to Veeam 9 or not. We began to experience backup failures the day after going to v9 and haven't had an issue since. We have cleaned out the registry once but I believe given enough time, we would have experienced backup failures in v8.

I like the idea of a weekly scheduled clean and will look into implementing something similar.

I will continue to try to find a more permanent solution. If you find any more information, please let me know.

[foggy]

There's a KB articale created for this issue.

[yunales]

thanks for finding this gold nugget!

I was always wondering why the physical proxy servers were so slow over time...