v7 and Dell MD3000i errors

[aeccles]

I just installed v7 into a setup of Server 2012 and an MD3000i

I'm getting the following three errors throughout the night. I've gotten the messages at the exact time jobs start AS WELL as other times when jobs are not running (perhaps some background checks veeam does?)
Error #1
Event Description: Faulting application name: SMPROV~1.EXE, version: 10.84.26.2, time stamp: 0x50405699 Faulting module name: ntdll.dll, version: 6.2.9200.16420, time stamp: 0x505ab405 Exception code: 0xc0000374 Fault offset: 0x00000000000ea485 Faulting process id: 0x1128 Faulting application start time: 0x01ceaafb22b60747 Faulting application path: C:\PROGRA~2\Dell\MDSTOR~1\VDS_VSS\provider\SMPROV~1.EXE
Faulting module path: C:\Windows\SYSTEM32\ntdll.dll Report Id: 45d2fbba-16ef-11e3-93f6-001517cc709f

Error #2
Event Description: An attempt was made to issue a command to a password protected storage system and the correct password is not known.

Error #3
Event Description: The Modular Disk Storage Hardware Provider service terminated unexpectedly. It has done this 10 time(s).

There is also a post on a dell forum about the exact same messages although he is using different VM backup software.
http://en.community.dell.com/support-fo ... ?pi21859=2


Anyone have any insite on this? I'm traveling today and I'll open a ticket with Veeam on Monday when I am back in the office.

Thanks

[Vitaliy S.]

Hello,

Can you please tell me if you select your hyper-v server in the backup console and click the "Rescan" button do you see the same errors in the event log of your Dell MD3000i storage?

Thank you!

[aeccles]

A little more information on this...

I just noticed this... I have 3 Hyper-V hosts and they are all listed under Managed Servers/Microsoft Hyper-V within Veeam. However, I've only backed up VMs on two of the hosts. Those two hosts are the ones generating the event logs. The third host hasn't reported anything.

[Vitaliy S.]

I'm not an MD3000i expert, but based on the event description it seems like the connection to your storage might be protected with some kind of a password, so when the backup job tries to retrieve VMs data from the storage and does not provide any passwords, it generates this event in the event log. Anyway, try a quick test with a "Rescan" button and let me know the results. Thanks!

[aeccles]

Hello,

Can you please tell me if you select your hyper-v server in the backup console and click the "Rescan" button do you see the same errors in the event log of your Dell MD3000i storage?

Thank you!

Hi Vitaliy, I just ran the rescan and YES - all three errors occurred.

[Vitaliy S.]

Ok, now everything is clear, but it doesn't explain why you still have successful backup jobs. BTW, you DO have successful backup jobs with these errors, right?

To resolve this issue you should look at the configuration of your storage connection. Every 24 hours Veeam performs automatic object/topology discovery action to check whether anything changed in your environment or not. The same procedure is triggered when you run the backup job, this is required for VM backup operation. Now reading back through the error message above, I would recommend contacting Dell support to verify where this password protection option is configured. Once you disable this option or configure some sort of trust relationship between Dell storage and Veeam backup proxy/backup server, this error should go away.

Hope this helps!

[aeccles]

I do not have my SAN setup to share the LUNS with the Veeam server, so I'm using network backup. Do you think this lack of access could be causing it? If so, is there a way I can tell Veeam to default to network backup instead of direct access?


And yes, all of my backups are running and are error free as far as Veeam is reporting

Thanks

[Vitaliy S.]

I believe by saying network backup, you refer to on-host backup mode? Are you using software or hardware VSS provider? What other errors do you see in the event log?

[aeccles]

I haven't gotten a clear response from Dell, but it looks like they are suggesting that this may be happening because I did not purchase the premium support features for this SAN. One of those features is for Snapshotting virtual disks.

Does that make sense to you? I could see how it could come up when the jobs run, but I don't know why it would happen when Veeam performs it's daily topology discovery.

As for your last questions. Yes, I was referring to on-host backup mode. ( I think I confused some Hyper-V/VMware terms)
And sorry if this is a stupid question, but how do I know if I am using a software or hardware VSS? Perhaps the feature I was talking about above would have been it?

Thank you

[Vitaliy S.]

I haven't gotten a clear response from Dell, but it looks like they are suggesting that this may be happening because I did not purchase the premium support features for this SAN. One of those features is for Snapshotting virtual disks.

Does that make sense to you?

Not really, as during scan job there is no virtual disk snapshotting operation.

As for your last questions. Yes, I was referring to on-host backup mode. ( I think I confused some Hyper-V/VMware terms)
And sorry if this is a stupid question, but how do I know if I am using a software or hardware VSS? Perhaps the feature I was talking about above would have been it?

Can you run this command on your backup server/on-host proxy - "vssadmin list providers"? What is the output?

[aeccles]

Here's the results of the list providers:


C:\Windows\system32>vssadmin list providers
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2012 Microsoft Corp.

Provider name: 'SmVssProvider'
Provider type: Hardware
Provider Id: {24e4e8a3-69cb-4370-8b7b-c5276cd49765}
Version: 10.84.02

Provider name: 'Microsoft File Share Shadow Copy provider'
Provider type: Fileshare
Provider Id: {89300202-3cec-4981-9171-19f59559e0f2}
Version: 1.0.0.1

Provider name: 'Microsoft Software Shadow Copy provider 1.0'
Provider type: System
Provider Id: {b5946137-7b9f-4925-af80-51abd60b20d5}
Version: 1.0.0.7


C:\Windows\system32>

[Vitaliy S.]

Looks like you're using hardware VSS provider - "Provider name: 'SmVssProvider'" to backup your VMs. Try updating it to the most recent version, if it doesn't help, then I believe you can ignore this message (assuming all your backups are valid and backup jobs are running fine).

Also I wonder if there is a way to disable password protection of your volumes since this is a "premium support feature for this SAN" which you are not using anyway.

[aeccles]

OK, to hopefully wrap this up and help out others who may run into this...

It looks like the solution to this is to uninstall the "SmVssProvider" which, when you run "vssadmin list providers," is listed as:
Provider name: 'SmVssProvider'
Provider type: Hardware
Provider Id: {24e4e8a3-69cb-4370-8b7b-c5276cd49765}
Version: 10.84.02

That service initiates the premium add-on Snapshot service that you can buy for the MD line through Dell. If you didn't buy it, you don't need this provider. (as far as I understand it at least)

To uninstall this provider, navigate to the VDS-VSS uninstall folder in the MD Storage folder. By Default it is at:
C:\Program Files (x86)\Dell\MD Storage Software\VDS_VSS\Uninstall Modular Disk Storage Hardware Provider (Be careful here and double check your path. There is more than one uninstaller with a similar path and name.)
Once in that folder, launch the "Uninstall Modular Disk Storage Hardware Provider.exe"
Note that the uninstaller is very rudimentary - there's no GUI or confirmation that it completed. The folder and service will just disappear.

This will remove the "SmVssProvider" and the related Windows Service:
"Modular Disk Storage Hardware Provider" (SMPROVIDER1.EXE - Same as SmProv~1.exe).


In my case, the "Modular Disk Storage Hardware Provider" service was set to "Automatic" start, although it was not running, so I was unable to uninstall the provider without stopping I/O or causing any disruption to my VMs. There was no reboot request either. YMMV though so proceed with caution.

THANKS VERY MUCH to Vitaliy for leading me to the solution!

[Vitaliy S.]

Aeccles, can you please clarify one more thing before we wrap this up?

I want you to look through your backup job sessions and verify what VSS prodiver was used previously (search for this line - "Creating snapshot....")? Was it a "SmVssProvider"? Also what provider is used now after you have uninstalled SmVssProvider?

[aeccles]

Aeccles, can you please clarify one more thing before we wrap this up?

I want you to look through your backup job sessions and verify what VSS prodiver was used previously (search for this line - "Creating snapshot....")? Was it a "SmVssProvider"? Also what provider is used now after you have uninstalled SmVssProvider?

In both cases it was "Microsoft Software Shadow Copy provider 1.0 (mode: Veeam application-aware processing.)

[Vitaliy S.]

Ok, then you can keep using your current configuration, I've just thought that you were using hardware provider previously.

[timofcourse]

I realize this thread is a bit old, but I am having a similar issue as mentioned in the o.p. Unfortunately when trying to uninstall using this referenced file, nothing seems to happen. The folder and services stay untouched and the SmVssProvider are not removed.

Any one happen to have any ideas how to force the removal of this?

Thanks!

[veremin]

Hi, Tim, since the said hardware provider is shipped by Dell, the Dell support or online community might be a better place to ask this sort of questions. Thanks.

[aeccles]

I realize this thread is a bit old, but I am having a similar issue as mentioned in the o.p. Unfortunately when trying to uninstall using this referenced file, nothing seems to happen. The folder and services stay untouched and the SmVssProvider are not removed.

Any one happen to have any ideas how to force the removal of this?

Thanks!

Tim,
Is the "Modular Disk Storage Hardware Provider" service started on your host system? If so you may need to restart for the provider to be removed.

[timofcourse]

Bah... I figured it out. Run as administrator! Not sure why I never thought of that until now. That removes it as previously described.

Thanks!

[JamesNT]

I have acquired a Dell MD3200 direct attach storage and am getting the same error. Does removing this provider solve the problem? Our backups using Veeam 9.0 are running fine but it is annoying to see the error in the event logs.

JamesNT

[timofcourse]

Uninstalling did resolve the issue for us.