v7 and Dell MD3000i errors

Hyper-V specific discussions

v7 and Dell MD3000i errors

Veeam Logoby aeccles » Fri Sep 06, 2013 1:44 pm

I just installed v7 into a setup of Server 2012 and an MD3000i

I'm getting the following three errors throughout the night. I've gotten the messages at the exact time jobs start AS WELL as other times when jobs are not running (perhaps some background checks veeam does?)
Error #1
Event Description: Faulting application name: SMPROV~1.EXE, version: 10.84.26.2, time stamp: 0x50405699 Faulting module name: ntdll.dll, version: 6.2.9200.16420, time stamp: 0x505ab405 Exception code: 0xc0000374 Fault offset: 0x00000000000ea485 Faulting process id: 0x1128 Faulting application start time: 0x01ceaafb22b60747 Faulting application path: C:\PROGRA~2\Dell\MDSTOR~1\VDS_VSS\provider\SMPROV~1.EXE
Faulting module path: C:\Windows\SYSTEM32\ntdll.dll Report Id: 45d2fbba-16ef-11e3-93f6-001517cc709f

Error #2
Event Description: An attempt was made to issue a command to a password protected storage system and the correct password is not known.

Error #3
Event Description: The Modular Disk Storage Hardware Provider service terminated unexpectedly. It has done this 10 time(s).

There is also a post on a dell forum about the exact same messages although he is using different VM backup software.
http://en.community.dell.com/support-fo ... ?pi21859=2


Anyone have any insite on this? I'm traveling today and I'll open a ticket with Veeam on Monday when I am back in the office.

Thanks
aeccles
Enthusiast
 
Posts: 80
Liked: 6 times
Joined: Tue May 01, 2012 3:00 pm

Re: v7 and Dell MD3000i errors

Veeam Logoby Vitaliy S. » Fri Sep 06, 2013 2:40 pm

Hello,

Can you please tell me if you select your hyper-v server in the backup console and click the "Rescan" button do you see the same errors in the event log of your Dell MD3000i storage?

Thank you!
Vitaliy S.
Veeam Software
 
Posts: 19545
Liked: 1099 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: v7 and Dell MD3000i errors

Veeam Logoby aeccles » Fri Sep 06, 2013 2:44 pm

A little more information on this...

I just noticed this... I have 3 Hyper-V hosts and they are all listed under Managed Servers/Microsoft Hyper-V within Veeam. However, I've only backed up VMs on two of the hosts. Those two hosts are the ones generating the event logs. The third host hasn't reported anything.
aeccles
Enthusiast
 
Posts: 80
Liked: 6 times
Joined: Tue May 01, 2012 3:00 pm

Re: v7 and Dell MD3000i errors

Veeam Logoby Vitaliy S. » Fri Sep 06, 2013 2:51 pm

I'm not an MD3000i expert, but based on the event description it seems like the connection to your storage might be protected with some kind of a password, so when the backup job tries to retrieve VMs data from the storage and does not provide any passwords, it generates this event in the event log. Anyway, try a quick test with a "Rescan" button and let me know the results. Thanks!
Vitaliy S.
Veeam Software
 
Posts: 19545
Liked: 1099 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: v7 and Dell MD3000i errors

Veeam Logoby aeccles » Fri Sep 06, 2013 3:35 pm

Vitaliy S. wrote:Hello,

Can you please tell me if you select your hyper-v server in the backup console and click the "Rescan" button do you see the same errors in the event log of your Dell MD3000i storage?

Thank you!


Hi Vitaliy, I just ran the rescan and YES - all three errors occurred.
aeccles
Enthusiast
 
Posts: 80
Liked: 6 times
Joined: Tue May 01, 2012 3:00 pm

Re: v7 and Dell MD3000i errors

Veeam Logoby Vitaliy S. » Fri Sep 06, 2013 3:57 pm

Ok, now everything is clear, but it doesn't explain why you still have successful backup jobs. BTW, you DO have successful backup jobs with these errors, right? ;)

To resolve this issue you should look at the configuration of your storage connection. Every 24 hours Veeam performs automatic object/topology discovery action to check whether anything changed in your environment or not. The same procedure is triggered when you run the backup job, this is required for VM backup operation. Now reading back through the error message above, I would recommend contacting Dell support to verify where this password protection option is configured. Once you disable this option or configure some sort of trust relationship between Dell storage and Veeam backup proxy/backup server, this error should go away.

Hope this helps!
Vitaliy S.
Veeam Software
 
Posts: 19545
Liked: 1099 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: v7 and Dell MD3000i errors

Veeam Logoby aeccles » Fri Sep 06, 2013 7:17 pm

I do not have my SAN setup to share the LUNS with the Veeam server, so I'm using network backup. Do you think this lack of access could be causing it? If so, is there a way I can tell Veeam to default to network backup instead of direct access?


And yes, all of my backups are running and are error free as far as Veeam is reporting

Thanks
aeccles
Enthusiast
 
Posts: 80
Liked: 6 times
Joined: Tue May 01, 2012 3:00 pm

Re: v7 and Dell MD3000i errors

Veeam Logoby Vitaliy S. » Fri Sep 06, 2013 8:07 pm

I believe by saying network backup, you refer to on-host backup mode? Are you using software or hardware VSS provider? What other errors do you see in the event log?
Vitaliy S.
Veeam Software
 
Posts: 19545
Liked: 1099 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: v7 and Dell MD3000i errors

Veeam Logoby aeccles » Mon Sep 09, 2013 4:58 pm

I haven't gotten a clear response from Dell, but it looks like they are suggesting that this may be happening because I did not purchase the premium support features for this SAN. One of those features is for Snapshotting virtual disks.

Does that make sense to you? I could see how it could come up when the jobs run, but I don't know why it would happen when Veeam performs it's daily topology discovery.

As for your last questions. Yes, I was referring to on-host backup mode. ( I think I confused some Hyper-V/VMware terms)
And sorry if this is a stupid question, but how do I know if I am using a software or hardware VSS? Perhaps the feature I was talking about above would have been it?

Thank you
aeccles
Enthusiast
 
Posts: 80
Liked: 6 times
Joined: Tue May 01, 2012 3:00 pm

Re: v7 and Dell MD3000i errors

Veeam Logoby Vitaliy S. » Mon Sep 09, 2013 8:15 pm

aeccles wrote:I haven't gotten a clear response from Dell, but it looks like they are suggesting that this may be happening because I did not purchase the premium support features for this SAN. One of those features is for Snapshotting virtual disks.

Does that make sense to you?

Not really, as during scan job there is no virtual disk snapshotting operation.

aeccles wrote:As for your last questions. Yes, I was referring to on-host backup mode. ( I think I confused some Hyper-V/VMware terms)
And sorry if this is a stupid question, but how do I know if I am using a software or hardware VSS? Perhaps the feature I was talking about above would have been it?

Can you run this command on your backup server/on-host proxy - "vssadmin list providers"? What is the output?
Vitaliy S.
Veeam Software
 
Posts: 19545
Liked: 1099 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: v7 and Dell MD3000i errors

Veeam Logoby aeccles » Mon Sep 09, 2013 8:20 pm

Here's the results of the list providers:


C:\Windows\system32>vssadmin list providers
vssadmin 1.1 - Volume Shadow Copy Service administrative command-line tool
(C) Copyright 2001-2012 Microsoft Corp.

Provider name: 'SmVssProvider'
Provider type: Hardware
Provider Id: {24e4e8a3-69cb-4370-8b7b-c5276cd49765}
Version: 10.84.02

Provider name: 'Microsoft File Share Shadow Copy provider'
Provider type: Fileshare
Provider Id: {89300202-3cec-4981-9171-19f59559e0f2}
Version: 1.0.0.1

Provider name: 'Microsoft Software Shadow Copy provider 1.0'
Provider type: System
Provider Id: {b5946137-7b9f-4925-af80-51abd60b20d5}
Version: 1.0.0.7


C:\Windows\system32>
aeccles
Enthusiast
 
Posts: 80
Liked: 6 times
Joined: Tue May 01, 2012 3:00 pm

Re: v7 and Dell MD3000i errors

Veeam Logoby Vitaliy S. » Mon Sep 09, 2013 8:31 pm

Looks like you're using hardware VSS provider - "Provider name: 'SmVssProvider'" to backup your VMs. Try updating it to the most recent version, if it doesn't help, then I believe you can ignore this message (assuming all your backups are valid and backup jobs are running fine).

Also I wonder if there is a way to disable password protection of your volumes since this is a "premium support feature for this SAN" which you are not using anyway.
Vitaliy S.
Veeam Software
 
Posts: 19545
Liked: 1099 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: v7 and Dell MD3000i errors

Veeam Logoby aeccles » Wed Sep 11, 2013 1:42 pm 1 person likes this post

OK, to hopefully wrap this up and help out others who may run into this...

It looks like the solution to this is to uninstall the "SmVssProvider" which, when you run "vssadmin list providers," is listed as:
Provider name: 'SmVssProvider'
Provider type: Hardware
Provider Id: {24e4e8a3-69cb-4370-8b7b-c5276cd49765}
Version: 10.84.02

That service initiates the premium add-on Snapshot service that you can buy for the MD line through Dell. If you didn't buy it, you don't need this provider. (as far as I understand it at least)

To uninstall this provider, navigate to the VDS-VSS uninstall folder in the MD Storage folder. By Default it is at:
C:\Program Files (x86)\Dell\MD Storage Software\VDS_VSS\Uninstall Modular Disk Storage Hardware Provider (Be careful here and double check your path. There is more than one uninstaller with a similar path and name.)
Once in that folder, launch the "Uninstall Modular Disk Storage Hardware Provider.exe"
Note that the uninstaller is very rudimentary - there's no GUI or confirmation that it completed. The folder and service will just disappear.

This will remove the "SmVssProvider" and the related Windows Service:
"Modular Disk Storage Hardware Provider" (SMPROVIDER1.EXE - Same as SmProv~1.exe).


In my case, the "Modular Disk Storage Hardware Provider" service was set to "Automatic" start, although it was not running, so I was unable to uninstall the provider without stopping I/O or causing any disruption to my VMs. There was no reboot request either. YMMV though so proceed with caution.

THANKS VERY MUCH to Vitaliy for leading me to the solution!
aeccles
Enthusiast
 
Posts: 80
Liked: 6 times
Joined: Tue May 01, 2012 3:00 pm

Re: v7 and Dell MD3000i errors

Veeam Logoby Vitaliy S. » Wed Sep 11, 2013 2:17 pm

Aeccles, can you please clarify one more thing before we wrap this up? :)

I want you to look through your backup job sessions and verify what VSS prodiver was used previously (search for this line - "Creating snapshot....")? Was it a "SmVssProvider"? Also what provider is used now after you have uninstalled SmVssProvider?
Vitaliy S.
Veeam Software
 
Posts: 19545
Liked: 1099 times
Joined: Mon Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov

Re: v7 and Dell MD3000i errors

Veeam Logoby aeccles » Wed Sep 11, 2013 2:26 pm

Vitaliy S. wrote:Aeccles, can you please clarify one more thing before we wrap this up? :)

I want you to look through your backup job sessions and verify what VSS prodiver was used previously (search for this line - "Creating snapshot....")? Was it a "SmVssProvider"? Also what provider is used now after you have uninstalled SmVssProvider?



In both cases it was "Microsoft Software Shadow Copy provider 1.0 (mode: Veeam application-aware processing.)
aeccles
Enthusiast
 
Posts: 80
Liked: 6 times
Joined: Tue May 01, 2012 3:00 pm

Next

Return to Microsoft Hyper-V



Who is online

Users browsing this forum: No registered users and 9 guests