Discussions specific to the Microsoft Hyper-V hypervisor
wim17
Novice
Posts: 8
Liked: never
Joined: May 15, 2017 7:42 am
Full Name: Wim van Rooij
Contact:

Disconnecting D drives

Post by wim17 » Jul 04, 2017 7:30 am

Hi Forum,

In our datacenter we sometimes come across the strange thing that D drives from Domain Controller servers disconnect. This rarety usually happens just after Veeam removes the snapshot. My colleagues are urging me to open a support case with Veeam because it is obvious a Veeam bug but I think it is more of a resources problem.
Just after the removal of de snapshot the domain controller with get this error in the event log:
ID 153
The IO operation at logical block address 0x10 for Disk 1 (PDO name: \Device\0000002d) was retried.

Are there any knows cases like this? The funny thing is it will only trigger on Domain controllers.

wim17
Novice
Posts: 8
Liked: never
Joined: May 15, 2017 7:42 am
Full Name: Wim van Rooij
Contact:

Re: Disconnecting D drives

Post by wim17 » Jul 04, 2017 9:25 am

Little edit. The Veeam jobs don't give any errors and provide a good back-up.

tdewin
Veeam Software
Posts: 1344
Liked: 443 times
Joined: Mar 02, 2012 1:40 pm
Full Name: Timothy Dewin
Contact:

Re: Disconnecting D drives

Post by tdewin » Jul 04, 2017 10:19 am

Are you running vsphere 5.5 or older? You might consider Backup From Storage Snapshot and/or upgrading to 6 : https://www.virtualtothecore.com/en/vsp ... hing-past/

In the end we just do a snapshot and remove it. You can see if you can emulate the problem by creating a snapshot and keep it open for the time it takes to do the backup. Then remove the snapshot and see if you get the same issue

wim17
Novice
Posts: 8
Liked: never
Joined: May 15, 2017 7:42 am
Full Name: Wim van Rooij
Contact:

Re: Disconnecting D drives

Post by wim17 » Jul 04, 2017 11:21 am

Thanks for the reply.
We are usieng Hyper-V 2016 and hardware version 5 on the VM's.
I should have said Checkpoint, sorry.

evander
Enthusiast
Posts: 64
Liked: 4 times
Joined: Nov 17, 2011 7:55 am
Contact:

Re: Disconnecting D drives

Post by evander » Jul 04, 2017 3:07 pm

I don't believe this is related to Veeam at all but rather just your disk taking a bit of strain during a backup run, regardless of the backup vendor. I checked my Domain Controller and I have a few similar events in my logs too. Mostly its just retries that don't add up to much to worry about.
When you say your D drive is getting disconnected do you mean completely disconnected as in you lose your D drive and have to manually reconnect or just that its seems like the D drive has disconnected because of the errors messages listed above?

If possible I would suggest you migrate your AD virtual Machine to another storage platform or datastore and see if you get the same issues. Is your underlying storage good? Raid levels etc?

wim17
Novice
Posts: 8
Liked: never
Joined: May 15, 2017 7:42 am
Full Name: Wim van Rooij
Contact:

Re: Disconnecting D drives

Post by wim17 » Jul 05, 2017 7:01 am

Really disconnected. We have to reboot the machines in order to get them back. The server itself is still running but with a lot of errors, after the reboot they work fine again. All of the servers are on a 3par that is configered by HP, we never had any troubles with it.

evander
Enthusiast
Posts: 64
Liked: 4 times
Joined: Nov 17, 2011 7:55 am
Contact:

Re: Disconnecting D drives

Post by evander » Jul 05, 2017 7:22 am

Ouch thats not cool. Are the C drive and D drive on different LUNS?
I use VMware rather than Hyper V so cant offer too much advise on the setup of that but is your Domain Controller doing anything other than being a Domain Controller? The weirdest thing for me is that you say it only affects the domain controllers. Are they running on a different Windows Server version to the other servers? I'm thinking the retry value is different on them than the other servers perhaps but if they all share the same underlying infrastructure then they should all equally be affected. Pure Domain Controllers are also not very "busy" servers with large amounts of data changes so backups should be easy to complete

Does this help at all: https://community.spiceworks.com/topic/ ... under-load

wim17
Novice
Posts: 8
Liked: never
Joined: May 15, 2017 7:42 am
Full Name: Wim van Rooij
Contact:

Re: Disconnecting D drives

Post by wim17 » Jul 05, 2017 7:58 am

Thanks for the link, I will check that out.

The Domain Controllers do nothing else then the DC tasks. The Windows version is the same then on other Windows servers. The Domain Controllers are not all in the same job. We host a lot of customers on the same platform and devided the Veeam jobs by Customer, so all of the jobs have a DC, a database server, fileserver, terminal server, etc etc. Almost all of those jobs have had the problem with the DC and not with the other servers. De C: and D: are on the same storage. The problems occurs during the first backup job (so no retry).

vmniels
Veeam Software
Posts: 2070
Liked: 458 times
Joined: Jul 15, 2013 11:09 am
Full Name: Niels Engelen
Contact:

Re: Disconnecting D drives

Post by vmniels » Jul 05, 2017 8:22 am

You can always contact support for insight however I don't think this is related due to Veeam but a reaction of the checkpoint towards the VM (which we do create though).
VCP-DCV
Veeam Certified Architect (VMCA)
http://foonet.be

ESWIT
Lurker
Posts: 1
Liked: never
Joined: Nov 23, 2016 10:07 am
Full Name: ESW IT
Contact:

Re: Disconnecting D drives

Post by ESWIT » Oct 19, 2017 6:56 am

Hi,

we have the same problem with a disconnected drive during a backup. We recently upgraded our HyperV infrastructure to 2016.
In our case the error occurs on a 2012R2 SQL Server VM with a non system disk. Seconds before the disk disappears we can see disk io retries on the disk.

____The IO operation at logical block address 0x0 for Disk 1 (PDO name: \Device\00000030) was retried.
____Faulting application name: sqlservr.exe, version: 0.0.0.0, time stamp: 0x5764deac (sql databases resides on the problematic disk)

the backup job on veeam runs fine without problems. It happens when the snapshot is closed/merged and it's very randomly. You can have no problems for weeks or it can happen 2 times a day (we backup our SQL server 4 times a day incremental)
I think this is more a hyperv2016 related problem as the original poster also have a 2016 infrastructure. We had no problems with 2012R2.

Dear Veeam support, maybe you can use your excellent connections to the HyperV Team from Microsoft if there is a 'known' problem which is not that public yet?

Thank you
Christoph Werner

willa.t
Novice
Posts: 6
Liked: 1 time
Joined: Nov 25, 2013 3:33 pm
Full Name: Thomas Willa
Contact:

Re: Disconnecting D drives

Post by willa.t » Apr 03, 2018 12:20 pm

Hello togther,

we have the same issue? You are already found an solution?

Thank, Thomas

foggy
Veeam Software
Posts: 17125
Liked: 1399 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Disconnecting D drives

Post by foggy » Apr 03, 2018 3:33 pm

Hi Thomas, I don't see any single case number here, so cannot check. Please contact technical support directly.

Cullan
Service Provider
Posts: 93
Liked: 13 times
Joined: May 21, 2014 8:47 am
Full Name: Cullan Simpson
Location: New Zealand
Contact:

Re: Disconnecting D drives

Post by Cullan » Apr 04, 2018 4:56 am 2 people like this post

Hi there,

Just confirming we are also having this issue. We recently updated some of our hosts from 2012 R2 to Server 2016. This issue only affects VMs running on the 2016 hosts and the guests affected so far run SQL and IIS so I don't think the issue is AD related.
We opened a case with Veeam #02433873 but they could not find anything in the logs. We currently have a case open with Microsoft.

Safe to say it's an issue with Hyper-V on Server 2016.

willa.t
Novice
Posts: 6
Liked: 1 time
Joined: Nov 25, 2013 3:33 pm
Full Name: Thomas Willa
Contact:

Re: Disconnecting D drives

Post by willa.t » Apr 04, 2018 12:24 pm 1 person likes this post

Hi Cullan,

I can confirm that the Issue is Windows Server 2016 related. We also have trouble with VM, which run with SQL Server Version 2014 to 2016. The problem is 100% not AD related, but it occurs when the checkpoint is merging. I was able to reproduce this issue with generating heavy IOs on a test VM and create / delete Checkpoints manuelly. Onward it dosen't matter on which Storage Subsystem the VM are located. We have this issue also with local SSD Storage.

I also think this is one more issue of Hyper-V 2016 and have opened a MS Support call, too :-)

If I have any news, I let you know.

foggy
Veeam Software
Posts: 17125
Liked: 1399 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Disconnecting D drives

Post by foggy » Apr 04, 2018 1:04 pm

Thanks, guys, I'd appreciate if you share the results of your findings.

Post Reply

Who is online

Users browsing this forum: DGrinev, foggy, pdemidov, PTide and 21 guests