Event 157: Disk X has been surprise removed.

VMware specific discussions

Event 157: Disk X has been surprise removed.

Veeam Logoby MrSpock » Tue Jan 26, 2016 9:12 pm

Hi,

I get a lot of event 157 (Disk X has been surprise removed.) in the event log on my Veeam Backup server (Windows Server 2012 R2) during backup jobs. Why is this happening?

Image

Thanks,

Johan
MrSpock
Enthusiast
 
Posts: 34
Liked: 1 time
Joined: Fri Apr 24, 2009 10:16 pm

Re: Event 157: Disk X has been surprise removed.

Veeam Logoby foggy » Tue Jan 26, 2016 9:15 pm

Johan, does your backup server act as hotadd proxy during your jobs?
foggy
Veeam Software
 
Posts: 14742
Liked: 1080 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Event 157: Disk X has been surprise removed.

Veeam Logoby MrSpock » Tue Jan 26, 2016 10:29 pm

Hi, foggy.

Yes, I suppose so, but not deliberately. The backup server is virtual and located on a host with local storage. The backup server is backing up VMs located both on the same host and on other hosts. There is no SAN involved, just local storage. The action logs say that "hotadd" is used on the VMs located on the same host, otherwise "nbd". Does Veeam Backup release the hotadded disks improperly?

Johan
MrSpock
Enthusiast
 
Posts: 34
Liked: 1 time
Joined: Fri Apr 24, 2009 10:16 pm

Re: Event 157: Disk X has been surprise removed.

Veeam Logoby foggy » Wed Jan 27, 2016 9:58 am

Do those events always correspond to the time when the job finishes processing some VM disk using hotadd (according to the job session log)?
foggy
Veeam Software
 
Posts: 14742
Liked: 1080 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Event 157: Disk X has been surprise removed.

Veeam Logoby MrSpock » Wed Jan 27, 2016 11:55 am

Hi,

Yes, I have searched through all logs from the last backups session. I can see that this error is always logged by Veeam Backup one second before event 157 is written to the Windows event log.

Code: Select all
error -[05124] [Originator@6876 sub=Default] TryReleaseDevice: DynLinkApi_SetupDiRemoveDeviceInterface failed. Error = 0xe000021b.\n

I didn't find 0xe000021b anywhere else in the logs.

Bigger snippet:

Code: Select all
[26.01.2016 20:55:51] <  6016> cli| -------------------------------------------------------------------------------
[26.01.2016 20:55:51] <  6016> cli| HotAdd.DetachDisks
[26.01.2016 20:55:51] <  6016> cli| - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
[26.01.2016 20:55:51] <  6016> dsk| Removing attached vmdk disks.
[26.01.2016 20:55:51] <  6016> dsk| Connection cookie: [<hostname>:moref=3:hotadd:nbd], hex:[3ef44ed0]
[26.01.2016 20:55:51] <  6016> dsk| Connection cookie: [<hostname>:moref=3:hotadd:nbd], hex:[3ef44ed0]
[26.01.2016 20:55:51] <  6016> dsk| Entering in ~disk_handle_t()
[26.01.2016 20:55:51] <  6016> dsk| Closing disk in ~disk_handle_t()
[26.01.2016 20:55:51] <  4464> dsk|   Executing VDL_Close::Do()
[26.01.2016 20:55:51] <  4464> dsk|     VixDiskLib: VixDiskLib_Close: Close disk.\n
[26.01.2016 20:55:51] <  4464> dsk|     W32Util_CloseDismountHandle: Unlocking and closing handles for 0 volumes on PhysicalDrive6...\n
[26.01.2016 20:55:51] <  4464> dsk|     FILE: FileRemoveDirectoryRetry: Non-retriable error encountered (C:\Windows\TEMP\VeeamBackup\VeeamAgent-8032\vmware-SYSTEM\564d3fdb-8bd3-089f-2f53-63490704048d-3\hotadd\VmdkStubs): The directory is not empty (145)\n
[26.01.2016 20:55:51] <  4464> dsk|     FILE: FileRemoveDirectoryRetry: Non-retriable error encountered (C:\Windows\TEMP\VeeamBackup\VeeamAgent-8032\vmware-SYSTEM\564d3fdb-8bd3-089f-2f53-63490704048d-3\hotadd\VmdkStubs): The directory is not empty (145)\n
[26.01.2016 20:55:56] <  2836> dsk|     2016-01-26T20:55:56.294+01:00 error -[02836] [Originator@6876 sub=Default] TryReleaseDevice: DynLinkApi_SetupDiRemoveDeviceInterface failed. Error = 0xe000021b.\n
[26.01.2016 20:55:56] <  2836> dsk|     2016-01-26T20:55:56.294+01:00 error -[02836] [Originator@6876 sub=Default] TryReleaseDevice: DynLinkApi_SetupDiRemoveDeviceInterface failed. Error = 0xe000021b.\n

[26.01.2016 20:56:04] <  6016> dsk| Entering in ~disk_handle_t()
[26.01.2016 20:56:04] <  6016> dsk| Closing disk in ~disk_handle_t()
[26.01.2016 20:56:04] <  4464> dsk|   Executing VDL_Close::Do()
[26.01.2016 20:56:04] <  4464> dsk|     VixDiskLib: VixDiskLib_Close: Close disk.\n
[26.01.2016 20:56:04] <  4464> dsk|     W32Util_CloseDismountHandle: Unlocking and closing handles for 0 volumes on PhysicalDrive7...\n
[26.01.2016 20:56:09] <  2836> dsk|     2016-01-26T20:56:09.263+01:00 error -[02836] [Originator@6876 sub=Default] TryReleaseDevice: DynLinkApi_SetupDiRemoveDeviceInterface failed. Error = 0xe000021b.\n


Thanks for your help,

Johan
MrSpock
Enthusiast
 
Posts: 34
Liked: 1 time
Joined: Fri Apr 24, 2009 10:16 pm

Re: Event 157: Disk X has been surprise removed.

Veeam Logoby foggy » Wed Jan 27, 2016 1:44 pm

This this behavior looks to be typical for Windows 2012 R2.
foggy
Veeam Software
 
Posts: 14742
Liked: 1080 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Event 157: Disk X has been surprise removed.

Veeam Logoby MrSpock » Wed Jan 27, 2016 1:59 pm

That may explain the Windows log event, but why is Veeam logging an error while trying to remove the device? It doesn't look like expected behavior. I have to admit that the hotadd feature feels a bit unreliable. Is it possible to disable hotadd in Veeam Backup?

Johan
MrSpock
Enthusiast
 
Posts: 34
Liked: 1 time
Joined: Fri Apr 24, 2009 10:16 pm

Re: Event 157: Disk X has been surprise removed.

Veeam Logoby foggy » Wed Jan 27, 2016 5:16 pm

This error comes from vSphere, when it detaches disk from the proxy server (Veeam B&R just translates vSphere logs for operations, where it calls vSphere to do something). So this is internal to VMware and tells nothing against hotadd reliability and overall Veeam B&R operation. You can trust hotadd, since it's proved its reliability over all 8 previous versions of Veeam B&R. Besides, as you can see from the provided link, the same occurs in case of Hyper-V.
foggy
Veeam Software
 
Posts: 14742
Liked: 1080 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: Event 157: Disk X has been surprise removed.

Veeam Logoby Thernlund » Sun Jul 03, 2016 9:28 am

I'm seeing this behavior too. I checked Disk Management during a backup and it appears that Veeam mounts the VMDK files as offline disks to back them up. This error seems to appear in the event log when it disconnects the disk.

Is that an accurate representation? That is, Veeam does mount the VMDK files for a given VM during a backup?

As an aside, I use the term 'error' loosely. In reality, it's a warning, not an error. I'm happy to ignore warnings assuming my observations above are expected behavior.
Thernlund
Enthusiast
 
Posts: 29
Liked: 3 times
Joined: Wed Sep 09, 2015 12:02 am
Full Name: Terry Hernlund

Re: Event 157: Disk X has been surprise removed.

Veeam Logoby v.Eremin » Mon Jul 04, 2016 10:39 am

Is that an accurate representation?

Correct. Virtual disks of given VM are attached (hot-added) to a specific proxy server. Thanks.
v.Eremin
Veeam Software
 
Posts: 13266
Liked: 968 times
Joined: Fri Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin


Return to VMware vSphere



Who is online

Users browsing this forum: Google [Bot] and 7 guests