Host-based backup of VMware vSphere VMs.
Post Reply
MrSpock
Service Provider
Posts: 46
Liked: 3 times
Joined: Apr 24, 2009 10:16 pm
Contact:

Event 157: Disk X has been surprise removed.

Post by MrSpock »

Hi,

I get a lot of event 157 (Disk X has been surprise removed.) in the event log on my Veeam Backup server (Windows Server 2012 R2) during backup jobs. Why is this happening?

Image

Thanks,

Johan
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Event 157: Disk X has been surprise removed.

Post by foggy »

Johan, does your backup server act as hotadd proxy during your jobs?
MrSpock
Service Provider
Posts: 46
Liked: 3 times
Joined: Apr 24, 2009 10:16 pm
Contact:

Re: Event 157: Disk X has been surprise removed.

Post by MrSpock »

Hi, foggy.

Yes, I suppose so, but not deliberately. The backup server is virtual and located on a host with local storage. The backup server is backing up VMs located both on the same host and on other hosts. There is no SAN involved, just local storage. The action logs say that "hotadd" is used on the VMs located on the same host, otherwise "nbd". Does Veeam Backup release the hotadded disks improperly?

Johan
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Event 157: Disk X has been surprise removed.

Post by foggy »

Do those events always correspond to the time when the job finishes processing some VM disk using hotadd (according to the job session log)?
MrSpock
Service Provider
Posts: 46
Liked: 3 times
Joined: Apr 24, 2009 10:16 pm
Contact:

Re: Event 157: Disk X has been surprise removed.

Post by MrSpock »

Hi,

Yes, I have searched through all logs from the last backups session. I can see that this error is always logged by Veeam Backup one second before event 157 is written to the Windows event log.

Code: Select all

error -[05124] [Originator@6876 sub=Default] TryReleaseDevice: DynLinkApi_SetupDiRemoveDeviceInterface failed. Error = 0xe000021b.\n
I didn't find 0xe000021b anywhere else in the logs.

Bigger snippet:

Code: Select all

[26.01.2016 20:55:51] <  6016> cli| -------------------------------------------------------------------------------
[26.01.2016 20:55:51] <  6016> cli| HotAdd.DetachDisks
[26.01.2016 20:55:51] <  6016> cli| - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
[26.01.2016 20:55:51] <  6016> dsk| Removing attached vmdk disks.
[26.01.2016 20:55:51] <  6016> dsk| Connection cookie: [<hostname>:moref=3:hotadd:nbd], hex:[3ef44ed0]
[26.01.2016 20:55:51] <  6016> dsk| Connection cookie: [<hostname>:moref=3:hotadd:nbd], hex:[3ef44ed0]
[26.01.2016 20:55:51] <  6016> dsk| Entering in ~disk_handle_t()
[26.01.2016 20:55:51] <  6016> dsk| Closing disk in ~disk_handle_t()
[26.01.2016 20:55:51] <  4464> dsk|   Executing VDL_Close::Do()
[26.01.2016 20:55:51] <  4464> dsk|     VixDiskLib: VixDiskLib_Close: Close disk.\n
[26.01.2016 20:55:51] <  4464> dsk|     W32Util_CloseDismountHandle: Unlocking and closing handles for 0 volumes on PhysicalDrive6...\n
[26.01.2016 20:55:51] <  4464> dsk|     FILE: FileRemoveDirectoryRetry: Non-retriable error encountered (C:\Windows\TEMP\VeeamBackup\VeeamAgent-8032\vmware-SYSTEM\564d3fdb-8bd3-089f-2f53-63490704048d-3\hotadd\VmdkStubs): The directory is not empty (145)\n
[26.01.2016 20:55:51] <  4464> dsk|     FILE: FileRemoveDirectoryRetry: Non-retriable error encountered (C:\Windows\TEMP\VeeamBackup\VeeamAgent-8032\vmware-SYSTEM\564d3fdb-8bd3-089f-2f53-63490704048d-3\hotadd\VmdkStubs): The directory is not empty (145)\n
[26.01.2016 20:55:56] <  2836> dsk|     2016-01-26T20:55:56.294+01:00 error -[02836] [Originator@6876 sub=Default] TryReleaseDevice: DynLinkApi_SetupDiRemoveDeviceInterface failed. Error = 0xe000021b.\n
[26.01.2016 20:55:56] <  2836> dsk|     2016-01-26T20:55:56.294+01:00 error -[02836] [Originator@6876 sub=Default] TryReleaseDevice: DynLinkApi_SetupDiRemoveDeviceInterface failed. Error = 0xe000021b.\n

[26.01.2016 20:56:04] <  6016> dsk| Entering in ~disk_handle_t()
[26.01.2016 20:56:04] <  6016> dsk| Closing disk in ~disk_handle_t()
[26.01.2016 20:56:04] <  4464> dsk|   Executing VDL_Close::Do()
[26.01.2016 20:56:04] <  4464> dsk|     VixDiskLib: VixDiskLib_Close: Close disk.\n
[26.01.2016 20:56:04] <  4464> dsk|     W32Util_CloseDismountHandle: Unlocking and closing handles for 0 volumes on PhysicalDrive7...\n
[26.01.2016 20:56:09] <  2836> dsk|     2016-01-26T20:56:09.263+01:00 error -[02836] [Originator@6876 sub=Default] TryReleaseDevice: DynLinkApi_SetupDiRemoveDeviceInterface failed. Error = 0xe000021b.\n
Thanks for your help,

Johan
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Event 157: Disk X has been surprise removed.

Post by foggy »

This this behavior looks to be typical for Windows 2012 R2.
MrSpock
Service Provider
Posts: 46
Liked: 3 times
Joined: Apr 24, 2009 10:16 pm
Contact:

Re: Event 157: Disk X has been surprise removed.

Post by MrSpock »

That may explain the Windows log event, but why is Veeam logging an error while trying to remove the device? It doesn't look like expected behavior. I have to admit that the hotadd feature feels a bit unreliable. Is it possible to disable hotadd in Veeam Backup?

Johan
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Event 157: Disk X has been surprise removed.

Post by foggy »

This error comes from vSphere, when it detaches disk from the proxy server (Veeam B&R just translates vSphere logs for operations, where it calls vSphere to do something). So this is internal to VMware and tells nothing against hotadd reliability and overall Veeam B&R operation. You can trust hotadd, since it's proved its reliability over all 8 previous versions of Veeam B&R. Besides, as you can see from the provided link, the same occurs in case of Hyper-V.
Thernlund
Enthusiast
Posts: 29
Liked: 3 times
Joined: Sep 09, 2015 12:02 am
Full Name: Terry Hernlund
Contact:

Re: Event 157: Disk X has been surprise removed.

Post by Thernlund »

I'm seeing this behavior too. I checked Disk Management during a backup and it appears that Veeam mounts the VMDK files as offline disks to back them up. This error seems to appear in the event log when it disconnects the disk.

Is that an accurate representation? That is, Veeam does mount the VMDK files for a given VM during a backup?

As an aside, I use the term 'error' loosely. In reality, it's a warning, not an error. I'm happy to ignore warnings assuming my observations above are expected behavior.
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Event 157: Disk X has been surprise removed.

Post by veremin »

Is that an accurate representation?
Correct. Virtual disks of given VM are attached (hot-added) to a specific proxy server. Thanks.
Post Reply

Who is online

Users browsing this forum: No registered users and 74 guests