Event 157: Disk X has been surprise removed.

[MrSpock]

Hi,

I get a lot of event 157 (Disk X has been surprise removed.) in the event log on my Veeam Backup server (Windows Server 2012 R2) during backup jobs. Why is this happening?



Thanks,

Johan

[foggy]

Johan, does your backup server act as hotadd proxy during your jobs?

[MrSpock]

Hi, foggy.

Yes, I suppose so, but not deliberately. The backup server is virtual and located on a host with local storage. The backup server is backing up VMs located both on the same host and on other hosts. There is no SAN involved, just local storage. The action logs say that "hotadd" is used on the VMs located on the same host, otherwise "nbd". Does Veeam Backup release the hotadded disks improperly?

Johan

[foggy]

Do those events always correspond to the time when the job finishes processing some VM disk using hotadd (according to the job session log)?

[MrSpock]

Hi,

Yes, I have searched through all logs from the last backups session. I can see that this error is always logged by Veeam Backup one second before event 157 is written to the Windows event log.

Code: Select all

error -[05124] [Originator@6876 sub=Default] TryReleaseDevice: DynLinkApi_SetupDiRemoveDeviceInterface failed. Error = 0xe000021b.\n

I didn't find 0xe000021b anywhere else in the logs.

Bigger snippet:

Code: Select all

[26.01.2016 20:55:51] <  6016> cli| -------------------------------------------------------------------------------
[26.01.2016 20:55:51] <  6016> cli| HotAdd.DetachDisks
[26.01.2016 20:55:51] <  6016> cli| - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - - -
[26.01.2016 20:55:51] <  6016> dsk| Removing attached vmdk disks.
[26.01.2016 20:55:51] <  6016> dsk| Connection cookie: [<hostname>:moref=3:hotadd:nbd], hex:[3ef44ed0]
[26.01.2016 20:55:51] <  6016> dsk| Connection cookie: [<hostname>:moref=3:hotadd:nbd], hex:[3ef44ed0]
[26.01.2016 20:55:51] <  6016> dsk| Entering in ~disk_handle_t()
[26.01.2016 20:55:51] <  6016> dsk| Closing disk in ~disk_handle_t()
[26.01.2016 20:55:51] <  4464> dsk|   Executing VDL_Close::Do()
[26.01.2016 20:55:51] <  4464> dsk|     VixDiskLib: VixDiskLib_Close: Close disk.\n
[26.01.2016 20:55:51] <  4464> dsk|     W32Util_CloseDismountHandle: Unlocking and closing handles for 0 volumes on PhysicalDrive6...\n
[26.01.2016 20:55:51] <  4464> dsk|     FILE: FileRemoveDirectoryRetry: Non-retriable error encountered (C:\Windows\TEMP\VeeamBackup\VeeamAgent-8032\vmware-SYSTEM\564d3fdb-8bd3-089f-2f53-63490704048d-3\hotadd\VmdkStubs): The directory is not empty (145)\n
[26.01.2016 20:55:51] <  4464> dsk|     FILE: FileRemoveDirectoryRetry: Non-retriable error encountered (C:\Windows\TEMP\VeeamBackup\VeeamAgent-8032\vmware-SYSTEM\564d3fdb-8bd3-089f-2f53-63490704048d-3\hotadd\VmdkStubs): The directory is not empty (145)\n
[26.01.2016 20:55:56] <  2836> dsk|     2016-01-26T20:55:56.294+01:00 error -[02836] [Originator@6876 sub=Default] TryReleaseDevice: DynLinkApi_SetupDiRemoveDeviceInterface failed. Error = 0xe000021b.\n
[26.01.2016 20:55:56] <  2836> dsk|     2016-01-26T20:55:56.294+01:00 error -[02836] [Originator@6876 sub=Default] TryReleaseDevice: DynLinkApi_SetupDiRemoveDeviceInterface failed. Error = 0xe000021b.\n

[26.01.2016 20:56:04] <  6016> dsk| Entering in ~disk_handle_t()
[26.01.2016 20:56:04] <  6016> dsk| Closing disk in ~disk_handle_t()
[26.01.2016 20:56:04] <  4464> dsk|   Executing VDL_Close::Do()
[26.01.2016 20:56:04] <  4464> dsk|     VixDiskLib: VixDiskLib_Close: Close disk.\n
[26.01.2016 20:56:04] <  4464> dsk|     W32Util_CloseDismountHandle: Unlocking and closing handles for 0 volumes on PhysicalDrive7...\n
[26.01.2016 20:56:09] <  2836> dsk|     2016-01-26T20:56:09.263+01:00 error -[02836] [Originator@6876 sub=Default] TryReleaseDevice: DynLinkApi_SetupDiRemoveDeviceInterface failed. Error = 0xe000021b.\n

Thanks for your help,

Johan

[foggy]

This this behavior looks to be typical for Windows 2012 R2.

[MrSpock]

That may explain the Windows log event, but why is Veeam logging an error while trying to remove the device? It doesn't look like expected behavior. I have to admit that the hotadd feature feels a bit unreliable. Is it possible to disable hotadd in Veeam Backup?

Johan

[foggy]

This error comes from vSphere, when it detaches disk from the proxy server (Veeam B&R just translates vSphere logs for operations, where it calls vSphere to do something). So this is internal to VMware and tells nothing against hotadd reliability and overall Veeam B&R operation. You can trust hotadd, since it's proved its reliability over all 8 previous versions of Veeam B&R. Besides, as you can see from the provided link, the same occurs in case of Hyper-V.

[Thernlund]

I'm seeing this behavior too. I checked Disk Management during a backup and it appears that Veeam mounts the VMDK files as offline disks to back them up. This error seems to appear in the event log when it disconnects the disk.

Is that an accurate representation? That is, Veeam does mount the VMDK files for a given VM during a backup?

As an aside, I use the term 'error' loosely. In reality, it's a warning, not an error. I'm happy to ignore warnings assuming my observations above are expected behavior.

[veremin]

Is that an accurate representation?

Correct. Virtual disks of given VM are attached (hot-added) to a specific proxy server. Thanks.