hotadd disks visible in Disk Manager on Proxy

[natalron]

Hi All,

I have an issue in that I am using a Veeam backup proxy for hotadd backup.

But some of my target VMs are visible in the proxy servers disk manager as mounted drives after the hotadd.

This then triggers alerts in my monitoring systems for disk threshold breeches. etc.

Most VMs disks appear in the proxy server as "offline" disks, but some actually are mounted and assigned drive letters.

Trying to determine why this is the case ...

Thanks in Advance,

Nat

[foggy]

Ronnie, having disks of the backed up VM attached (I believe, in read-only mode) to the proxy server is completely expected behavior. This is just how hotadd mode works. Are you saying that those disks remain attached to the proxy after the backup completes?

[natalron]

Hi Alexander,

Thanks for the reply. What I mean is .... that always the disks of the backed up VM attach to the proxy server, which I understand is normal.

But mainly the drives are shown in the proxy server disk manager as "offline" disks ... but for some VMs they attach and mount with a drive letter .. then my monitoring software detects the drive added and sends a disk threshold alert.

Does that make sense?

Thank you.

Cheers,

Nat

[veremin]

I believe that you'd better contact VMware team with this issue, since the Hot Add functionality is orchestrated solely by VMware, while, VB&R just appropriates it in order to backup VMs. Thanks.

[natalron]

Thanks. I might try logging a case with VMware.

[Vitaliy S.]

Ronnie, can you please update this topic with the reason why some disks are online and some are offline during hotadd process? Meanwhile, I will try to reproduce it my labs. Thanks!

[natalron]

Yes. I can add more details after I have spoken to VMware and determined why some disks are offline and some online during the backup process. Cheers.

[arsprod]

I've been having one vm show up now daily with multiple hard drives attached that shouldn't be. The vcenter db looks fine as does everything else so I suspected veeam might be involved. Sure enough, as soon as one of my replication jobs starts this vm (not the one replicating) reconfigures itself and adds hard drives. The vm that adds drives is a veeam proxy. Anyone else see this bizarre behavior?

[foggy]

Aaron, this bizarre behavior is nothing but hotadd transport mode in action.

[arsprod]

Oh... well don't I feel stupid! I'm assuming this is new to 7 since I've not seen it before? Do the disks stay there or supposed to be removed once the job is finished? Thanks

[Vitaliy S.]

HotAdd mode is not new to v7 and disks are not supposed to be left on the proxy server after job finishes. If you see otherwise, I would recommend investigating this issue with the support team.

[arsprod]

Thanks!

[Guido]

There are 11 disks connected to my proxy backup server. No backups are running at this time. Some disks belong to other production vm's and some from replica's.
How can this happen and how to resolve?

[foggy]

Guido, most likely you're backing up in hotadd mode, so disks attachment to the proxy server is totally expected (see above for details). However what is not expected is that the attached disks are not removed after the backup, so I suggest contacting support to investigate that. Thanks.

[Peejay62]

Every now and then I notice that disks processed with hotadd are not released from a proxy. Is this something to live with or should this always work, releasing the disks after the backup completes?
Just as a workaround we made a script to check the proxys if they have "unwanted disks" mounted and release them if appropriate. I have been digging in some logfiles to see if there are any messages on not releasing disk, cannot find anything though.
SO any experience or suggestions on this?

thanks, Peter

[foggy]

Peter, this is definitely not expected, so please contact support for investigation. Thanks.

[Peejay62]

Ok, thanks, will do that.

btw, shouldn't I receive a message when a subject is merged or a reply is given? my post was merged into this one but took me some time to figure that out (did I post it???, put me in doubt..), didn't receive a notification or somethng..

[foggy]

You should receive a notification on the new post in the thread in case you're subscribed to it.

[iwik]

Hi All,

I have an issue in that I am using a Veeam backup proxy for hotadd backup.

But some of my target VMs are visible in the proxy servers disk manager as mounted drives after the hotadd.

This then triggers alerts in my monitoring systems for disk threshold breeches. etc.

Most VMs disks appear in the proxy server as "offline" disks, but some actually are mounted and assigned drive letters.

Trying to determine why this is the case ...

Thanks in Advance,

Nat

Hi,

today same thing happened to us. During backup our monitoring discovered new drives on veeam proxy server. This drives had assigned letters and according to monitoring, windows was able to detect occupied space on this disks.
I know, that is OK that disks are added and visible in disk manager, but I'm not sure about that driver assigment. Even worse, when I check the event viewer:

Code: Select all

Log Name:      System
Source:        Ntfs
Description:
The file system structure on the disk is corrupt and unusable. Please run the chkdsk utility on the volume F:.

Logs shows, this error is not new and it is repeating. Can veeam confirm this is "feature"?

[Vitaliy S.]

Can you please tell us what configuration is set for automount and SAN POLICY settings on the proxy server?

[iwik]

Ok, got it:

Code: Select all

DISKPART> SAN
SAN Policy  : Offline Shared
DISKPART> automount
Automatic mounting of new volumes enabled.
DISKPART>

I guess I need to disable automount, right?

[foggy]

Yes, you can disable automount (unless the proxy is also a DC) to avoid this in future.

[pidthepiper]

Hi there,
#01822452 case ID

So I have come across this issue, the jobs complete successfully but when I look at the VM properties of the proxy, it still has multiple vmdks attached, which means the source vms are running on snapshots.

I noticed it the other day, and noticed it today. I am having to manually remove the vmdks from the proxy myself.

Has anyone had this issue before?

Veeam v8 Update 3 and ESXi 5.5u3b EP10 (latest)

[widmerkarl]

Hi,

As stated in KB 1775 (https://www.veeam.com/kb1775) the cause can be the VMware API call to remove the VMDK which was not received or did not complete properly.

Did support respond in the meantime?

How about updating Veeam to latest version 9 update 1?

[pidthepiper]

Yep Support have given me that KB and https://www.veeam.com/kb1882

My question is why would I disable automount?

v9 is on the cards, but there is no real reason for me to upgrade just yet, as v8 is fully supported

[widmerkarl]

Have a look at this Microsoft article regarding AutoMount:

https://technet.microsoft.com/en-us/lib ... s.11).aspx

When enabled (the default), Windows automatically mounts the file system for a new basic or dynamic volume when it is added to the system, and then assigns a drive letter to the volume.

The proxy with hot-add does not need to mount the filesystem as far as i know. The proxy just adds the VMDK to itself to read the block based data out of it, for backing it up to the repository.

[pidthepiper]

The proxy is a 2012 r2 vm

I will disable automout and see what happens.

You are my new hero kawiMTF! thank you for your quick response

[widmerkarl]

Never mind. I help where I can

[pidthepiper]

So I disabled auto mount and the issue still persists

I have notified support of this, but I am wondering is there anything else I can do?

[pidthepiper]

Every now and then I notice that disks processed with hotadd are not released from a proxy. Is this something to live with or should this always work, releasing the disks after the backup completes?
Just as a workaround we made a script to check the proxys if they have "unwanted disks" mounted and release them if appropriate. I have been digging in some logfiles to see if there are any messages on not releasing disk, cannot find anything though.
SO any experience or suggestions on this?

thanks, Peter

Could you share this script. My Main Veeam VM doesn't suffer from this issue, but my proxies do, every morning they had disks still attached and I have to un attach them from the proxy and then run disk consolidation on the actual VMs that were backed up. This shouldn't be happening but it is, but only on the proxies!

I have created a new proxy but I get the same issue, I find it odd that only the proxies suffer from this and not the main Veeam VM. I am running v8 update 3, and I am going to upgrade to v9 now to see if that helps at all. I set the proxies to nbd mode which solved the issue, but now there is an issue with nbd mode: vmware-vsphere-f24/repeted-vddk-error-1 ... ml#p197816 so I am back to trying to fix this issue