Virtual Appliance Transport mode disables automount

[rday]

We have a small environment:
- single ESXi server with 6 VMs, all storage is local hard drives on the ESXi server.
- one of the VMs is the Veeam server, configured as Backup Proxy and Backup Repository, and everything else Veeam.
- primary repository is on some slower local mounted disks, mounted on the Veeam server as E: drive
- second repository is (rotating) USB HDD, directly connected to the ESXi, with USB passthrough to the Veeam server VM, and mounted on Veeam server as F: drive.
- Backup job backs up all the VMs, including C: Drive of Veeam server, to the primary repository, and a Backup copy job then copies the backup to the USB

This setup has worked quite well, with a daily swap of the USBs without anyone having to login to the servers.

However, I recently upgraded from Veeam B&R 9u4 to Veeam B&R 10 (no other changes) and since then the USB drives have not been automounting. I spent several days trying to find out what was happening. Eventually I found out that the Backup job was, somehow, disabling automount (diskpart > automount) everytime it ran.
I found one line in the Veeam B&R user guide under Backup Proxy > Transport Modes > Virtual Appliance > Limitations for the Virtual Appliance mode
[For Microsoft Windows proxy] Before running a data protection task, Veeam Backup & Replication disables the volume automount feature.
For Veeam B&R 9u4, this is not mentioned.
There is no hint anywhere that this is a change between V10 and previous versions.

So I have changed the Backup Proxy Transport Mode setting from "Automatic Selection" to "Direct storage access" which seems to have stopped the backup jobs disabling automount.

My questions:
- is this the best way to cope with this dubious new "feature"?
- Is Direct storage access the best Transport mode for us?
- why did Veeam do this? And especially why did they not mention it in the release notes, as it is a significant change from previous versions.

Thanks

[Andreas Neufert]

There are 2 settings influencing how Windows enable and mount volumes.
SAN
https://docs.microsoft.com/de-de/window ... mmands/san
and Automount
https://docs.microsoft.com/en-us/previo ... v%3Dws.11)

SAN handles if the volumes goes "online" automatically in the Windows diskmanager.
Automount (if enabled) takes any "online" volume and mount any partition as drive letter in windows. On the same process it writes a signature to the disk/volume.

We disable both to avoid that windows overwrite any data at backup when we mount the VM disks for backup. All Veeam versions disable this for the Proxy configurations independantly from the transport protocol that you use.

In your environment you need to make sure that the SAN policy bring only USB volumes online automatically so that enabled automount can mount those volumes.

[rday]

Hi Andeas,

Thank you for this information.

For your information, the SAN Policy was already "Offline Shared" as you suggest, and the USB HDDs in question have all previously been mounted and the disk signature written. This issue is occurring on a subsequent mount of the USB HDDs.

If "All Veeam versions disable this for the Proxy configurations independantly from the transport protocol that you use":
- why did Veeam B&R PRIOR to v10 not disable Automount?
- why does changing the Transport mode for Backup Proxies in Veeam B&R 10 to anything other than Virtual Appliance mode, stop Veeam from disabling Automount?

So, what is the Veeam recommended way of allowing USB HDDs that are part of a Rotating Backup Repository to 'automount' on Windows servers that are also Backup Proxies?
And why did Veeam not advise of this change for Virtual Appliance Transport mode for Veeam B&R 10?

Thanks
Rick

[Andreas Neufert]

I is no a change in the Virtual Appliance mode. Whenever you install a proxy server or the Backup server itself which has a Proxy role installed we protect the processing by changing the way that windows mounts or enabled disks. This depends on Windows Version and Veeam version and is done to protect you from having issues because of it. In your case you need to leave the SAN policy as is and enable automount.

Why this affected you is, that we likely enforce the setting again if you do major version updates.

[BrentO]

I've also been running this way for YEARS with several small business customers. I've spent the better half of a week trying to figure this out too and I agree with the OP that you can enable automount but it gets disable again.

Why even have the "backed by rotating drives" setting if this is going to be the way it is handled going forward?

These are small customers backing up 3-5 VMs. Hard to have 2 VMs dedicated for backup when they only have 3 servers for production.

[Andreas Neufert]

Is there any support case with logs that I can escalate ?

[Gostev]

Actually, it's a known change in v10... I'm well aware of it. This was done to avoid data loss with hot add transport mode due to some Windows bug/peculiarity with its automount logic. This was thoroughly investigated last year including with Microsoft R&D (took many months), and unfortunately what is implemented in v10 is the only possible workaround.

However, for such small environments as mentioned above (just a few VMs), it should not be a problem to switch to the network transport mode?

[Paolo Lenzini]

My situation is also similar to the one mentioned above. With version 9.5 everything was fine, while 10, every time it backs up, disables automount. I solved it with a workaround: I created a batch file that re-enables automount and I scheduled it, in this way when I change the disk automount is active, then when it does the backup it is deactivated again. The content of the batch file is: "diskpart / s Automount.txt" and that of the Automount.txt file is: "automount enable"

[jeri]

Actually, it's a known change in v10... I'm well aware of it. This was done to avoid data loss with hot add transport mode due to some Windows bug/peculiarity with its automount logic. This was thoroughly investigated last year including with Microsoft R&D (took many months), and unfortunately what is implemented in v10 is the only possible workaround.

However, for such small environments as mentioned above (just a few VMs), it should not be a problem to switch to the network transport mode?

is the "problem" fixed in Version 10a RTM (build 10.0.1.4854) on July 21, 2020 ?

[Gostev]

This change was purposely made to prevent a data loss scenario with hot add transport mode on Windows-based backup proxy servers. In other words, it's not a bug/problem waiting to be fixed.

[jeri]

okay thanks gostev.

But I solved the Problem
with this tool: https://www.uwe-sieber.de/usbdlm.html

[Gostev]

Excellent tool, thanks for sharing!

[osag2020]

This change was purposely made to prevent a data loss scenario with hot add transport mode on Windows-based backup proxy servers. In other words, it's not a bug/problem waiting to be fixed.

We got the same issue. When you change a default behavior, so why not add an option for this? So users like us can disable this function, if needed?

We made a little powershell script, which checks for the driver letter and manually mount the USB-HDD. Seems to work fine for us.

[Gostev]

We don't give our users options that lead to data loss and unrecoverable backups.

[mkretzer]

We were the customer with the initial data loss by this. Sorry but Gostev is 100 % right.

The issue would have never been detected in our system if we would not have done regular *manual* restore tests. I don't even know if surebackup would have detected this as the system always booted up (system disk was never affected). It only corrupted volumes in specific situatuation - dedup on, large volumes for example. The backup file (the vbk and so on) was 100 % ok as the issue happens at the reading of the data. Sadly one of our most important fileserver matched this prerequisites and nearly all our restore points of this server were corrupted.

Imagine you are in the situation that you have to restore and you find out all your points on all your copies and tapes are corrupted because at some point the proxy had automount on. BTW active full does not help here!

[osag2020]

We don't give our users options that lead to data loss and unrecoverable backups.

What would be the best practive for swapping external backup targets for backup copy jobs? Maybe something can be configured differently, so we don't "stumble" over this.
The script is currently working fine for us, but it seems to be a "dirty" solution.

[Gostev]

This should not be relevant to you at all, as backup copy jobs target backup repositories, while discussed here is a [virtual] backup proxy. These two are separate and different components.

[osag2020]

We do have a normal backup job, whichs seems to disable automount on the system. But after this backup job, we also do a backup copy job on the same system. Maybe we should create ticket for this?

[veremin]

What do you mean by the same system? Machine that runs a Virtual proxy server is the same machine that has rotated drive attached (pass-through or something)? But since forum does not seem to clarify your deployment questions, it might be better to reach support team indeed. Thanks!

[fnt]

Simple solution without scripts or tools:

Assign each of your external drives an individual driveletter in the upper range like R:, S: and T:
These drives are then 'known mount points' as you can see in HKLM\SYSTEM\MountedDevices
Windows will mount these known drives even is automount is disabled.
Veeam recognizes the changed drive letter and will backup without issues.