Feature request - the case for a copy mirror function?

[Moopere]

Hello Veeamers,

I've been struggling with a problem for a long while and I now feel like I've fiddled about to a point where I'm probably trying too hard to work around an inherent limitation - thus a feature request

When using slow repositories, in my case removeable USB disks, merges are killing me. The inherent slowness of these devices for the basic data transfer process is accepted, the killer is merges which effectively halve the throughput yet again (reading, then writing the merged files). What I've observed is thrashing of the external USB disk for periods of 16-24 hours or even more, depending on the size of the merge required. For backups of more than 1TB the merge window required is more than 24 hours, even with incrementals of 400-600GB, and this, of course, bumps into the common requirement to have a daily offsite backup - not to forget that I'm talking about the merge component of the backup to USB here, we also need time to copy across fresh incrementals - its only after the new incrementals are copied that the merge process begins.

Using both types of backup copy doesn't really make any difference. Pruning or mirroring, both require merges. To this extent, 'mirroring' really isn't mirroring at all.

What I think would be useful, probably for any number of cases, but certainly for copy jobs out to slow repositories, is a true 'mirror' copy. A copy option that copies from the specified source repository to the destination only and exactly what is in the source. Extra files that may exist in the destination would be deleted, as per usual mirroring functionality (think robocopy /mir)

I'm aware of: HKLM\Software\Veeam\Veeam Backup and Replication ForceDeleteBackupFiles

This I think provides the basic requirement, but its a sledgehammer in that when invoked it operates against _all_ repositories. Probably mostly fine, but I think it would be less potentially dangerous if it were a per job option.

Essentially, the copy job functionality that currently exists for Tape would be great. Theres no inherent merging process when dealing with tape jobs. You present your air-gap media, copy files (or backups) to it, remove the media, done .. and as quick as your source and destination (and transport) allow.

[HannesK]

Hello,

merges are killing me.

REFS / XFS should fix that. https://www.veeam.com/blog/advanced-ref ... suite.html As alternative active full backup copy jobs can be used. (see next link)

Pruning or mirroring, both require merges.

pruning with "read the entire restore point..." and GFS set does not merge https://helpcenter.veeam.com/docs/backu ... ml?ver=100

I think "read the entire restore point..." does what you mean with "like tape copy".

Best regards,
Hannes

[mweissen13]

Hello!
I am in the same boat as Moopere.

Regarding ReFS or XFS. I tried that, but the performance of those with USB disks is hardly usable in practice. Also, both file systems were not really designed for USB disks (which could be disconnected without warning) but for a proper stable server environment with UPS and Battery Backed RAID in place.

Also when using the "This repository is backed by rotated hard drives" option GFS is not available in Copy Jobs. And also the "Read the entire restore point from source backup insted..." option is greyed out.

To sum it up, the support for backups on external USB disks is currently rudimentary at best in practice.

https://imgur.com/a/6qydrAC

[HannesK]

Also, both file systems were not really designed for USB disks

well good point. I only suggested it, because I use it since many years for my small home-lab. But you are right

option is greyed out.

unfortunately 2:0 for you. I checked it with V11 beta and it is a UI issue of the beta version.

[Moopere]

I've played with this a fair bit. I really don't believe there is a reasonable work-around to the problem of merges on slow storage.

A Robocopy script would almost certainly provide the desired result, but the problem with USB disks is that they can, and do, change drive letter when removed and reinserted - so I'd have to be pretty fancy with my script to determine the drive letter and voracity of the inserted media.

The option to use the registry hack: "HKLM\Software\Veeam\Veeam Backup and Replication ForceDeleteBackupFiles" will probably mostly work - so long as the USB disks really are changed on every backup cycle. If the change is missed then the file deletion invoked by "ForceDeleteBackupFiles" won't occur (as the backup chain isn't broken) and we're in merge hell again.

The thing is, the infrastructure is already there in Veeam. Tape copies of jobs (or files) works in the way you'd want your removeable disks to work as well. Tape copies drop user selected files (or backup jobs) onto tape ... just that, no merging no messing. Ideally there would be an option to do the same with USB disks.

[tdewin]

Here is some sample code. Make a folder "uuid" and in this folder a file "uuid.txt". Paste in the content of the variable $lookfor (for example d:\uuid\uuid.txt contains "adfdca28-12ad-11eb-adc1-0242ac120002"). You can have different uuid on different disk for different scripts

Code: Select all

$lookfor = "adfdca28-12ad-11eb-adc1-0242ac120002"
$vols = Get-Volume | ? { $_.DriveLetter -ne $null }
$disk = $null
foreach($vol in $vols) {
    $uuidpath = ("{0}:\{1}" -f $vol.driveletter,"uuid\uuid.txt")
    if (Test-Path $uuidpath) {
        $content = (get-content $uuidpath).Trim()
        if ($content -eq $lookfor) {
            $disk = $vol.DriveLetter 
        }
    }
}

if ($disk -ne $null) {
    write-host "found disk $disk"
} else {
    write-host "couldn't find the disk with uuid"
}

Alternatively, you could also look for the filesystem label but that is not necessarily unique

Code: Select all

Get-Volume | ? { $_.FileSystemLabel -eq "my drive"  }

Or the unique label but not sure if they are consistent across reboots

Code: Select all

# first find the uuid by get-volume | select driveletter,uniqueid
Get-Volume | ? { $_.UniqueId -eq "my uuid"  }

[Moopere]

@tdewin - thanks for the code, this has given me something to think about. I was previously really concentrating on drive letters without looking at options to make an external script far more robust - like using a group of UUID's or perhaps even grouping the external media by way of a common disk name.

[Moopere]

Thinking about this some more. I appreciate and am going to follow up work arounds, including scripting - I think the call-to-action as a feature request still stands though. This problem will be an issue I'd imagine for any repository that suffers low IOPS. There could be any number of use scenarios where this will be a problem, not only SMB users running cheap USB disks.

It feels like most, maybe all the functionality being asked for already exists in Veeam, its just a matter of making it available to rotated media in the copy job setup wizard. In particular:

- There is already a method for determining if a rotated disk is known to Veeam.
- The ability to use only source data, rather than synthesizing, is available to GFS backups. Imagine if we had such an option for _every_ copy backup job
- It would be great if rotated media could benefit from being part of a pool. At the moment, rotated media seems to be any ad-hoc media which Veeam has seen before ... So the concept of retention based upon pool membership is not available. This would go a long way towards building some sound logic to allow the erasure of old (expired) media without having to kludge a hacked mass delete script to keep the media clean. Nevertheless, Veeam can already recognise, somehow, the media to be used (the logic which allows the changing of drive letters) - so an option to clear the media (mass delete) to clean it before the copy job would keep things running cleanly over time.

In the meantime:
- Resigning myself to having to write a script to manage rotated media
- Although initially I thought Robocopy would be deployed here, I'm now thinking I just need to find the media thats inserted (thanks@tdewin, your code sample is making me think), delete the media entirely and then utilise the Veeam copy backup job in the normal way. Imagine if there was a tick box in the copy job wizard that only appeared for rotated media and allowed one to delete the media first.
- I suspect that the registry hack I was going employ won't be needed. If so, that will be great as it makes me nervous because it applies to _all_ repositories.


**[EDIT]**

Oh, and not needing to synthesize to rotated media would also allow the use of media which is roughly sized to the primary backup. At the moment, I believe its written in the docs, because of the way synthesizing works, one needs to provision roughly double the available space on the media as the primary backup will require.

[rayc@567.net.au]

There is a great little utility to fix the drive letter for external usb disks.
https://www.uwe-sieber.de/usbdlm_e.html
Have used this with many backup programs for years.
Lots of options, including using the internal UID of the disk or reading from volume label.
Enjoy.
Ray

[ITP-Stan]

I have a similar issue with a customer that uses multiple small 1-disk NAS devices for offsite copies.
What I do now is run a pre-job script that just deletes the files already on the disk.
That way when the actual backup-copy starts there are no old VBK or VIB files left, and it just starts to copy from scratch (with the rotated drives checkbox set).

One problem for us remains: Even copying 1 full takes about 3 days (because of the size, and the slow 1-disk NAS used).
So it's effectively a weekly offsite now.

We are now trying to convince the customer to move to CloudConnect so he can have daily offsite backup's.

[Moopere]

There is a great little utility to fix the drive letter for external usb disks.
https://www.uwe-sieber.de/usbdlm_e.html
Have used this with many backup programs for years.
Lots of options, including using the internal UID of the disk or reading from volume label.
Enjoy.
Ray

Mmm. @rayc This is a really valuable heads-up for anyone suffering this problem. Thank you for mentioning it

Initially I was cautious; any number of things might happen and cause problems with the 'auto delete' script I'd have to employ. However, this utility has a few ways to really help to reduce the likelihood of inadvertently deleting a USB connected drive that should be ignored by the backup system. Probably the volume-serial or volume name is going to work best (or both!).

At the moment I'm trialing it and my only simplistic safeguard is that the delete script I use only deletes the contents of the "\veeam" directory on the drive letter of choice.

This utility fixes another problem with Veeam, rotating drives and USB devices - changing drive letters is fine for Veeams basic functionality, but it only recognizes the disk when its written something to it. So, setting up 1-2 weeks worth of media, one per day, is quite a job - you have to insert the media and have Veeam write something to it (a small backup job) otherwise the media isn't recognised next time when the drive letter changes.

Using the utility which always presents the USB drive as the same drive letter fixes this entirely. The on-site backup manager simply places whatever disks I provide them, pulling from the top of a stack of new drives for example, without me needing to jump through hoops to prepare the USB disks. In the future I'll gain and utilise their individual volume serial numbers and integrate that into the utilities config file for added protection - but its still pretty trivial.

So now, during trials of this workaround I have the following benefits:

- No long winded preparation of the USB disks to be used
- A known window of time for the offsite copy of the backup. Yes, its longer than it probably needs to be 90% of the time, but its a known quantity that doesn't break when corner case situations arise
- I don't need to provide disks that are essentially double the size of the largest anticipated backup - when merging one needs to provide significant additional storage space to allow the merge to complete.... up to double the final size of the backup.

For mine, I still believe that there is enough interest in this scenario to warrant an active feature-request. I did my homework before posting initially - there are a _lot_ of posts relating to this or very similar problems that people are having. In essence, its treating a removable disk based media in the same way as a tape. So called 'air gap' backups are rising to the surface in industry blurb due to problems mostly relating to ransomware, so, whilst traditional tape use might might be declining, the need for air-gapped media that works easily and repeatably for end-users still exists

[Moopere]

So I've done a few days testing. Its working better than I would have initially hoped.

However, theres a rather nasty caveat that I'll have to think about.

Because rotated media isn't part of a retention pool, there are no rules relating to retaining data on them. The problem arises that backup duplication via copy-mirror or copy-prune are 'continuous' jobs. So, regardless of the scheduled and allowable data transfer times, or the 'starting at' time (in the case of prune), the job runs whenever the backup server starts, or the jobs starts (or restarts) or if the media is changed. After the job starts the pre-job scripts runs, blanks the media, then waits until the specified schedule and start timing ... at that point the scripts runs again.

Care will need to be taken:

- Remove the backup copy media as soon as practical after backup copy completes. Definitely before any modification to the backup job itself (as this invokes a restart of the job)
- When reinserting media to restore, ensure that any backup copy jobs are disabled - otherwise they will sense the media change and immediately blank the contents (!!!)

Would be nice if the pre-job script ran just before the actual job itself processed data rather than immediately. I guess there might be use cases for how it works now, but I can't think of any off the top of my head.

[Moopere]

The problem of copy jobs being 'continuous' and that pre-run scripts run at job start as well as at data transfer start is turning out to be a problem.

If a backup copy job completes then if the server shuts down (power failure for example) then restarts, the backup copy is deleted when the copy job restarts (as its continuous).

I might have to trigger the delete script from outside of Veeam which is just a management/maintenance pain. Or start rummaging around in Veeams powershell to see if I can find some way to determine if the job has started because Veeam services have started or if its actually ready to do some data movement.

Or perhaps I can have an end-of-job script drop a marker file onto the removeable disk with a timestamp in it or something to indicate a 'do-not-delete-before' time.