Veeam 7 Copy Job - Transform

[philscompany]

Hi there,

Just working out some issues with v7 at the moment and wonder if I'm doing this right. We have a nightly backup to a 4TB NAS for all our VMware VMs which works perfectly. I want off-site backups so have taken advantage of the new backup copy job for this. I have 5 rotating 2TB USB HDDs which are changed daily. The total size of the backup is about 1.4TB - without any additional restore points. The retain period on the NAS backup is 30 days. I have configured a copy job for this which should copy the NAS backup to the USB HDD but only keeping 2 restore points due to space constraints. It took a long time to finish but it did it - however, it then started a transform process which took hours and never got past 9% - I realised it had filled the drive as it was creating another file.

Can I stop it doing this transform procedure or is it an essential part of the process?

Many thanks,

Phil

[veremin]

Yep, transform operations are essential for backup copy job. In general, they are performed after the backup copying task and might include the following:

• Removing deleted VMs from restore points.
• Compacting a full backup file.
• Transformation of a backup chain.

The latter is worth being mentioned separately. In general, whenever a new restore point is transferred to target location, VB&R checks the backup copy job retention policy. If the specified number is exceeded, VB&R starts to transform backup chain to free up space for a new restore point.

Since the retention policy for backup copy job is 2, I believe, backup copy jobs, after each cycle, tries to transform backup chain in order release some space for the upcoming restore point(s).

Thanks.

[philscompany]

Understood - unfortunately this rather renders the job useless as there is nowhere near enough space on the USB HDDs for two copies of the file. Is there an alternative method I can use?

[veremin]

Actually, after initial synchronization backup copy job is forever incremental. Backup copy job doesn’t copy backup files as the whole, instead, it uses its special logic to create required restore points at target location. So, even though some space is certainly required at target location, it doesn’t necessarily means that you should have space to store two copies of backup data.

Thanks.

[philscompany]

OK, but we still have nowhere near enough room for the transform to run unfortunately - I'll have to find an alternative method for this I think.

Many thanks,

Phil

[veremin]

I’m still wondering why the processing took place even though the specified interval hadn't been exceeded yet. Do you have GFS rotation scheme or Compact full enabled? Thanks.

[philscompany]

Hi,

No, we don't have Compact Full enabled for this job. I must admit I don't know what GFS Rotation is - I haven't seen an option for that anywhere in the job settings, can you advise where I would find it?

Thanks,

Phil

[veremin]

Go to the settings of a backup copy job -> Target and see whether “Keep the following restore points for archival points” is ticked or not. It might have happened that the time specified for weekly backup coincided with the time of the latest backup copy run and backup copy job tried to create a new weekly backup.

Thanks.

[philscompany]

No, that box is not ticked.

I've also found that now I've come to swap to a new USB drive, the registry key ForceCreateMissingVBK doesn't seem to be working, I'm getting a series of 'file does not exist' VIB/VBK errors when trying to write to the blank drive.

[veremin]

Did you restart the VB&R service, or the whole server after applying the said regkey? Thanks.

[philscompany]

Yes, I did restart that service afterwards.

Thanks.

[veremin]

Then, I’d suggest contacting our support team, since this regkey is supposed to work with both backup copy and normal backup jobs. Thanks.

[philscompany]

I've logged a support call, they pointed me here:

http://www.veeam.com/kb1154

Which I hadn't seen before. As ours is a x64 server, I have placed the key in \HKEY_LOCAL_MACHINE\wow6432node\VeeaM\Veeam Backup and Replication\

I assume this is right? The KB article does not make it clear.

I restarted the backup service and now the job fails with this error:

27/08/2013 14:05:34 :: Error: Client error: File does not exist. File: [H:\Veeam\Backup Copy Job 1\Backup Copy Job 12013-08-22T000000.vib].
Failed to restore file from local backup. VFS link: [summary.xml]. Target file: [MemFs://Tar2Text]. CHMOD mask: [0].

I'm waiting for support to come back to me.

[foggy]

Phil, please specify the mentioned key under HKLM\SOFTWARE\VeeaM\Veeam Backup and Replication, restart the backup service, and see whether this helps. Thanks.

[michaelc]

Hi Foggy. I have just spoken to support and they have told me the backup copy job is only meant to work to fixed locations, not rotated media.
Previously in 6.5 and earlier we were using a robocopy job to synchronize the backups to the ~20 usb drives that are used in rotation but were hoping to have an inbuilt Veeam method for notification and possible performance/intelligence improvements (our first copy job completed at a rate that was significantly faster than robocopy)

So my question: Is the use of the ForceCreateMissingVBK registry key a supported option to allow veeam copy jobs to backup to rotated USB media? And are there any additional steps that need to be performed?

[Gostev]

So my question: Is the use of the ForceCreateMissingVBK registry key a supported option to allow veeam copy jobs to backup to rotated USB media? And are there any additional steps that need to be performed?

Yes, ForceCreateMissingVBK is supported for Backup Copy jobs, and no additional steps are required. The work has been done specifically to ensure this key applies to Backup Copy jobs as well as regular backup jobs, this was very recently (close to the release) and I still remember those discussions with R&D very well.

[foggy]

Yep, I've also talked to QC regarding that yesterday and they confirmed that the key works with backup copy jobs. The only caveat here (and, probably, this is what Phil has encountered above, as he was talking about VIBs) is that it works in cases where there were no incremental job passes performed prior to replacing the media. In other words, each media should contain a single VBK file only, before being replaced, no increments. Though, this behavior did not change since previous versions, including v6.5.

[philscompany]

I spoke to Veeam support who had a look at my jobs and said I should be using a straightforward backup, not a Backup Copy. They said the reg key was only designed for Backups, not Copy jobs. It's now working although when I attach a blank USB drive for the first time the job fails with an error still, however if you manually restart it, it works fine. Once the drives are all populated with a backup, I'm hoping it will be plain sailing.

This also removes the need for a Transform process so backups take only a couple of hours once the initial seed backup has taken place on each drive.

Thanks for your help, hope you get your problem fixed michaelc.

Cheers,

Phil

[michaelc]

Thanks Phil. We have actually gone back to using Robocopy exclusively and with the reduced backup window and changing over to a USB3 interface for the robocopy destination, things are working much better.
The advantage is we have the same data on the USB's as the Veeam backup server, so multiple restore points on each of the rotated drives.

[pendragoncrw]

With v7, I have wanted to move away from 3rd party sync tools to the backup copy job. Here's how I have my setup:

1. Primary backup storage is a Synology 8 drive NAS.
2. Plugged into the Synology we have USB drives that we rotate. For the previous solution each drive has the same share name (\\synologyname\external) and we simply copied the most recent VBK to it nightly.

After installing Veeam v7, I made sure to add the ForceCreateMissingVBK key to the Wow6432 node as indicated in the KB article and rebooted the server.

Our first week went perfectly. The initial sync took a while, subsequent days were fine.

After switching the drive yesterday, it all fell apart. The job copied the initial folder (named as the job), a 16mb vib file, the backup chain metadata file, and failed with a series of:
9/7/2013 11:53:03 AM :: Processing 'SERVERNAME' Error: Client error: File does not exist. File: [\\x.x.x.x\external\jobname\jobname2013-09-06T020000.vib].
Failed to restore file from local backup. VFS link: [summary.xml]. Target file: [MemFs://Tar2Text]. CHMOD mask: [0].

Shouldn't the regkey have forced the job to recreate the missing files? Am I using the backup copy job incorrectly?

I have opened a support case (00440974) with log files but wanted to check here first in case it was something I was overlooking or misinterpreting in the documentation.

Thanks,
Chris

[pendragoncrw]

Just heard from support. "Backup copy jobs can't be used for rotational media as it does not support that method."

I guess its back to my 3rd party tool.

Chris

[Vitaliy S.]

Chris, this key does work with backup copy jobs as well. Please review foggy's answer above for additional info.

[tsightler]

I can confirm that the key works, but would like to point out a few things.

First, the KB article has not been updated for V7 and in this version the key should be created in the regular hive, not the Wow6432 node. Second, as mentioned above, it will not work if any incremental runs have been created on the media, and there are a lot of things that can trigger the creation of an incremental, including doing things like running "Sync Now". This also means it won't work for people who do weekly rotations instead of daily. Also, you must make sure that the backups are deleted automatically using a scheduled script across holidays/weekends.

[pendragoncrw]

Thanks for the additional information Tsightler. I added the key to that node on a hunch before posting, but was still in the same boat.

When you say it won't work if any incremental runs have been created on the media, do you mean the source media or the destination media for the copy job?

Is there any way to get a weekly rotation to work with the backup copy job in Veeam?

Thanks,
Chris

[tsightler]

Basically, if there's a VIB file in the repository, then the registry key is not going to work. That's the easiest way to tell. As long as there's only a VBK file, it should work OK. I've tested it quite a bit. Remember you have to restart the Veeam service for the change to take effect.

I think the easiest way to use Backup Copy jobs with rotated media is to use a Powershell script that removes the backups from the the B&R console when the media is rotated. This causes the next job run to automatically start a full copy and it works with weekly backups/incrementals without any problems.

How do you handle clearing the media when you rotate it? Do you just manually delete the backup files or do you use a script that does it automatically? If you do it manually you can just go into the Veeam GUI, go to Backups...Disk, right click the Copy Job backup and choose "Remove from Backups" (not Remove from Disk). That way it's just like the copy job is starting over from scratch, which is exactly what you want since it's new blank media.

[pendragoncrw]

In the past, I used a sync utility that would just grab the vbk file (we're using reverse incrementals) and remove everything else on the target disk.

I will have to experiment with what you suggested (removing backups from the console)

After setting the regkey, etc. and clearing the media, I still got the errors when I ran a "sync now". Is sync now not the appropriate way to force an immediate test? Do I need to wait for the next copy window to come around to get it to behave correctly?

Thanks,
Chris

[tsightler]

Yes, Sync Now is a correct way. However, clearing the media is not enough. If you go to the backup in the GUI and click properties, you need to see only one single backup in the list. If there are more than one, then the reg key will not work. What I've seen in a lot of cases is that a user will create the initial job, and instead of waiting for that job to run, they immediately do a "Sync Now". This is really not required, and has the side effect of immediately create a VBK and VIB because the "Sync Now" forces a new job run, thus the original VBK is effectively empty and the VIB now contains all of the data. After this, the Regkey won't work.

Removing the backups from the console will always work and you don't even need the regkey. When I get some time I was thinking of creating a simple Powershell that you could put on the desktop. Each time you swap the media you simply run the Powershell and everything is ready to go. It could potentially even be smarter and automatically clear the media/Veeam console only if the media had actually been swapped, that way you could just schedule it via Powershell.

[pendragoncrw]

I know I would definitely pay for a PS utility that would do that. We try to automate things as much as possible for our clients and having to depend on a notification from them when they switch the drive to then login to the server and manage it.

Chris

[tsightler]

OK, so after spending a few cycles thinking about this I believe I've come up with a very simple script that can be useful for the vast majority of people wishing to use rotating media with the new Backup Copy jobs in V7. This script does not rely on the "ForceCreateMissingVBK" registry, although it doesn't hurt to have it.

The script is designed to be run as a post-job activity script of the Backup Copy job (available in the Advanced options of the job) and takes advantage of the fact that the "post-job" script on a Backup Copy job is actually not run until the copy interval expires, effectively making it the same as a "pre-job" script for the next interval. When the script runs it uses B&R Powershell to query the name of the base VBK file for the current active Backup Copy chain. It then checks to see if this VBK exist on the current media. If the file exist, the script simply exits since this means the media has not been changed since the last Backup Copy run so the next run will be an incremental. If the file does not exist, the script assumes the media has been changed and takes the following actions:

1. Deletes the backups from B&R using Powershell
2. Checks if any old backups exist on the media and, if so, deletes the entire folder for that backup

At this point when the new copy interval starts it will start a new chain with a new VBK. The script will work correctly for weekly rotations, or even if rotations are skipped, simply doing incremental runs on the days where media is not rotated. It can even be used for cases with more often than daily Backup Copies, for example if you run hourly Backup Copies but rotate media only daily.

The script has only a single variable, the name of the Backup Copy Job. Once this is set simply add the script to the post-job activity on the Backup Copy as so: powershell.exe "C:\path\to\script". The Backup Copy interval should probably be synchronized with the first backup job.

The only real negative I can see to this script as that it forces a complete full every time the media is rotated, but I think that's pretty common. It requires no interaction from the user at all since it handles both clearing old media and removing the entries from B&R. In my testing I was unable to break it except by doing things like manually removing the backups from the B&R console without also manually clearing the media.

I'd be very interested in anyone who is looking to use the new Backup Copy Jobs with rotated media testing this out and letting me know how it works. I have an idea for a more sophisticated version that leverages multiple jobs (one for each media) and supports incremental forever even when swapping media. Might be useful for those with slower USB devices, but I think this script should be significantly better than using third party tools.

Code: Select all

asnp VeeamPSSnapin

$jobname = "<Name_of_Backup_Copy_Job>"  # Name of Backup Copy Job

# Find full backup file and path for current Backup Copy Job
$backup = Get-VBRBackup -Name $jobname
$backupfilepath = ($backup.GetStorages() | ?{$_.GetStorageType() -eq "vbk"}).FilePath
$backupdirpath = ($backup.GetStorages() | ?{$_.GetStorageType() -eq "vbk"}).DirPath

# If current active VBK is found on media then media has not been rotated
if (-not(Test-Path "$backupfilepath")) {
    # Current active VBK was not found indicating media has been changed
    # Remove Backups from Veeam GUI
    $backup.Delete()
    # Remove Backup Files from Media
    if (Test-Path "$backupdirpath") {
        Remove-Item -Path "$backupdirpath" -Recurse
    }
}

[buwprof]

Code: Select all

asnp VeeamPSSnapin

$jobname = "<Name_of_Backup_Copy_Job>"  # Name of Backup Copy Job

# Find full backup file and path for current Backup Copy Job
$backup = Get-VBRBackup -Name $jobname
$backupfilepath = ($backup.GetStorages() | ?{$_.GetStorageType() -eq "vbk"}).FilePath
$backupdirpath = ($backup.GetStorages() | ?{$_.GetStorageType() -eq "vbk"}).DirPath

# If current active VBK is found on media then media has not been rotated
if (-not(Test-Path "$backupfilepath")) {
    # Current active VBK was not found indicating media has been changed
    # Remove Backups from Veeam GUI
    $backup.Delete()
    # Remove Backup Files from Media
    if (Test-Path "$backupdirpath") {
        Remove-Item -Path "$backupdirpath" -Recurse
    }
}

Hi there,

I have tested that one with a Test Backup. Worked ok, but i had some issues with a Copy Job that was continuous. So i changed a bit of that code. That prevents the creation of "my CopyJob_1" directories.

Code: Select all

# Import Veeam Snapin and we do not want Errors
asnp -Name VeeamPSSnapin -ErrorAction SilentlyContinue

# JobName here
$jobName          = "05 COPY Test"

# Get Backup Specs
$backup           = Get-VBRBackup -Name $jobName

# Get FilePath
$backupfilepath   = ($backup.GetStorages() | ?{$_.GetStorageType() -eq "vbk"}).FilePath

# Get Directory for deleting
$backupdirpath    = ($backup.GetStorages() | ?{$_.GetStorageType() -eq "vbk"}).DirPath

# If current active VBK is found on media then media has not been rotated
if (-not(Test-Path "$backupfilepath")) {
    # Current active VBK was not found indicating media has been changed
    # First DisableScheduler, If not then the Copy Job will Create a new Directory like $Jobname_1
    
    # Get-Job Object
    $CopyJob      = Get-VBRJob -Name $jobName
    
    # Disbale Scheduler
    $CopyJob.DisableScheduler()
    
    # then Wait until Job is Stopped (and is not running anymore)
    while (-not($CopyJob.IsStopped())) { 
        # Do Nothing 
    }
    
    # If stopped then Delete from VeeamGUI
    $backup.Delete()
    
    # Remove Backup Files from Media
    if (Test-Path "$backupdirpath") {
        # Remove whole Directory
        Remove-Item -Path "$backupdirpath" -Recurse
    }
    # Enable Scheduler
    $CopyJob.EnableScheduler()
}

I have added the stopping o the Copy Job and waiting till it's stopped. Then deleting everything. That worked fine for me.
I will give it a try on a small BackupJob.

Thanks a lot tsightler that was what i was looking at.