v7 Backup Copy Job removable media (rotated drives)

[farid.shahab]

Kindly take a look at the answer in this existing discussion. Thanks.

Hi Eremin,

Can you share this topic with me please? I am having the same problem. the RDX removable disk is not being recognized at the repository selection. thanks. Farid

[farid.shahab]

Use Windows Disk Manager to create a NTFS Partition and select a drive letter for this. Windows will remember the drive letter.

I think it is not about the drive letter, it is because Veeam doesn't recognise removable device as a repository backup, veeam only accept local disks for its repository. please have a slution for this. Farid

[Gostev]

Veeam doesn't recognise removable device as a repository backup, veeam only accept local disks for its repository

That is certainly not true. Plugged-in removable drive is no different from local disk.

[bunger]

Gostev - I have an open case with support ( currently escalated to Senior Support ) as the regkey and hotfix don't seem to fix the issue for our environment.

While I am awaiting a fix or "next steps" from them, is there a workaround I could use? Would the scripts previously mentioned do the trick?

[foggy]

Bill, what is the actual scenario that doesn't work?

[bunger]

Bill, what is the actual scenario that doesn't work?

I have a backup job that backs up VMs to an internal hard drive. I then has an associated Backup Copy job that should copy the latest backup to a swappable external hard drive. The Backup Copy job is failing with the following error:


1/6/2014 3:43:29 PM :: Error: The system cannot find the path specified.

CreateDirectory(A:\Copy Backup to Rotating) function call failed.
Cannot create folder. Folder path: [A:\Copy Backup to Rotating].
--tr:FC: Failed to create directory. Directory path: [A:\Copy Backup to Rotating].
--tr:Failed to call DoRpc. CmdName: [FcCreateDir] inParam: [<InputArguments><DirName value="A:\Copy Backup to Rotating" /></InputArguments>].

We are running v7 R2, have the registry key defined, and the hot fix installed. Still no go...

[veremin]

Hi, Bill, I'm wondering whether these scripts can solve your issues with using backup copy job in conjunction with rotated drives. Thanks.

[bunger]

I just heard back from Senior Support and don't feel like I am getting very far. Here is what they have to say:

We have additionally discussed this with R&D team.
As it turned out ForceCreateMissingVBK was initially designed to be used for regular backup jobs that use reversed incremental method.
For backup copy it would also work if there is only single .vbk and no .vib files. If there was a backup chain (.VBK + .VIBs) it won't work as simply ignoring previous chain would possibly cause configuration conflicts in future.

And then...

As far as I understand case summary you need to use backup copy job to copy backups to a repository with rotated drives, right ?
By default after changing the medias copy job will stop working as it will not be able to find files produced during previous sessions.
To get it to work again you need to clean up configuration records.
In order to do so, please open Veeam GUI -> Backups, then right click on one produced by corresponding copy job and click "Remove from backups" (please see attached screenshot).

Neither of these helps me get the backup copy job working to removable media. The point I am stuck on is that I thought the registry key and hotfix addressed the issue, but it didn't for us. And based on the responses above, it doesn't sound there IS a fix?!

[veremin]

If applicable, you can use the scripts mentioned in my previous message. They should automate the cleaning procedure for you ( backup-> delete) for you, making backup copy job work without issues with removable media.

Thanks.

[Gostev]

Hi all, as promised in the weekly forum digest couple of weeks ago, here is the beta code for full support of all use cases around rotated drives. We are looking to include this code in the next product patch, but would like to get your feedback first, to make sure we have addressed all your requirements. The goal here was to remove the need for ANY scripting on your side completely. I would hate to find out we've missed something after the patch is released, so I appreciate your help with this.

This code adds rotated drives support for all backup modes of primary backup jobs, as well as for Backup Copy jobs. It also adds optional backup repository auto cleanup functionality. This code must be installed on B&R 7.0 R2.

Registry values controlling the functionality:

ForceCreateMissingVBK (DWORD)
Set to 1 to enable support for rotated drives. With this functionality enabled, if any backup file from the latest full backup chain is missing (such as when the existing hard drive is replaced by another one), jobs will start the new backup chain and create the new full backup (instead of failing out).

ForceDeleteBackupFiles (DWORD)
Enables automatic cleanup of the backup repository from ALL existing files if any backup file from the latest full backup chain is missing (such as when the existing hard drive is replaced by the hard drive brought back from offsite location, and containing some older backup files).
Set to 3 to make the job delete the entire contents of the backup job’s folder only.
Set to 5 to make the job delete the entire contents of the root backup repository folder, thus deleting any files belonging to other jobs along. Note that if backup repository is pointed at the volume's root folder, the entire volume's contents will be erased.

Download here:
ftp://ftp00445433:ctDjaMKR@supportftp.veeam.com/

[bunger]

I have been working on the powershell script this morning and it doesn't seem to work.

My backup architecture is as follows:

Backup Job called "Nightly Backup" backs up VMs to internal drive and this backup location is d:\NightlyBackup

Backup Copy Job ( Attached to "Nightly Backup" ) should copy recent backup to rotating external drive. This rotating location is A:\

When running and stepping through the Powershell script, it is checking the location d:\NightlyBackup for the existence of a VBK. If it can't find a VBK, then it stops the scheduler, deletes the backup from the GUI, deletes the files from media, and then restarts the scheduler.

The problems that I see with the script is that it references the main backup job and not the backup copy job.
- It looks to d:\NightlyBackup for the existence of a VBK ( which always exists as it is not rotating )
- it runs $backup.Delete(), which is initialized to the backup job and not the backup copy job
- it deletes the backup files from the d:\NightlyBackup location and not A:\

What am I missing here?

[bunger]

Oh man. Is it Friday yet?

I just realized I was entering the main backup job name in the script instead of the copy job. My bad. <sigh> Ignore my previous post.

[tsightler]

You may also be interested in Anton's post above, from earlier today just before your post, where he's asking for testers of the beta patch that allows all of this to "just work" by setting a few registry keys, no scripting required.

[bunger]

I just installed the patch for clients - we'll see how it plays...

[STORM121]

Do we replace the current dll files with the new ones and do we have to create these new registry entries?

[teknomage]

I won't have a chance to try this in my lab until tomorrow, but I have a question. When the .DLL files are replaced, are those DWORDs created automatically or do they need to be created manually? If they need to be created manually, what hive do they need to be created in?

[foggy]

Provided .DLL files should replace the original ones (recommended to keep the old files somewhere, just to be on the safe side) and the keys should be created manually under Veeam B&R registry hive: HKEY_LOCAL_MACHINE\SOFTWARE\VeeaM\Veeam Backup and Replication.

[GregZTL]

Swapping to the new DLLs appears to break my ability to create new Repos with the message 'Detecting server configuration Error: Method not found: 'Veeam.Backup.Common.EOSPlatform Veeam.Backup.Model.CphysicalHostInfo.get_OsPlatform()' at the 'Apply' step. I'm running v7.0.0.690. Changing back to the original DLLs makes it work again, so I've used this as a work-around while I try the patch out.

[foggy]

I'm running v7.0.0.690.

Greg, as it is stated in Anton's post introducing the fix, you should be running R2 update (v7.0.0.771) to be able to use it:

This code must be installed on B&R 7.0 R2.

[GregZTL]

Ah, thanks foggy, a bit of a fail there on my part! I assumed that since I'd downloaded the most recent installer (.iso) I'd be on the latest version, I've found the R2а update to install and try again.

[foggy]

It would be much appreciated if you could post the results of your testing here as we are highly interested in any feedback. Thanks!

[sheath777]

I'm using a 64bit machine. Do the keys need to be added here:

HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication

Or here:

HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Veeam\Veeam Backup and Replication

I've seen other posts that say to place reg keys in the Wow6432Node on 64bit machines. Thanks!

[teknomage]

I have a question about this registry key:

ForceDeleteBackupFiles (DWORD)
Enables automatic cleanup of the backup repository from ALL existing files if any backup file from the latest full backup chain is missing (such as when the existing hard drive is replaced by the hard drive brought back from offsite location, and containing some older backup files).
Set to 3 to make the job delete the entire contents of the backup job’s folder only.
Set to 5 to make the job delete the entire contents of the root backup repository folder, thus deleting any files belonging to other jobs along. Note that if backup repository is pointed at the volume's root folder, the entire volume's contents will be erased.

I've tested the new patch with the following equipment:

• Physical server
  USB attached RDX drive (removable hard drives) with three drives
  ESXi host with two shell (tiny) VMs
  I have added the DWORD and set its value to 3.

I have created a job two backup my two VMs (I've tried forward and reverse incremental) and ran the job manually once per RDX drive for a couple of cycles. In my testing, I have run the job three times per cartridge (nine times total) and even through I have my retention policy set to 2, I I now have three .vbk files on my cartridges. At no point did I see the backup job clear out the backup folder I have defined.

Can anyone confirm they are seeing the same results or help me find a flaw in my testing procedures?

[Gostev]

HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication

This.

I have created a job two backup my two VMs (I've tried forward and reverse incremental) and ran the job manually once per RDX drive for a couple of cycles. In my testing, I have run the job three times per cartridge (nine times total) and even through I have my retention policy set to 2, I I now have three .vbk files on my cartridges. At no point did I see the backup job clear out the backup folder I have defined.

Well, the whole idea behind the rotated media is that you have the job create backup files on the external drive, then you take the drives offsite. Of course, you still want the backup files on the drives you take offsite, so I am not sure why you expect them to be deleted from your cartridges at this point.

I don't know how RDX works, but assuming "cartridges" are separate folders/volumes/locations on the device, the deletion will happen if you do something like this:
1. Run the job against Cartridge 1, this will create some backup files there. Take the cartridge offsite once full, or as needed.
2. Run the job against Cartridge 2, this will create some backup files there (including the new full backup), take the cartridge offsite.
3. Run the job against Cartridge 3, this will create some backup files there (including the new full backup), take the cartridge offsite.
4. No more cartridges left in the device, so you bring Cartridge 1 (with the oldest backups) from the offsite location back to your office, and plug it back in.
5. Run the job against Cartridge 1 again. The job will not see any backup files from the previous step 3 on this cartridge, understand that this is some old cartridge that you now want to be reused, and will clean it up from any existing files automatically.

Normally, people just use external USB or eSATA hard drives for this purpose. This is what we ourselves were doing at Veeam in the early days.

Does it make sense?

[teknomage]

That's essentially what I'm trying to accomplish except for filling the drive/cartridge each time. My intentions are to duplicate what we're doing currently with tape drives, which is to do a full backup every night and to take that backup drive offsite daily (regardless if its full or not).

My test is obviously not representative of an actual environment because of the tiny backup jobs. What you described will get me more than I was intending RPO-wise since most of the customers will have total backups smaller than the 1TB media I will typically be using. I will do more testing with more realistic backups jobs.

When does the clearing of the backup folder take place? I assume the job will realize that the folder/media is full and clear it before attempting to do the actual backup.

FYI: This is the type of drive we're currently standardized on using http://www.tandbergdata.com/us/index.cf ... -quikstor/

[Gostev]

I assume the job will realize that the folder/media is full and clear it before attempting to do the actual backup.

The erase is unconditional. As long as the job cannot find the previous restore point in the backup location, it will delete all existing files from the location defined by value of the corresponding registry key.

[teknomage]

Does it only erase if the drive is full?

Will the jobs be able to control the number of restore points through the setting in the job or is that not valid because the backup chain is not consistent?

[Gostev]

The erase is unconditional, meaning no disk space checks are performed.

You are implicitly controlling the number of restore points you keep by inserting the hard drive with the oldest restore points and letting the job delete them. But there are no hard drives tracking or anything like that in the current code. We will erase and use whatever drive you give us.

[teknomage]

That is not the behavior I saw in my testing. When I cycled through the drives the second and third time it kept adding .vbk files. After three times through i had three .vbk files.

Here are the registry settings I have configured. Are these correct?

[HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication]
"ForceCreateMissingVBK"=dword:00000001
"ForceDeleteBackupFiles"=dword:00000003

[Gostev]

Yes, your settings look good. Please open a support case and provide the debug logs (explain that this is beta, so that the logs are forwarded straight to R&D). Thanks!