Veeam 7 Copy Job - Transform

[tsightler]

I have tested that one with a Test Backup. Worked ok, but i had some issues with a Copy Job that was continuous. So i changed a bit of that code. That prevents the creation of "my CopyJob_1" directories.
...
I have added the stopping o the Copy Job and waiting till it's stopped. Then deleting everything. That worked fine for me.
I will give it a try on a small BackupJob.

Interesting, I can't reproduce this running it as a post-job activity as the job should be in the in-between state, but I can see how it might happen. I had even considered adding the Disable/Wait/Enable cycle to the script, but in my lab it works every time without that cycle, although I have another post-job activity for a copy job that changes the interval (for a Copy Job to do only end-of-month backups), where I use this exact technique so I can see why it might be required. Thanks for testing and the update/fixes and please keep me updated. I have some ideas for a few other minor enhancements.

[buwprof]

Interesting, I can't reproduce this running it as a post-job activity as the job should be in the in-between state, but I can see how it might happen. I had even considered adding the Disable/Wait/Enable cycle to the script, but in my lab it works every time without that cycle, although I have another post-job activity for a copy job that changes the interval (for a Copy Job to do only end-of-month backups), where I use this exact technique so I can see why it might be required. Thanks for testing and the update/fixes and please keep me updated. I have some ideas for a few other minor enhancements.

Maybe our System is slower or the job was "running"? In my testing I had some issues with that.
I have tested it while the Copyjob was "IDLE", aka running (continuous).
Then i had to stop the CopyJob, delete the "backups" from Backup, delete the _1 directory and restart the Job.
So i included this into the Job to prevent this, also we have more than one BackupJob so to get that script a bit generic i have modified it again not very cleanly but quick and working.

Code: Select all

# Get Parameters
param( $job )

# Import Veeam Snapin and we do not want Errors
asnp -Name VeeamPSSnapin -ErrorAction SilentlyContinue

# If we have a JobName then we go further
if ( $job ) {

    # JobName here
    $jobName          = $job

    # Get Backup Specs
    $backup           = Get-VBRBackup -Name $jobName

    if ( $backup.JobName ) {
        # Get FilePath
        $backupfilepath   = ( $backup.GetStorages() | ?{$_.GetStorageType() -eq "vbk"} ).FilePath

        # Get Directory for deleting
        $backupdirpath    = ( $backup.GetStorages() | ?{$_.GetStorageType() -eq "vbk"} ).DirPath

        # If current active VBK is found on media then media has not been rotated
        if ( -not( Test-Path "$backupfilepath" ) ) {

            # Current active VBK was not found indicating media has been changed
            # First DisableScheduler, If not then the Copy Job will Create a new Directory like $Jobname_1
    
            # Get-Job Object
            $CopyJob      = Get-VBRJob -Name $jobName
    
            # Disbale Scheduler
            $CopyJob.DisableScheduler()
    
            # then Wait until Job is Stopped (and is not running anymore)
            while ( -not( $CopyJob.IsStopped() ) ) { 
            # Do Nothing 
            }
    
            # If stopped then Delete from VeeamGUI
            $backup.Delete()
    
            # Remove Backup Files from Media
            if ( Test-Path "$backupdirpath" ) {

                # Remove whole Directory
                Remove-Item -Path "$backupdirpath" -Recurse
            }

            # Enable Scheduler
            $CopyJob.EnableScheduler()
        }
    } # UPPS NOT CORRECT JOBNAME - BUT WE ARE SILENT
} # NO JOBNAME given - WE ARE SILENT

Start it with: powershell.exe c:\Path\TO\CopyJobCleanup.ps1 -Job 'CopyJobTest'

[tsightler]

I have tested it while the Copyjob was "IDLE", aka running (continuous).

Ah, so you are saying you didn't run it as a post-job activity? If so, that would likely be the difference. Even so, probably doesn't hurt to have the disable/enable cycle in there. That's for the updates, I had thought about using a parameter for the job name but expected that most people with rotated media would only have one job.

[hongwangusa]

Is this a powershell script?

I didn't see any option to run a PS script in the advanced section.

[Vitaliy S.]

There is a section to run a script, you can specify any batch file to trigger the PS script in question. Thanks!

[tsightler]

I didn't see any option to run a PS script in the advanced section.

On the Target options in the wizard there's an Advanced button, and on the Advanced tab under the Post-job actiivty option you can select the "Run the following command" option and then use the following syntax to run a Powershell script:

Code: Select all

powershell.exe "C:\full\path\to\script.ps1"

[hongwangusa]

I run the script itself in command line. I got an error. saying no snap-in has been registered for windows PowerShell version 2.

[tsightler]

So it seems the Veeam Powershell snapin is either not installed on your system, or some other problem is preventing it from working. Are you able to start a Veeam Powershell prompt from the Veeam GUI (it's should be listed in the menu tab near the top)? If not, it might not be installed (it's an optional component). Otherwise you might want to contact Veeam support for assistance.

[hongwangusa]

I figured how to go pass it. I was running under powershell x86. once I changed to regualar ps. It tried to run. but it tells me that it can not delete the folder while the job is running.

I think it is something to do with the expiration interval. I have it to run everyday day. so what is the expiration interval. When exactly will the job run if I set it under post job script?

How can I test it from a command line?

[tsightler]

I think it is something to do with the expiration interval. I have it to run everyday day. so what is the expiration interval. When exactly will the job run if I set it under post job script?

It will run when the interval expires, so for example if you configure the copy job to run 1 time per day, and to start at 6PM, then it will run at 6PM, between the expiration of the old interval and the start of the new one.

IHow can I test it from a command line?

The script as I wrote it was designed to be run only as a post-job script because at that point the job is not running since it's "post-job". You can run it from the command line by manually disabling the job, then running the script, then reenabling the job.

There's also a modified version that does the disable/enable cycle automatically which is the most recent versions posted above by another user.

[zak2011]

Is it right to say that a Full Backup transformation during a Backup copy job is similar to a Synthetic Full for a normal backup job?

Thanks

[veremin]

Yep, the concept is the same - .vbk file is merged with the .vib one. The only difference is that backup copy transformation affects the beginning of the backup chain, instead of its end. Thanks.

[zak2011]

Thats interesting. However I remmember , configuring Synthetic Full to a dedup appliance was never recommended as the whole data had to be rehydrated on the dedup appliance and it would take forever.
Does this apply in the case of backup copy jobs going to a dedup appliance?

Thanks

[foggy]

Yes, the same considerations apply to the backup copy jobs. However, since the synthetic activity takes place on the secondary storage and does not affect production in any way, the time required to perform this operation is not that critical. A bit more on this here.

[zak2011]

Thankyou foggy!

[Regnor]

At the moment I'm also trying to configure a copy backup job for rotating media and the script from tsightler looks promissing as the VIB files are causing some errors. But I do also have another problem and perhaps someone here has an idea; I can't delete or remove the backups from the B&R console as it says they are used in an application group. I do in fact have an application group, but why does it point to the backup files which were created during the copy backup job?

[foggy]

Are you sure you're trying to delete the backups created by the backup copy job? Actually those cannot be even added to the application group (without creating a dummy backup job for them).

[Regnor]

Yes I am sure; Under Backups->Disks I tried to delete those backups which are listed under the copy backup job.

[foggy]

Confirmed with QC, this is an already known issue that will be addressed in the next Veeam B&R patch.

[Regnor]

Thanks for this information. I've deleted the application group and now it's working; not a big deal as I only need it for Virtual Labs.

[Regnor]

FYI; Backup Copy to USB works great with this script.
I have just created it as an scheduled task instead of a post job activitiy. With post job activity I had some problems like the script not being executed or just not working.

[yizhar]

Hi.

I must say that I'm very disappointed with all this.

Using rotated media should be a simple and safe feature with Veeam.
Not something that relies on 3rd party tools, or custom scripts which are prone to errors.
I was thinking and hoping that V7 backup copy jobs will be the solution, but it seems otherwise.

One of the problems with the suggested script that deletes backups -
What happens if the administrator connects a USB disk (or ask someone on site to do it), in the purpose for restoring,
but during preparations that backups get deleted by the script?


I suggest the following as feature request for next Veeam release:

Allow us to setup a special repository that point to the rotated media drive letter / share.
This will work similar to a media pool, and Veeam will know that it is a rotating media target and therefor will handle it accordingly.

Allow us to choose how many copies to store there, just latest VBK or also some kind of incrementals.

An option to select if we want to use existing data (a VBK from prior backup) as seed for the a new synthetic one, or simply to overwrite it.

Keep records of which backups are stored on each drive.

Maybe popup a question to the user when detecting disk rotation - asking if he wants to protect that backups on disk (for restore) or simply leave it for next backup which should be the default answer after reasonable timeout.

Do not overwrite or delete existing backups until last minute when the newest copy to USB begins.

Am I asking for too much?

Thanks,
Yizhar

[buwprof]

Hi Yizhar,

no you are not asking for too much. I have the same opinion. Veeam please add this to next Version!!
Before we started to use Veeam i had a long time for testing other Backup Software. And probably you know there is NO Software which has ALL the features you need. If one had that feature it lacks the other one.
Veeam was a compromise of all of them. It has a powershell API, which most of the other don't have.

Back to your Restore Scenario: When I am in need to restore something I know where the Script deletes my Backups (or this is documented in an HOWTO for other ADMINS) and i can deactivate it. If we are in need that someone (NOT IT) has to do the restore, there will be a good chance that this won't be a success, even when Veeam will handle the Rotated Media.

[Gostev]

I will discuss this feature with R&D, thanks for your feedback.

[Vasko]

Hi Guys,

cool Script!
Unfortunately the script does not work correctly when you run a backup job multiple times a day and switch the media in between those runs (for testing purposes obviously). Most likely because Veeam creates files with the current day of the month in the filename (like CopyTest2013-11-14T000000.vbk). Then you have this file on all of your USB drives and the script always detects it, thus not deleting the files or the GUI entrys.

Apart from that, I've added some echos for those like me who are testing the script against different scenarios.

Code: Select all

# Get Parameters
param( $job )

# Import Veeam Snapin and we do not want Errors
asnp -Name VeeamPSSnapin

# If we have a JobName then we go further
if ( $job ) {

    # JobName here
    $jobName          = $job

    # Get Backup Specs
    $backup           = Get-VBRBackup -Name $jobName

    if ( $backup.JobName ) {

        Write-Host "Cleanup Task for Job:"$job

        # Get FilePath
        $backupfilepath   = ( $backup.GetStorages() | ?{$_.GetStorageType() -eq "vbk"} ).FilePath
        # Get Directory for deleting
        $backupdirpath    = ( $backup.GetStorages() | ?{$_.GetStorageType() -eq "vbk"} ).DirPath

        # If current active VBK is found on media then media has not been rotated
        if ( -not( Test-Path "$backupfilepath" ) ) {

            Write-Host "Current active VBK "$backupfilepath" was not found indicating media has been changed!"
            Write-Host "Disabling scheduler..."
    
            # Get-Job Object
            $CopyJob      = Get-VBRJob -Name $jobName
    
            # Disbale Scheduler
            $CopyJob.DisableScheduler()
    
            Write-Host "Waiting until Job is stopped..."
            while ( -not( $CopyJob.IsStopped() ) ) { 
            #Do Nothing
            }
            Write-Host "Done!"
    
            Write-Host "Deleting Backups from the Veeam GUI ..."
            $backup.Delete()
            Write-Host "Done!"

            if ( Test-Path "$backupdirpath" ) {

                Write-Host "Removing Backup Files from Media ..."
                Remove-Item -Path "$backupdirpath" -Recurse
                Write-Host " Done!"
            }

            Write-Host "Enabling scheduler ..."
            $CopyJob.EnableScheduler()
            Write-Host "Done!"
        }
	else {
	    Write-Host "Media has not been changed!"
	}
    }
    else {
        Write-Host "Jobname not correct!"
    }
}
else {
    Write-Host "No Jobname given - specify with "-job "JOBNAME"""
}

[tscott]

So I added REG_DWORD of "ForceCreateMissingVBK" with a value of 1 under HKEY_LOCAL_MACHINE\SOFTWARE\VeeaM\Veeam Backup and Replication and also under HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Veeam\Veeam Backup and Replication then rebooted the server..

I have a VMware Backup Copy job that worked great for the 1st USB drive.. It failed on the second.. I added the above keys, deleted all files from the 2nd USB drive and chose to Sync Now and it still fails..

What am I doing wrong?

Thanks

[veremin]

H, Tom, have you already tried to put the proposed script into use and see whether it makes any difference? Thanks.

[buwprof]

Tom just look in the log files. There must be a reason why it fails. What Error?

We have here a small Environment so everything is on one box. If you have splitted it, maybe you have to add it on the "right" box.

I have added both keys too.

But when we change USB Media. We do not delete anything. The script is taking care of that.
-It deletes everything on path for the curren CopyJob.
-It deletes everything out of Veeam for the CopyJob.
-So you end up with a clean USB Media and a Clean Copyjob Restorepoint.

It has to be running after the Backupjob is done.

[tscott]

Tom just look in the log files. There must be a reason why it fails. What Error?

We have here a small Environment so everything is on one box. If you have splitted it, maybe you have to add it on the "right" box.

I have added both keys too.

But when we change USB Media. We do not delete anything. The script is taking care of that.
-It deletes everything on path for the curren CopyJob.
-It deletes everything out of Veeam for the CopyJob.
-So you end up with a clean USB Media and a Clean Copyjob Restorepoint.

It has to be running after the Backupjob is done.

I don't have any script added.. I see numerous ones in this thread and not sure which to use.. Here is the job error:

11/21/2013 9:37:24 AM :: Error: Client error: File does not exist. File: [E:\BACKUP to USB Copy\BACKUP to USB Copy2013-11-19T000000.vib].
Failed to restore file from local backup. VFS link: [summary.xml]. Target file: [MemFs://Tar2Text]. CHMOD mask: [0].

[buwprof]

Hm, this error indicates that a VIB (Incremental) isn't there.
I don' know if the Reg-Hack is for creating VIB's? I think it creates only VBK Files (First Full of CopyJob).

http://www.veeam.com/kb1154

Do you try to restore something? If so you took a restorepoint which needs a VBK + VIB File.

I am just doing this:

- Detach USB Media 01 after Copy.
- Attach USB Media 02.
- Running Backup Job, @ the End of the Job the Script checks
-- if VBK File is the same on disk and CopyJob if so do nothing (Every CopyJob has it's own VBK File)
--- else delete everything on USB Media for CopyJob.
--- Delete Restorepoints for Copyjob.
- The Copy Job will run
- Restart

If you have to Restore something from Media 01. You have to Import Media 01 VBK File etc. It will then be available under imported. Then restore what you like.

If you do not use a Script you can do it Manually.

I don't know if there is a possibility to left the restorepoints in database while you have another Media inserted in Veeam.

Try that one which will fit your best needs. there is no THIS or THAT script. I use mine and have no issues. But it depends on the environment /Strategy / Needs etc. . If not applicable then modify that script.