Comprehensive data protection for all workloads
tsightler
VP, Product Management
Posts: 6009
Liked: 2843 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Veeam 7 Copy Job - Transform

Post by tsightler »

buwprof wrote:I have tested that one with a Test Backup. Worked ok, but i had some issues with a Copy Job that was continuous. So i changed a bit of that code. That prevents the creation of "my CopyJob_1" directories.
...
I have added the stopping o the Copy Job and waiting till it's stopped. Then deleting everything. That worked fine for me.
I will give it a try on a small BackupJob.
Interesting, I can't reproduce this running it as a post-job activity as the job should be in the in-between state, but I can see how it might happen. I had even considered adding the Disable/Wait/Enable cycle to the script, but in my lab it works every time without that cycle, although I have another post-job activity for a copy job that changes the interval (for a Copy Job to do only end-of-month backups), where I use this exact technique so I can see why it might be required. Thanks for testing and the update/fixes and please keep me updated. I have some ideas for a few other minor enhancements.
buwprof
Influencer
Posts: 11
Liked: 2 times
Joined: Sep 04, 2013 2:09 pm
Full Name: Thomas Voß

Re: Veeam 7 Copy Job - Transform

Post by buwprof »

tsightler wrote:Interesting, I can't reproduce this running it as a post-job activity as the job should be in the in-between state, but I can see how it might happen. I had even considered adding the Disable/Wait/Enable cycle to the script, but in my lab it works every time without that cycle, although I have another post-job activity for a copy job that changes the interval (for a Copy Job to do only end-of-month backups), where I use this exact technique so I can see why it might be required. Thanks for testing and the update/fixes and please keep me updated. I have some ideas for a few other minor enhancements.
Maybe our System is slower or the job was "running"? In my testing I had some issues with that.
I have tested it while the Copyjob was "IDLE", aka running (continuous).
Then i had to stop the CopyJob, delete the "backups" from Backup, delete the _1 directory and restart the Job.
So i included this into the Job to prevent this, also we have more than one BackupJob so to get that script a bit generic i have modified it again not very cleanly but quick and working.

Code: Select all

# Get Parameters
param( $job )

# Import Veeam Snapin and we do not want Errors
asnp -Name VeeamPSSnapin -ErrorAction SilentlyContinue

# If we have a JobName then we go further
if ( $job ) {

    # JobName here
    $jobName          = $job

    # Get Backup Specs
    $backup           = Get-VBRBackup -Name $jobName

    if ( $backup.JobName ) {
        # Get FilePath
        $backupfilepath   = ( $backup.GetStorages() | ?{$_.GetStorageType() -eq "vbk"} ).FilePath

        # Get Directory for deleting
        $backupdirpath    = ( $backup.GetStorages() | ?{$_.GetStorageType() -eq "vbk"} ).DirPath

        # If current active VBK is found on media then media has not been rotated
        if ( -not( Test-Path "$backupfilepath" ) ) {

            # Current active VBK was not found indicating media has been changed
            # First DisableScheduler, If not then the Copy Job will Create a new Directory like $Jobname_1
    
            # Get-Job Object
            $CopyJob      = Get-VBRJob -Name $jobName
    
            # Disbale Scheduler
            $CopyJob.DisableScheduler()
    
            # then Wait until Job is Stopped (and is not running anymore)
            while ( -not( $CopyJob.IsStopped() ) ) { 
            # Do Nothing 
            }
    
            # If stopped then Delete from VeeamGUI
            $backup.Delete()
    
            # Remove Backup Files from Media
            if ( Test-Path "$backupdirpath" ) {

                # Remove whole Directory
                Remove-Item -Path "$backupdirpath" -Recurse
            }

            # Enable Scheduler
            $CopyJob.EnableScheduler()
        }
    } # UPPS NOT CORRECT JOBNAME - BUT WE ARE SILENT
} # NO JOBNAME given - WE ARE SILENT
Start it with: powershell.exe c:\Path\TO\CopyJobCleanup.ps1 -Job 'CopyJobTest'
tsightler
VP, Product Management
Posts: 6009
Liked: 2843 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Veeam 7 Copy Job - Transform

Post by tsightler »

buwprof wrote:I have tested it while the Copyjob was "IDLE", aka running (continuous).
Ah, so you are saying you didn't run it as a post-job activity? If so, that would likely be the difference. Even so, probably doesn't hurt to have the disable/enable cycle in there. That's for the updates, I had thought about using a parameter for the job name but expected that most people with rotated media would only have one job.
hongwangusa
Novice
Posts: 3
Liked: never
Joined: Jun 14, 2013 1:35 pm
Full Name: Hong Wang
Contact:

Re: Veeam 7 Copy Job - Transform

Post by hongwangusa »

Is this a powershell script?

I didn't see any option to run a PS script in the advanced section.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Veeam 7 Copy Job - Transform

Post by Vitaliy S. »

There is a section to run a script, you can specify any batch file to trigger the PS script in question. Thanks!
tsightler
VP, Product Management
Posts: 6009
Liked: 2843 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Veeam 7 Copy Job - Transform

Post by tsightler »

hongwangusa wrote:I didn't see any option to run a PS script in the advanced section.
On the Target options in the wizard there's an Advanced button, and on the Advanced tab under the Post-job actiivty option you can select the "Run the following command" option and then use the following syntax to run a Powershell script:

Code: Select all

powershell.exe "C:\full\path\to\script.ps1"
hongwangusa
Novice
Posts: 3
Liked: never
Joined: Jun 14, 2013 1:35 pm
Full Name: Hong Wang
Contact:

Re: Veeam 7 Copy Job - Transform

Post by hongwangusa »

I run the script itself in command line. I got an error. saying no snap-in has been registered for windows PowerShell version 2.
tsightler
VP, Product Management
Posts: 6009
Liked: 2843 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Veeam 7 Copy Job - Transform

Post by tsightler »

So it seems the Veeam Powershell snapin is either not installed on your system, or some other problem is preventing it from working. Are you able to start a Veeam Powershell prompt from the Veeam GUI (it's should be listed in the menu tab near the top)? If not, it might not be installed (it's an optional component). Otherwise you might want to contact Veeam support for assistance.
hongwangusa
Novice
Posts: 3
Liked: never
Joined: Jun 14, 2013 1:35 pm
Full Name: Hong Wang
Contact:

Re: Veeam 7 Copy Job - Transform

Post by hongwangusa »

I figured how to go pass it. I was running under powershell x86. once I changed to regualar ps. It tried to run. but it tells me that it can not delete the folder while the job is running.

I think it is something to do with the expiration interval. I have it to run everyday day. so what is the expiration interval. When exactly will the job run if I set it under post job script?

How can I test it from a command line?
tsightler
VP, Product Management
Posts: 6009
Liked: 2843 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Veeam 7 Copy Job - Transform

Post by tsightler »

hongwangusa wrote:I think it is something to do with the expiration interval. I have it to run everyday day. so what is the expiration interval. When exactly will the job run if I set it under post job script?
It will run when the interval expires, so for example if you configure the copy job to run 1 time per day, and to start at 6PM, then it will run at 6PM, between the expiration of the old interval and the start of the new one.
hongwangusa wrote:IHow can I test it from a command line?
The script as I wrote it was designed to be run only as a post-job script because at that point the job is not running since it's "post-job". You can run it from the command line by manually disabling the job, then running the script, then reenabling the job.

There's also a modified version that does the disable/enable cycle automatically which is the most recent versions posted above by another user.
zak2011
Veteran
Posts: 367
Liked: 41 times
Joined: May 15, 2012 2:21 pm
Full Name: Arun
Contact:

Re: Veeam 7 Copy Job - Transform

Post by zak2011 »

Is it right to say that a Full Backup transformation during a Backup copy job is similar to a Synthetic Full for a normal backup job?

Thanks
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Veeam 7 Copy Job - Transform

Post by veremin » 1 person likes this post

Yep, the concept is the same - .vbk file is merged with the .vib one. The only difference is that backup copy transformation affects the beginning of the backup chain, instead of its end. Thanks.
zak2011
Veteran
Posts: 367
Liked: 41 times
Joined: May 15, 2012 2:21 pm
Full Name: Arun
Contact:

Re: Veeam 7 Copy Job - Transform

Post by zak2011 »

Thats interesting. However I remmember , configuring Synthetic Full to a dedup appliance was never recommended as the whole data had to be rehydrated on the dedup appliance and it would take forever.
Does this apply in the case of backup copy jobs going to a dedup appliance?

Thanks
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Veeam 7 Copy Job - Transform

Post by foggy »

Yes, the same considerations apply to the backup copy jobs. However, since the synthetic activity takes place on the secondary storage and does not affect production in any way, the time required to perform this operation is not that critical. A bit more on this here.
zak2011
Veteran
Posts: 367
Liked: 41 times
Joined: May 15, 2012 2:21 pm
Full Name: Arun
Contact:

Re: Veeam 7 Copy Job - Transform

Post by zak2011 »

Thankyou foggy!
Regnor
Veeam Software
Posts: 934
Liked: 287 times
Joined: Jan 31, 2011 11:17 am
Full Name: Max
Contact:

Re: Veeam 7 Copy Job - Transform

Post by Regnor »

At the moment I'm also trying to configure a copy backup job for rotating media and the script from tsightler looks promissing as the VIB files are causing some errors. But I do also have another problem and perhaps someone here has an idea; I can't delete or remove the backups from the B&R console as it says they are used in an application group. I do in fact have an application group, but why does it point to the backup files which were created during the copy backup job?
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Veeam 7 Copy Job - Transform

Post by foggy »

Are you sure you're trying to delete the backups created by the backup copy job? Actually those cannot be even added to the application group (without creating a dummy backup job for them).
Regnor
Veeam Software
Posts: 934
Liked: 287 times
Joined: Jan 31, 2011 11:17 am
Full Name: Max
Contact:

Re: Veeam 7 Copy Job - Transform

Post by Regnor »

Yes I am sure; Under Backups->Disks I tried to delete those backups which are listed under the copy backup job.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Veeam 7 Copy Job - Transform

Post by foggy »

Confirmed with QC, this is an already known issue that will be addressed in the next Veeam B&R patch.
Regnor
Veeam Software
Posts: 934
Liked: 287 times
Joined: Jan 31, 2011 11:17 am
Full Name: Max
Contact:

Re: Veeam 7 Copy Job - Transform

Post by Regnor »

Thanks for this information. I've deleted the application group and now it's working; not a big deal as I only need it for Virtual Labs.
Regnor
Veeam Software
Posts: 934
Liked: 287 times
Joined: Jan 31, 2011 11:17 am
Full Name: Max
Contact:

Re: Veeam 7 Copy Job - Transform

Post by Regnor » 1 person likes this post

FYI; Backup Copy to USB works great with this script.
I have just created it as an scheduled task instead of a post job activitiy. With post job activity I had some problems like the script not being executed or just not working.
yizhar
Service Provider
Posts: 181
Liked: 48 times
Joined: Sep 03, 2012 5:28 am
Full Name: Yizhar Hurwitz
Contact:

Re: Veeam 7 Copy Job - Transform

Post by yizhar »

Hi.

I must say that I'm very disappointed with all this.

Using rotated media should be a simple and safe feature with Veeam.
Not something that relies on 3rd party tools, or custom scripts which are prone to errors.
I was thinking and hoping that V7 backup copy jobs will be the solution, but it seems otherwise.

One of the problems with the suggested script that deletes backups -
What happens if the administrator connects a USB disk (or ask someone on site to do it), in the purpose for restoring,
but during preparations that backups get deleted by the script?


I suggest the following as feature request for next Veeam release:

Allow us to setup a special repository that point to the rotated media drive letter / share.
This will work similar to a media pool, and Veeam will know that it is a rotating media target and therefor will handle it accordingly.

Allow us to choose how many copies to store there, just latest VBK or also some kind of incrementals.

An option to select if we want to use existing data (a VBK from prior backup) as seed for the a new synthetic one, or simply to overwrite it.

Keep records of which backups are stored on each drive.

Maybe popup a question to the user when detecting disk rotation - asking if he wants to protect that backups on disk (for restore) or simply leave it for next backup which should be the default answer after reasonable timeout.

Do not overwrite or delete existing backups until last minute when the newest copy to USB begins.

Am I asking for too much?

Thanks,
Yizhar
buwprof
Influencer
Posts: 11
Liked: 2 times
Joined: Sep 04, 2013 2:09 pm
Full Name: Thomas Voß

Re: Veeam 7 Copy Job - Transform

Post by buwprof »

Hi Yizhar,

no you are not asking for too much. I have the same opinion. Veeam please add this to next Version!!
Before we started to use Veeam i had a long time for testing other Backup Software. And probably you know there is NO Software which has ALL the features you need. If one had that feature it lacks the other one.
Veeam was a compromise of all of them. It has a powershell API, which most of the other don't have.

Back to your Restore Scenario: When I am in need to restore something I know where the Script deletes my Backups (or this is documented in an HOWTO for other ADMINS) and i can deactivate it. If we are in need that someone (NOT IT) has to do the restore, there will be a good chance that this won't be a success, even when Veeam will handle the Rotated Media.
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Veeam 7 Copy Job - Transform

Post by Gostev » 1 person likes this post

I will discuss this feature with R&D, thanks for your feedback.
Vasko
Influencer
Posts: 11
Liked: 1 time
Joined: Oct 31, 2013 12:11 pm
Full Name: Felix Geschwindner
Contact:

Re: Veeam 7 Copy Job - Transform

Post by Vasko »

Hi Guys,

cool Script!
Unfortunately the script does not work correctly when you run a backup job multiple times a day and switch the media in between those runs (for testing purposes obviously). Most likely because Veeam creates files with the current day of the month in the filename (like CopyTest2013-11-14T000000.vbk). Then you have this file on all of your USB drives and the script always detects it, thus not deleting the files or the GUI entrys.

Apart from that, I've added some echos for those like me who are testing the script against different scenarios.

Code: Select all

# Get Parameters
param( $job )

# Import Veeam Snapin and we do not want Errors
asnp -Name VeeamPSSnapin

# If we have a JobName then we go further
if ( $job ) {

    # JobName here
    $jobName          = $job

    # Get Backup Specs
    $backup           = Get-VBRBackup -Name $jobName

    if ( $backup.JobName ) {

        Write-Host "Cleanup Task for Job:"$job

        # Get FilePath
        $backupfilepath   = ( $backup.GetStorages() | ?{$_.GetStorageType() -eq "vbk"} ).FilePath
        # Get Directory for deleting
        $backupdirpath    = ( $backup.GetStorages() | ?{$_.GetStorageType() -eq "vbk"} ).DirPath

        # If current active VBK is found on media then media has not been rotated
        if ( -not( Test-Path "$backupfilepath" ) ) {

            Write-Host "Current active VBK "$backupfilepath" was not found indicating media has been changed!"
            Write-Host "Disabling scheduler..."
    
            # Get-Job Object
            $CopyJob      = Get-VBRJob -Name $jobName
    
            # Disbale Scheduler
            $CopyJob.DisableScheduler()
    
            Write-Host "Waiting until Job is stopped..."
            while ( -not( $CopyJob.IsStopped() ) ) { 
            #Do Nothing
            }
            Write-Host "Done!"
    
            Write-Host "Deleting Backups from the Veeam GUI ..."
            $backup.Delete()
            Write-Host "Done!"

            if ( Test-Path "$backupdirpath" ) {

                Write-Host "Removing Backup Files from Media ..."
                Remove-Item -Path "$backupdirpath" -Recurse
                Write-Host " Done!"
            }

            Write-Host "Enabling scheduler ..."
            $CopyJob.EnableScheduler()
            Write-Host "Done!"
        }
	else {
	    Write-Host "Media has not been changed!"
	}
    }
    else {
        Write-Host "Jobname not correct!"
    }
}
else {
    Write-Host "No Jobname given - specify with "-job "JOBNAME"""
}
tscott
Enthusiast
Posts: 26
Liked: never
Joined: Feb 07, 2013 8:49 pm
Full Name: Tom Scott
Contact:

Re: Veeam 7 Copy Job - Transform

Post by tscott »

So I added REG_DWORD of "ForceCreateMissingVBK" with a value of 1 under HKEY_LOCAL_MACHINE\SOFTWARE\VeeaM\Veeam Backup and Replication and also under HKEY_LOCAL_MACHINE\SOFTWARE\Wow6432Node\Veeam\Veeam Backup and Replication then rebooted the server..

I have a VMware Backup Copy job that worked great for the 1st USB drive.. It failed on the second.. I added the above keys, deleted all files from the 2nd USB drive and chose to Sync Now and it still fails..

What am I doing wrong?

Thanks
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Veeam 7 Copy Job - Transform

Post by veremin »

H, Tom, have you already tried to put the proposed script into use and see whether it makes any difference? Thanks.
buwprof
Influencer
Posts: 11
Liked: 2 times
Joined: Sep 04, 2013 2:09 pm
Full Name: Thomas Voß

Re: Veeam 7 Copy Job - Transform

Post by buwprof »

Tom just look in the log files. There must be a reason why it fails. What Error?

We have here a small Environment so everything is on one box. If you have splitted it, maybe you have to add it on the "right" box.

I have added both keys too.

But when we change USB Media. We do not delete anything. The script is taking care of that.
-It deletes everything on path for the curren CopyJob.
-It deletes everything out of Veeam for the CopyJob.
-So you end up with a clean USB Media and a Clean Copyjob Restorepoint.

It has to be running after the Backupjob is done.
tscott
Enthusiast
Posts: 26
Liked: never
Joined: Feb 07, 2013 8:49 pm
Full Name: Tom Scott
Contact:

Re: Veeam 7 Copy Job - Transform

Post by tscott »

buwprof wrote:Tom just look in the log files. There must be a reason why it fails. What Error?

We have here a small Environment so everything is on one box. If you have splitted it, maybe you have to add it on the "right" box.

I have added both keys too.

But when we change USB Media. We do not delete anything. The script is taking care of that.
-It deletes everything on path for the curren CopyJob.
-It deletes everything out of Veeam for the CopyJob.
-So you end up with a clean USB Media and a Clean Copyjob Restorepoint.

It has to be running after the Backupjob is done.
I don't have any script added.. I see numerous ones in this thread and not sure which to use.. Here is the job error:

11/21/2013 9:37:24 AM :: Error: Client error: File does not exist. File: [E:\BACKUP to USB Copy\BACKUP to USB Copy2013-11-19T000000.vib].
Failed to restore file from local backup. VFS link: [summary.xml]. Target file: [MemFs://Tar2Text]. CHMOD mask: [0].
buwprof
Influencer
Posts: 11
Liked: 2 times
Joined: Sep 04, 2013 2:09 pm
Full Name: Thomas Voß

Re: Veeam 7 Copy Job - Transform

Post by buwprof »

Hm, this error indicates that a VIB (Incremental) isn't there.
I don' know if the Reg-Hack is for creating VIB's? I think it creates only VBK Files (First Full of CopyJob).

http://www.veeam.com/kb1154

Do you try to restore something? If so you took a restorepoint which needs a VBK + VIB File.

I am just doing this:

- Detach USB Media 01 after Copy.
- Attach USB Media 02.
- Running Backup Job, @ the End of the Job the Script checks
-- if VBK File is the same on disk and CopyJob if so do nothing (Every CopyJob has it's own VBK File)
--- else delete everything on USB Media for CopyJob.
--- Delete Restorepoints for Copyjob.
- The Copy Job will run
- Restart

If you have to Restore something from Media 01. You have to Import Media 01 VBK File etc. It will then be available under imported. Then restore what you like.

If you do not use a Script you can do it Manually.

I don't know if there is a possibility to left the restorepoints in database while you have another Media inserted in Veeam.

Try that one which will fit your best needs. there is no THIS or THAT script. I use mine and have no issues. But it depends on the environment /Strategy / Needs etc. . If not applicable then modify that script.
Post Reply

Who is online

Users browsing this forum: No registered users and 244 guests