Backup Copy Job - Why so complicated?

[davek0974]

Hi

We have just got a new backup provider running after the loss of our XP box running Veeam 6, Veeam 8 appears to be finally running smoothly now after some issues in setting it all up, mostly due to Windows security etc.

I have 6 VM's to care for, we have 5 portable USB drives to give us off-site storage, these are rotated daily Monday to Friday (closed at weekends).

I have set Veeam to process jobs serially so only one job is running at any time, we close at 4pm, the backups start at 6pm and seem to be easily finishing by 11pm each day. I scheduled the backups to start at 5min intervals and let veeam sort out when they can actually run.

I then set backup copy jobs, one for each VM, once per day, the data transfer window is set at midnight to 10am. This was supposed to allow me to unplug the USB disk after 10am each day and plug in a new one. The portable repository is set to be backed by removable media.

It sort of works but I get a lot of warnings about copy interval expired which make no sense.

I don't like warnings, I like green flags on my backups, I have no idea if the data is correct or I have missed something??

I have also found that it runs even when VB&R is closed down - the copy still tried to happen even when I was formatting new portable disks, why is that?

What is the correct way to do what I need??

Thanks

[PTide]

Hi

I have set Veeam to process jobs serially so only one job is running at any time

May I ask you what's the reason for such design? Why not to place all VMs in a single job?

I scheduled the backups to start at 5min intervals and let veeam sort out when they can actually run.

Just to make sure that we are on the same page - did you use this option + backup window?

What is the correct way to do what I need??

Please specify your goal. Do you want to get a separate backup chain for each VM on your USB drives or is it something else?

Thank you.

[davek0974]

1 - I prefer to see each vm as a separate job - personal preference.
2 - The backups should start from 6pm onwards, sequentially.
3 - My goal is to have backups for each VM stored on the machine - this much appears to be working OK. Then I need a copy of the backup for each VM on USB disk ONCE per day only and must be a copy of the backup made previously at 6pm. These copies should start after midnight as I don't like trying to write the backups to the main disk at the same time as trying to read the data to be copied, personal preference again here.

Then we need a "safe" window in which to unplug the USB drive each day and fit the next one.

That is all we need.

Dave

[PTide]

The backups should start from 6pm onwards, sequentially.

There are two ways to do that:

- job chaining (has some drawbacks)
- limiting the amount of concurrent tasks to 1, so your proxy server will wait until the previous job finishes. The other potion is to do the same for repository.

I the latter case all your jobs will have to use the same proxy/repo. Also if any of your VMs has a number of disks > 1 that will result in splitting the job into several phases - disk1 processing, disk2 processing, ... disk N processing -, which might result in slower job processing.

Then I need a copy of the backup for each VM on USB disk ONCE per day only and must be a copy of the backup made previously at 6pm.

For that you'll have to set up a separate backup copy job for each backup job because a single backup copy job always creates one backup file, no matter how many source VMs/jobs are included. Each backup copy job should start 5-7 minutes later than its source backup job starts and have an interval of 24 hours and the allowed backup windows should be set from midnight to 10am. So your "safety window" is going to be between 10am and midnight.

Thank you.

[davek0974]

Yes, the concurrent tasks is set to one.
Yes, some VM's have multiple disks, but does not seem to bother it.
The 6 backups are scheduled to start at 6pm, 6.05, 6.10, 6.15, 6.20, 6.25 each week day.
The backup copy jobs are scheduled to run at 0.05, 0.10, 0.15, 0.20, 0.25, 0.30 each week day.
I have the backup jobs set to once every day, is that the same as an interval of 24hrs????

With these settings i still get the "Copy interval has expired" error on the first run and then they seem to run ok an hour or so later???

[PTide]

The backup copy jobs are scheduled to run at 0.05, 0.10, 0.15, 0.20, 0.25, 0.30 each week day.

Please try
6:10 - 6:15 - 6:20 - 6: 25 - 6:30 - 6:35

I have the backup jobs set to once every day, is that the same as an interval of 24hrs????

Yes.

[davek0974]

Ok, got that set, not totally sure why yet but will see how it runs tonight.

Thanks

[davek0974]

Nope, I now get an email from backup with success immediately followed by an email from backup copy job with warning "Some VMs were not processed during the copy interval" for every job.

The backup copy job does eventually complete at about 3am

[PTide]

"Some VMs were not processed during the copy interval" for every job.

Were any files created on your backup copy repository?

[davek0974]

Yes, eventually at around 3.00am

[PTide]

Did it copied all 6 restore points or just some of them?

[davek0974]

Just one point plus the full backup for each job

[PTide]

If it copied full backup for each job then what is that "one point"? Backup copy job creates either one initial full .vbk or one increment. Please elaborate.

[davek0974]

Seems to be a ".vib" file?
Plus the one full backup copy.

[foggy]

Let the job do it's job during the next interval and if the issue persists, please contact technical support to have a look at your setup. Thanks.

[davek0974]

Still not right,

My data transfer window is Mon-Fri 00.00 to 10.00 am

On friday, backup completed 100%, success emails for all jobs

After that i get warnings saturday morning and sunday about backup interval again,
There was NO backup copy of the full backup on Friday evening, all i have on the portable disk is three files for each job - a .vbk, and two .vib files but they are all the same size, 30,884 and all are too small to be real backup copies.

Something is not working correctly i think

[PTide]

Something is not working correctly i think

Please open a case with support and post your case ID here.

Thank you.

[ChrisGundry]

The backup copy job will copy the latest restore point it sees in the 'source' repository when the job starts.
So if your local backup starts at 6pm and finishes at 11pm then your backup copy job will not start doing anything until 11pm.

So I would suggest you tell your backup copy job to start at 10pm, every 1 day. Set the number of restore points to keep to be what you need for your policy. The job will start at 10pm daily and see no new restore points when it starts, then it will wait until it sees a new restore point which it will when the source job finishes at 11pm. It will then start copying the restore point to the destination repository which in your case is the USB drive.

The backup copy job creates incremental restore points. So the .vbk file is the original backup file and then each other time it runs it will create a significantly smaller .vib incremental file. This is incremental so only contains changes and will be much smaller as a result. Say you are keeping 10 restore points. When the job runs for the 11th time it will merge the original vbk file from run 1 with the vib file from run 2. This process will take a long time.

I think you will always get an error when you unplug the USB drive when the backup copy job is still running in an idle state.

Personally if you don't require a local backup and another backup on the USB drive I would suggest you do the following:
1. Put all of the VM's in a single job (I would recommend this anyway, its much more efficient as you get de-dupe between jobs/VM's etc)
2. Run the job directly to the USB drive - To do that In a typical 7 days, rotating USB drives each day etc I would setup a repository for each drive and a job for each drive. Then set the jobs to run Monday, Tuesday etc. If you do that then as soon as the job finishes you can remove the drive. That way you get a single job/email with the status of all VM's and do not need to use a backupcopy job at all. I have done this on many many setups and it works great.

Notes:
Reverse incremental performance is affected by storage performance. USB storage performance is usually very poor.
Depending on how many restore points you want to retain reverse incremental is my usual choice and if your running 7 days/jobs/drives and keep 10 restore points per drive then you end up with daily backups for 10 weeks! Tidy

[davek0974]

Thats what i thought would happen but it backed up friday and copy did nothing apart from some very small .vib files all of which were exactly the same size, no full copy.

Thanks

Have opened a case.

[davek0974]

This morning i changed one of the copy jobs to hourly, after the schedule changed to the inactive transfer window i had scheduled.

It still has transferred data despite being told not to, the empty USB drive now has a folder on it for that job, contains three files - the chain, a .vib and a .vbk.

However the vib and vbk files are the same size and not big enough to contain much at all.

Why does the scheduler refuse to do what it says, this should be a simple task????

[foggy]

Having hourly backup copy when the source backup job is scheduled to run daily does not have sense, since there are simply nothing new to copy each hour. So copy job will create an empty restore point 23 out of 24 cycles per day. Anyway, I believe support engineers will be able to review your jobs configuration to verify the reasons of the observed behavior.

[davek0974]

Yes but if the backup USB disk has changed, the main and USB contents cannot match any more so as soon as the next hourly window came around I would expect it to start copying again???

It does not

Hence my title - "Why So Complicated"

[foggy]

Provided the repository is configured for rotated drives, yes, it should create the same restore point on each new disk inserted prior each new interval within a day. So, once again, better investigate this with the help of support.

[davek0974]

Case is opened


Thanks