Help with backup copy job (continuous)

[kjstech]

Hello, using 8.0.0.917 in a VMWare environment.

I backup to a FreeNAS box primarily but want to ensure I have a copy of these backups so instead of using rsync between two FreeNAS boxes, I am trying the backup copy job in Veeam. I set it to run continuously instead of picking a set time. I figured this would just make sure the backups between the two appliances were in sync. I'm not understanding why this is not the case because right now the status is Idle and the backups are not in sync. Shouldn't it never go idle unless the backups are in fact in sync?

I have a warning email for this job that shows 17 VM's out of 41 VM's sucessfull. The rest show Warning - Operation was canceled by user (and it shows our admin account). No one was here to cancel the job. Did another backup cancel the job? But if this is set to continuous, why didn't it resume once the other backup was completed? I also tried to force it and it asked me if I wanted to resync the backups, I said yes, but its still idle.

So on the destination I have a 1.56 TB vbk file, a 442GB .vib file, an 11 GB .vib file, a 145 GB vib file and a 30MB vib file. Total size is 2.01 TB.

On the source for this backup job is less, only 1.82 TB but there's two smaller vbk files and there are 8 vib files and 1 vbm file.

I guess I'm just looking at trying to understand what is going on here before I switch back to rsync which never syncs up because it is too slow (even over dedicated crossover interface between appliances).

[foggy]

Keith, the first thing you should keep in mind is that backup copy job does not simply copy backup files created by the corresponding backup job, but rather synthetically creates restore points on target repository using data available in the source repository, copying the latest VM state during each synchronization interval. Also, note that in case of reverse incremental mode, backup copy job is interrupted when it's source backup job starts (as it needs to write into the full backup file). I recommend to get acquainted with the corresponding user guide section and if it does not clear the things up for you, ask for further clarification here. Thanks.

[veremin]

Also, didn’t you specify backup window for the backup copy job by any chance? Thanks.

[kjstech]

I have it set to continuous. There is a backup window for other jobs though. Does that matter?

Sorry for the late reply. I didn't see this topic was updated until I received Sunday evening's veeam email.

Right now the Backup Copy Job 1 (41 VMs) has 24 warnings.
Sunday January 4, 2015 5:04:44 AM
The job heading says - Some VMs were not processed during the copy interval.
Start time 5:04:55 AM
End time 6:07:40 AM (+1)
Duration: 5:43:22
Processed and Read: 116.9 GB
Transferred: 65.7 GB (1.8x)
Backup Size: 73.6 GB
Dedupe: 1.0x
Compression: 1.6x

The first 17 VMs all say success.
The rest say Warning - Initial copy did not complete, and will resume on the next copy interval.

The previous day, Saturday January 3 9:58:22 AM has 1 error, 16 success, and 24 warnings.

The first failure says Incremental copy was not processed during the copy interval.
The 24 warnings all say the same as last night's job.... Initial copy did not complete and will resume on the next copy interval.


Any ideas how to get this to just run straight through?

[Gostev]

What is the connectivity between the two FreeNAS boxes?

[kjstech]

1gig Ethernet.

[Gostev]

Not an issue then. What about target FreeNAS box (amount, type and speed of hard drives + RAID level).

[kjstech]

Ok its freenas 9.2 with Intel core2duo E7300 @2.66 GHz, 8GB RAM and two 4TB WD RED SATA hard drives in raid-0. This yields a 7.1 TB size file system.

[Gostev]

What are the corresponding Backup Copy job's session bottleneck statistics?
Also, any hints in the Backup Copy job session log itself, or in VM-level log for VMs with warnings?

[kjstech]

It says:
Duration 11:43:39
Processing Rate: 64 MB/s
Bottleneck: Target
In the individual jobs it shows target is in the upper 90% range, in fact a few say 99% target, network 1%, Proxy 17%, Source 40%.

[kjstech]

The uptime on the freenas box was 257 days. I rebooted it so maybe that will help if there was a memory leak or performance issue. We shall see tomorrow.

[foggy]

Bottleneck target means that the write speed to the target storage is the weakest point in data processing chain that does not allow the job to finish transfer within the synchronization interval. You may want to seed the initial job run.

[kjstech]

Seed it? The file names are different. How do you go about this?

Example:
Source: vnxfs12015-01-03T012808.vbk
Destination: Backup Copy Job 12014-12-31T000000.vbk

[foggy]

I was talking about transferring the original backup to the target side and then mapping backup copy job to it.

[kjstech]

Thank you, I am trying that now. On my second freenas box I ssh'd to it and mounted the share and I am copying the file vnxfs12015-01-03T012808.vbk which is 730GB.

Then I take it I create a new backup copy job and say its for job vnxfs1 (thats a group of VM's that is stored i this file), and click map backup and browse to my second freenas box \ vnxfs1 (folder I created that is the same name as the original folder that the backup job creates).

We do full backups on weekends, hence the file is dated 1/3, but I take it after tonight backup runs, the backup copy job will take everything that incrementally changed between 1/3 and now and just copy that data?

[foggy]

Correct.

[kjstech]

This did not work. All jobs failed with this error:
Processing compuername Error: File does not exist. File: [\\fnbackup2\backups\backups\vnxfs1\vnxfs12015-01-07T000109.vib]. Failed to restore file from local backup. VFS link: [summary.xml]. Target file: [MemFs://frontend::CDataTransferCommandSet::RestoreText_{caed2871-32f0-4b3e-b302-31087e279547}]. CHMOD mask: [0]. Agent failed to process method {DataTransfer.RestoreText}.
Processing brick Error: File does not exist. File: [\\fnbackup2\backups\backups\vnxfs1\vnxfs12015-01-07T000109.vib]. Failed to restore file from local backup. VFS link: [summary.xml]. Target file: [MemFs://frontend::CDataTransferCommandSet::RestoreText_{07067a72-b578-401b-b80f-573df946ca45}]. CHMOD mask: [0]. Agent failed to process method {DataTransfer.RestoreText}.
Some VMs were not processed during the copy interval


I copied the lasts full from last Saturday and I also had to copy a vnxfs1.vbm file or else when I rescanned this repository it wouldn't see the backup to map too.
Directory of X:\backups\vnxfs1

01/08/2015 02:01 AM <DIR> .
01/07/2015 04:19 PM <DIR> ..
01/08/2015 02:01 AM 31,584,256 copy vnxfs1 to fnbackup22015-01-08T020000
.vib
01/08/2015 02:01 AM 962,013 vnxfs1.vbm
01/07/2015 04:22 PM 31,584,256 copy vnxfs1 to fnbackup22015-01-07T020000
.vib
01/07/2015 02:44 PM 784,897,746,944 vnxfs12015-01-03T012808.vbk
4 File(s) 784,961,877,469 bytes
2 Dir(s) 5,116,510,679,040 bytes free

[kjstech]

It worked last night. I copied all the files not just the main one, but all the incremental as well and now it works..

It's weird, veeam must have a dual personality. It says the job was successful but I have this strange message at the top of the notification email. It does look like this must have been cached because the dates are from before I corrected this issue. The job was successful on Friday January 9, 2015 2:00:08 AM:

Code: Select all

Failed to merge full backup file. Terminated at 1/9/2015 3:54:23 AM
Failed to create restore point 1/4/2015 12:00:54 AM
Failed to generate points Error: Access is denied. Failed to create or open file [\\fnbackup2\backups\backups\vnxfs1\vnxfs12015-01-03T012808.vbk]. Failed to backup text locally. Backup: [veeamfs:0:f9894088-fce2-4012-835b-a128c05a131c (vm-17575)\summary.xml@\\fnbackup2\backups\backups\vnxfs1\vnxfs12015-01-03T012808.vbkVBK: 'veeamfs:0:f9894088-fce2-4012-835b-a128c05a131c (vm-17575)\summary.xml@\\fnbackup2\backups\backups\vnxfs1\vnxfs12015-01-03T012808.vbk']. Agent failed to process method {DataTransfer.Bac
Failed to merge full backup file. Terminated at 1/9/2015 3:57:44 AM
Failed to create restore point 1/4/2015 12:00:54 AM
Failed to generate points Error: Access is denied. Failed to create or open file [\\fnbackup2\backups\backups\vnxfs1\vnxfs12015-01-03T012808.vbk]. Failed to backup text locally. Backup: [veeamfs:0:f9894088-fce2-4012-835b-a128c05a131c (vm-17575)\summary.xml@\\fnbackup2\backups\backups\vnxfs1\vnxfs12015-01-03T012808.vbkVBK: 'veeamfs:0:f9894088-fce2-4012-835b-a128c05a131c (vm-17575)\summary.xml@\\fnbackup2\backups\backups\vnxfs1\vnxfs12015-01-03T012808.vbk']. Agent failed to process method {DataTransfer.Bac
Failed to merge full backup file. Terminated at 1/9/2015 4:02:07 AM
Failed to create restore point 1/4/2015 12:00:54 AM
Failed to generate points Error: Access is denied. Failed to create or open file [\\fnbackup2\backups\backups\vnxfs1\vnxfs12015-01-03T012808.vbk]. Failed to backup text locally. Backup: [veeamfs:0:f9894088-fce2-4012-835b-a128c05a131c (vm-17575)\summary.xml@\\fnbackup2\backups\backups\vnxfs1\vnxfs12015-01-03T012808.vbkVBK: 'veeamfs:0:f9894088-fce2-4012-835b-a128c05a131c (vm-17575)\summary.xml@\\fnbackup2\backups\backups\vnxfs1\vnxfs12015-01-03T012808.vbk']. Agent failed to process method {DataTransfer.Bac
Failed to merge full backup file. Terminated at 1/9/2015 4:08:29 AM
Failed to create restore point 1/4/2015 12:00:54 AM
Failed to generate points Error: Access is denied. Failed to create or open file [\\fnbackup2\backups\backups\vnxfs1\vnxfs12015-01-03T012808.vbk]. Failed to backup text locally. Backup: [veeamfs:0:f9894088-fce2-4012-835b-a128c05a131c (vm-17575)\summary.xml@\\fnbackup2\backups\backups\vnxfs1\vnxfs12015-01-03T012808.vbkVBK: 'veeamfs:0:f9894088-fce2-4012-835b-a128c05a131c (vm-17575)\summary.xml@\\fnbackup2\backups\backups\vnxfs1\vnxfs12015-01-03T012808.vbk']. Agent failed to process method {DataTransfer.Bac
Failed to merge full backup file. Terminated at 1/9/2015 4:18:53 AM
Failed to create restore point 1/4/2015 12:00:54 AM
Failed to generate points Error: Access is denied. Failed to create or open file [\\fnbackup2\backups\backups\vnxfs1\vnxfs12015-01-03T012808.vbk]. Failed to backup text locally. Backup: [veeamfs:0:f9894088-fce2-4012-835b-a128c05a131c (vm-17575)\summary.xml@\\fnbackup2\backups\backups\vnxfs1\vnxfs12015-01-03T012808.vbkVBK: 'veeamfs:0:f9894088-fce2-4012-835b-a128c05a131c (vm-17575)\summary.xml@\\fnbackup2\backups\backups\vnxfs1\vnxfs12015-01-03T012808.vbk']. Agent failed to process method {DataTransfer.Bac
Failed to merge full backup file. Terminated at 1/9/2015 4:37:19 AM
Failed to create restore point 1/4/2015 12:00:54 AM
Failed to generate points Error: Access is denied. Failed to create or open file [\\fnbackup2\backups\backups\vnxfs1\vnxfs12015-01-03T012808.vbk]. Failed to backup text locally. Backup: [veeamfs:0:f9894088-fce2-4012-835b-a128c05a131c (vm-17575)\summary.xml@\\fnbackup2\backups\backups\vnxfs1\vnxfs12015-01-03T012808.vbkVBK: 'veeamfs:0:f9894088-fce2-4012-835b-a128c05a131c (vm-17575)\summary.xml@\\fnbackup2\backups\backups\vnxfs1\vnxfs12015-01-03T012808.vbk']. Agent failed to process method {DataTransfer.Bac

[foggy]

It worked last night. I copied all the files not just the main one, but all the incremental as well and now it works..

Yes, you need the entire chain to be available when mapping the job.

Please contact technical support regarding the error messages.