Best type of backup job for BACKUP COPIES

[mafiasilk]

We are taking backups offsite with Veeam Backup Copies, is there any type of Backup Job Veeam recommend for this?
Is it ok to use Forever Incremental Backups as the source job - or is there a better option.

[foggy]

Asif, for backup copy job, it doesn't matter what source backup jobs are set to: forward or reversed incremental mode.

[mafiasilk]

Thanks for the reply,
My thinking around this question was about how much backup data is going to be sent over the WAN.
If it's a reverse incremental I was thinking more data will be sent as the job will be like a new Full Backup, whereas the forever incremental's will be smaller.

Is that right?

[foggy]

No, since backup copy job does not copy files but rather synthetically creates restore points in remote location from the changed blocks extracted from the source storage. So in both cases only changes will be transferred over to target.

[mafiasilk]

Brilliant!
Thats the technical answer I needed.

Thank you very much.

[foggy]

You're welcome. Feel free to ask for any additional clarification, if required.

[Gostev]

Forward incremental provides much faster source (sequential I/O when read incremental data, as opposed to random I/O with reversed incremental). I guess this makes it "best" out of the two. However, Backup Copy will work from either, and with fast primary backup storage, the difference will be negligible anyway.

[itfnb]

I just had this thought:

Does the method a Backup Job is using (Forward or Reverse Incremental) impact the performance of a Backup Copy Job using its files?

I currently am using an internal RAID volume for short-term storage (1 week) of my Backup Jobs that are using Reverse Incremental. My main GFS Backup Copy Job (going to an external SAS enclosure) says the "Source" is the bottleneck. Would there be any performance advantage on the Backup Copy Job by switching the Backup Jobs to Forward Incremental?

Thanks!
-Jim

[foggy]

Jim, please review considerations given above for an answer. Thanks.

[itfnb]

Thanks!

[darryl]

We're a version 7 customer, currently backing up to (slow SATA) HP MSA disk then offloading to tape outside of Veeam. Our Veeam backup jobs are a mix of forward incremental (reduce I/O) with synthetic fulls and reverse incrementals (to save space). Our B2D array is pretty slow…

We have an entirely new environment now, with a surprisingly quick HP MSA 2040 (10k SAS). Long term archival will be handled by a pair of HP StoreOnce 4500s, one local to the Veeam server and one at a DR site. The 4500s will handle replication themselves. We’re using ver 8.

Question: Due to having the backup copy jobs to the StoreOnce, are there any special considerations (besides the normal ones) we should be looking at for the backup mode for the primary backup jobs? Currently thinking of Reverse incremental or forever incremental. Does this decision affect the backup copy jobs in any way?

Eg:
Reverse incremental to MSA (fast disk).
- We’ll keep 20 or so restore points as part of the backup job.
Backup copy job to local StoreOnce (this is always done the same way, regardless of the source?).
- We’ll implement a GFS rotation on the backup copy job.

Thanks.

[foggy]

Please review the thread above and feel free to ask for any additional clarification, if required. Thanks.

[obroni]

I understand for the bulk data part of the transfer forward incremental is probably faster than reverse, but does this apply to the vmx,vmxf,nvram part of the job?

We have a Forward Incremental backup job with around 60 vm's and including retention points 11TB of data. We also have a Copy Job set to copy all VM's, we are noticing that there is around a 10 minute delay between each VM starting and the actual disk data getting copied.

Looking through the job logs I can see that it seems to be accessing more than just the current .vib file. Also looking at performance monitor in windows I can see a large amount of disk activity (200MB/s+) to the vbk and all the vibs during this 10 minute period. This is suggesting it is having to do some sort of scan through all the incrementals to get the vmx,vmxf,nvram files.

I'm wondering if reverse incrementals would be any faster.
Any ideas?

[foggy]

Nick, previous restore points are indeed opened during backup copy job from forward incremental chain (though for metadata reading, vmx, vmxf, or nvram files have nothing to do with it). While with reverse incremental only the latest restore point (full) needs to be opened, the performance benefit you could get from this fact is likely to be negated by random read that is done in this case.

[obroni]

Hi Foggy,

Thanks for your reply.

I'm definitely seeing bursts of traffic that roughly correspond to the saving of the metadata files. As soon as the green tick appears on one of them, the disk activity stops until the next file begins. I can investigate this some more and provide screenshots if its helpful.

In terms of performance, I think it will only be faster if the VM is large and has a large amount of changed blocks and you have a fast enough WAN link. In my scenario some small VM's only spend a couple of minutes copying data but they can be scanning the vib files up to 10 minutes prior to the data copy starting.

[foggy]

You can always test both methods and decide which one performs better in your particular setup.

[obroni]

Hi Foggy,

I've done some tests and reverse incremental is faster over the lifetime of the job. This is mainly due to the WAN being the bottleneck rather than the random IO of the source disks and that the VM's are not doing this large read behaviour at the start.

I would be really interested to know if you have any knowledge about the pre processing behaviour.

Is this 10 minute delay where the disks seem to read in around 50-100GB of data at the start of each normal?

I'm guessing this would be effected by the number of incrementals but I was wondering if it would be effected by size of VIB/VBK's and/or number of VM's in the job?

Many Thanks,
Nick

[foggy]

It is indeed affected by how deep the incremental chain is, as well as the number of VMs in the job and (which is more important) their size. Basically, for each 1 TB of source VM data you need 308 MB to store metadata with the default block size of 1 MB (actually, 154 MB *2 due to redundancy) and entire metadata is read from each restore point when the backup copy job starts. You can do some calculations based on the size of your VMs and decide on whether what you're seeing is expected.

[obroni]

From that calculation I would say that what I am seeing is not expected. There is probably around 10TB of source VM data, so by your figures roughly 3GB of metadata. I'm seeing the source repository disks read 150-200MB/s for a period of around 10 minutes during the vmx/vmxf/nvram stage.

[foggy]

I'd suggest to collect logs for the jobs in question and let our engineers look at what is actually happening during that time.

[obroni]

I have found what is causing the delay, the block from the log below happens for each vmx,vmxf,nvram and vmdk file at the start of each VM. Each processed file seems to take around 2 minutes or a total of 10 (depending on number of vmdks'). Whatever the DataTransfer.RestoreText function does, it seems to read through each vib and vbk and consumes in our case a large amount of time.

As requested I will raise a support ticket and get them to look at it.

Code: Select all

[13.11.2015 10:02:15] <01> Info             [AP] (9eab) command: 'Invoke: DataTransfer.RestoreText\n{\n  (EStringArray) Links[] =\n  {\n    
veeamfs:0:3b47358e-4701-4cbc-84b6-986892554a6c (vm-27186)/********.vmx@F:\VEEAM_BACKUP\Windows Servers\Windows Servers2015-11-12T193142.vib,\n    
veeamfs:0:3b47358e-4701-4cbc-84b6-986892554a6c (vm-27186)/********.vmx@F:\VEEAM_BACKUP\Windows Servers\Windows Servers2015-11-11T193143.vib,\n    
veeamfs:0:3b47358e-4701-4cbc-84b6-986892554a6c (vm-27186)/********.vmx@F:\VEEAM_BACKUP\Windows Servers\Windows Servers2015-11-10T193147.vib,\n    
veeamfs:0:3b47358e-4701-4cbc-84b6-986892554a6c (vm-27186)/********.vmx@F:\VEEAM_BACKUP\Windows Servers\Windows Servers2015-11-09T193144.vib,\n    
veeamfs:0:3b47358e-4701-4cbc-84b6-986892554a6c (vm-27186)/********.vmx@F:\VEEAM_BACKUP\Windows Servers\Windows Servers2015-11-08T193128.vib,\n    
veeamfs:0:3b47358e-4701-4cbc-84b6-986892554a6c (vm-27186)/********.vmx@F:\VEEAM_BACKUP\Windows Servers\Windows Servers2015-11-06T193148.vib,\n    
veeamfs:0:3b47358e-4701-4cbc-84b6-986892554a6c (vm-27186)/********.vmx@F:\VEEAM_BACKUP\Windows Servers\Windows Servers2015-11-05T193138.vib,\n    
veeamfs:0:3b47358e-4701-4cbc-84b6-986892554a6c (vm-27186)/********.vmx@F:\VEEAM_BACKUP\Windows Servers\Windows Servers2015-11-04T193148.vib,\n    
veeamfs:0:3b47358e-4701-4cbc-84b6-986892554a6c (vm-27186)/********.vmx@F:\VEEAM_BACKUP\Windows Servers\Windows Servers2015-11-03T193143.vib,\n    
veeamfs:0:3b47358e-4701-4cbc-84b6-986892554a6c (vm-27186)/********.vmx@F:\VEEAM_BACKUP\Windows Servers\Windows Servers2015-11-02T193132.vib,\n    
veeamfs:0:3b47358e-4701-4cbc-84b6-986892554a6c (vm-27186)/********.vmx@F:\VEEAM_BACKUP\Windows Servers\Windows Servers2015-11-01T193150.vib,\n    
veeamfs:0:3b47358e-4701-4cbc-84b6-986892554a6c (vm-27186)/********.vmx@F:\VEEAM_BACKUP\Windows Servers\Windows Servers2015-10-30T193144.vib,\n    
veeamfs:0:3b47358e-4701-4cbc-84b6-986892554a6c (vm-27186)/********.vmx@F:\VEEAM_BACKUP\Windows Servers\Windows Servers2015-10-29T193145.vib,\n    
veeamfs:0:3b47358e-4701-4cbc-84b6-986892554a6c (vm-27186)/********.vmx@F:\VEEAM_BACKUP\Windows Servers\Windows Servers2015-10-29T001514.vib,\n    
veeamfs:0:3b47358e-4701-4cbc-84b6-986892554a6c (vm-27186)/********.vmx@F:\VEEAM_BACKUP\Windows Servers\Windows Servers2015-10-28T193358.vib,\n    
veeamfs:0:3b47358e-4701-4cbc-84b6-986892554a6c (vm-27186)/********.vmx@F:\VEEAM_BACKUP\Windows Servers\Windows Servers2015-10-27T193137.vib,\n    
veeamfs:0:3b47358e-4701-4cbc-84b6-986892554a6c (vm-27186)/********.vmx@F:\VEEAM_BACKUP\Windows Servers\Windows Servers2015-10-26T193146.vib,\n    
veeamfs:0:3b47358e-4701-4cbc-84b6-986892554a6c (vm-27186)/********.vmx@F:\VEEAM_BACKUP\Windows Servers\Windows Servers2015-10-25T193135.vib,\n    
veeamfs:0:3b47358e-4701-4cbc-84b6-986892554a6c (vm-27186)/********.vmx@F:\VEEAM_BACKUP\Windows Servers\Windows Servers2015-10-23T193147.vib,\n    
veeamfs:0:3b47358e-4701-4cbc-84b6-986892554a6c (vm-27186)/********.vmx@F:\VEEAM_BACKUP\Windows Servers\Windows Servers2015-10-22T231637.vbk,\n                                                                                                                                
                   \n  }\n  (EBoolean) Option.Remote = false\n}\n'
[13.11.2015 10:03:41] <52> Info           [AP] (9eab) output: <VCPCommandResult result="true" exception="" />>