Backup Copy Jobs - Failing when weekly rollover occurs.

[sunnydays]

Hi Guys,

I thought I would log this issue here as I continue to go around in circles with support and this issue is treated lightly each time I log it!
The current case is #01680532. There was another case which the support engineer spent some time investigating, but I can't find it in my account, case #01156232.

A quick summary of the issue,

We have the following daily backup job running,

• - Forward Incremental - Forever
  - 28 Day Retention
  - Huawei N2000 Repositories ( 4 x1GB NICS - Windows 2012 / Veeam Transport)
  - Compress / Inline Dedupe Enabled

We then have a backup copy job to a deduplication device,

• - 7 Day Retention
  - 5 Weeklys
  - 3 Monthlys
  - Quantum DXi4700 Repository (3x1GB NICS - SMB / CIFS)
  - Compress / Inline Dedupe Disabled

The backup copy jobs seem to run as they should. However at any given time we start receiving the following error (can be 1 week or even 4 weeks),

• Error: File does not exist. File: [\\10.55.64.220\V1-FLE-PStQRep01\V1-PP-BB-Copy_1\V1-PP-BB-Copy_12016-01-19T000000.vbk]. Agent failed to process method {Stg.OpenRead}. All tasks have already been processed

Below is an extract that Veeam support found fr another issue

A couple of other times this was due to Windows updates and an unexpected reboot. The last two times this has occurred, there were no interruptions.

We have completed the following under different directions / cases,

• - Upgrade from Update 2 to 2b (Error at the time: The process cannot access the file because it is being used by another process Failed to rename file[/color])
  
  Starting v8 Update 2, target data mover agent caches metadata from backup files and keeps backup copy job chain opened for this purpose. This was done to improve performance of storages that are not capable of high random I/O. If you're satisfied with the backup copy performance, you can disable caching by setting the AgentReadOnlyCache registry value 0 (this, however, will affect all your jobs) or 2 (this will affect only those jobs that have inline deduplication turned off).

• - Increase the SessTimeout to 300 seconds (\HKLM\SYSTEM\CurrentControlSet\Services\LanmanWorkstation\Parameters\SessTimeout)

When this issue occurs, there doesn't appear to be a way to recover and we have to start the entire chain again. Considering we are trying to achieve a Grandfather-father-son retention period, this is a little concerning.

Anyone's experiences or thoughts around this would be appreciated!

Thanks,
Sunny

[sunnydays]

Oh and there are some new features in v9 that I am interested in that may alleviate this issue or reduce the impact,

• -Active full backups with backup copy jobs. This added feature will help improve local (on-site) backup copy performance and reduce the load on deduplication appliances by eliminating the data rehydration required to process the backup copy job retention policy, or to create a new GFS (Grandfather-Father-Son) restore point. Enabling this option will disable a full backup transformation (oldest incremental backups will no longer be merged into the full backup file for retention processing). Instead, GFS full backup files will be created by copying the most recent VM state data from the primary backup storage in its entirety.

• -Backup copy parallel processing. With v9, backup copy jobs will now process multiple VMs in
  parallel, just like primary backup jobs. This improves the backup copy and retention processing
  performance due to removing “dead time” between each VM, and will further speed up processing
  when per-VM backup file chains are enabled on the target backup repository

Is there anyone out there that use these functions in their production environment?

Cheers,
Sunny