Using files from Backup Copy for replication & tape failing

[ddayton]

Veeam B&R 9.5 Update 2
VMware ESXi 5.5 Update3b

For several years I was running a set of 3 backup jobs with reverse incremental to a local NAS. I had 3 backup copy jobs that copied the local NAS backups to a remote NAS. I had 3 replication jobs that used the remote NAS to create a replica of the VMs on our remote VMware host. The remote host has a VM that is running as a Veeam Backup Proxy and has the remote NAS configured as its local drive using iSCSI, so most network traffic is kept off site. All worked great.

Recently I purchased a new server for the remote site with a tape library that has a LTO-6 SAS drive, so as recommended in the manual for tape backup I was changing from reverse incremental to forward incremental. I upgraded Veeam B&R from Update 1 to Update 2 before I began the changes, so not sure if there is any relation to this update.

I first disabled all jobs. I used Veeam to delete all of the local NAS backups leaving the remote NAS and remote host untouched. I created 4 new backup jobs, adding 1 new job for three more new VMs from a new VMware host and executed the 4 backup jobs. This ran overnite successfully, so the next morning I created 4 new backup copy jobs to copy the local NAS to the remote NAS and used Veeam to delete the old backup files on the remote NAS before I enabled them. This ran overnite successfully, so I created 4 new replication jobs that use the remote NAS as a source to create the VM replicas on the remote host. This was successful for all but three VMs that were in the 4th job (30 out of 33 were successful). In the 4th and largest job there are 16 VMs and the first night 3 failed to replicate using the remote NAS as the source. Error was failed to decompress LZ4 block; Incorrect header. I created a replication job for just one VM that failed and it failed again. I modified the relication job to use the source of the local NAS instead and it replicated ok. The difference in the NAS settings are the local NAS was set to "Use per-VM backup files" and the remote NAS was not. Today I made the remote NAS setting the same to see if that makes a difference, so I'll use Veeam to delete what is currently on the remote NAS.

Anyway over the weekend the tape job ran for the first time and it ran in to the same issue as the replication job using the remote NAS backup files.

The tape and replication jobs are now disabled until the backup copy jobs run again to the modified remote NAS and then tomorrow I will re-enable the replication jobs and let you know if this change was successful or not. If this fails then I'll try disabling the per-VM files on both sides.

I'm not sure if anyone else has used backup copy between two NAS devices that were not both configured to use per-VM files, but it might be a problem.

[foggy]

Typically this error means issues with the underlying storage. I recommend letting our support looking closer into this.

[ddayton]

No problems with the storage while using Reverse Incremental. But anyway after making the secondary NAS use the same per-VM file the problem went away. So i'm not sure if this was the change to Forward Incremental, but making the backup copy source and target the same fixed the issue. It was odd that the last cycle of the 33 VMs only 2 errored on the replicate and tape backup sourced by the remote NAS. It was always the same 2 VMs except the first run had 3 fail.

So problem resolved except with the file per-VM I need larger remote NAS storage now.