Backup copy job and SOBR offload behavior when performance tier fills

[sasilik]

We decided to try out how backups work with S3 compatible cloud storage and I configured VEEAM for that. I added scale-out storage where there was 17TB local storage as performance tier and S3 compatible cloud storage as capacity tier. I created backup copy job for one backup job which vbk file size was 7TB. Settings were that it copies every 1 day, restore points to keep 2, weekly 4. Scale out repository Placement policy was Data Locality, capacity tier was set to move backups to object storage when backup files are older than 2 days and override was set at 90%

After first run of backup copy job I realized that total size of per VM vbk files are ~10TB and there may be free space problems afterward but as I was testing then I decided to let it run. First run was on friday and I had weekly schedule set on weekend. On monday I had one vib file for every VM and for some VM's I also had weekly GFS file, <VMname_W.vbk>. And about 1TB free space and bunch of failed VM's in backup copy job. It seems that job tried to create weekly backup from one VM which was about 1TB and it failed because of lack of disk space. And after that the backup copy job did not do weekly backups from any other VM's, even not from ones which were 20GB. Also for the files which failed there started to appear vib files. At the end of the week some had even 5 vib files after vbk. I also hoped that maybe offload job will move weekly backup files to cloud storage but that also didn't happen.

I opened case, ID# 03808439, but it wasn't really helpful for understanding what was happening. As the whole mess was quite probably because of lack of disk space then I just deleted all backup files and created new backup copy job with less data. But I am curios if such beavior is expected after backup copy fails because performance tier has too less space for one big VM.

[veremin]

So,

- The backup copy job failed to create a GFS backup due to lack of free space
- Then, it stopped, since there was no space available (even if it had started processing other VMs, it would have eventually failed, once it would have reached this huge VM)
- Offload activity didn't transfer anything, because there were no healthy GFS restore points (they only points of backup copy job that are subject to offload)

The described behaviour looks totally expected. I can only suggest increasing disk space up to the point where it's enough to accommodate at least two full backups.

Thanks!

[sasilik]

Just wanted to clarify, "healthy GFS restore points", these are points when all VM's in backup copy job are processed? Not just per VM individual weekly backup files?

[veremin]

Correct, backup is considered successful, if all VMs it consists of were processed successfully. Thanks!

[mkh]

did you use ReFS on the performance tier? ReFs coupled with synthetic fulls can really help on the disk usage in this use case, where you want a full every week.
I know this would probably not save you when a full backup is 10 TB out of the 17 available, but it would be close

[sasilik]

It was NTFS. This was only testing and I tried to figure out how things work and how good is a speed with our service provider. Results were such that we did not go forward with it.