Discussions specific to object storage
Post Reply
sasilik
Enthusiast
Posts: 54
Liked: 3 times
Joined: Jun 12, 2014 11:01 am
Full Name: Markko Meriniit
Contact:

Backup copy job and SOBR offload behavior when performance tier fills

Post by sasilik » Oct 16, 2019 7:41 am

We decided to try out how backups work with S3 compatible cloud storage and I configured VEEAM for that. I added scale-out storage where there was 17TB local storage as performance tier and S3 compatible cloud storage as capacity tier. I created backup copy job for one backup job which vbk file size was 7TB. Settings were that it copies every 1 day, restore points to keep 2, weekly 4. Scale out repository Placement policy was Data Locality, capacity tier was set to move backups to object storage when backup files are older than 2 days and override was set at 90%

After first run of backup copy job I realized that total size of per VM vbk files are ~10TB and there may be free space problems afterward but as I was testing then I decided to let it run. First run was on friday and I had weekly schedule set on weekend. On monday I had one vib file for every VM and for some VM's I also had weekly GFS file, <VMname_W.vbk>. And about 1TB free space and bunch of failed VM's in backup copy job. It seems that job tried to create weekly backup from one VM which was about 1TB and it failed because of lack of disk space. And after that the backup copy job did not do weekly backups from any other VM's, even not from ones which were 20GB. Also for the files which failed there started to appear vib files. At the end of the week some had even 5 vib files after vbk. I also hoped that maybe offload job will move weekly backup files to cloud storage but that also didn't happen.

I opened case, ID# 03808439, but it wasn't really helpful for understanding what was happening. As the whole mess was quite probably because of lack of disk space then I just deleted all backup files and created new backup copy job with less data. But I am curios if such beavior is expected after backup copy fails because performance tier has too less space for one big VM.

veremin
Product Manager
Posts: 17017
Liked: 1462 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Backup copy job and SOBR offload behavior when performance tier fills

Post by veremin » Oct 16, 2019 5:22 pm

So,

- The backup copy job failed to create a GFS backup due to lack of free space
- Then, it stopped, since there was no space available (even if it had started processing other VMs, it would have eventually failed, once it would have reached this huge VM)
- Offload activity didn't transfer anything, because there were no healthy GFS restore points (they only points of backup copy job that are subject to offload)

The described behaviour looks totally expected. I can only suggest increasing disk space up to the point where it's enough to accommodate at least two full backups.

Thanks!

sasilik
Enthusiast
Posts: 54
Liked: 3 times
Joined: Jun 12, 2014 11:01 am
Full Name: Markko Meriniit
Contact:

Re: Backup copy job and SOBR offload behavior when performance tier fills

Post by sasilik » Oct 16, 2019 7:46 pm

Just wanted to clarify, "healthy GFS restore points", these are points when all VM's in backup copy job are processed? Not just per VM individual weekly backup files?

veremin
Product Manager
Posts: 17017
Liked: 1462 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Backup copy job and SOBR offload behavior when performance tier fills

Post by veremin » Oct 17, 2019 4:35 pm 1 person likes this post

Correct, backup is considered successful, if all VMs it consists of were processed successfully. Thanks!

Post Reply

Who is online

Users browsing this forum: M.Lewis and 3 guests