Glacier -- Why is Veeam trying to access old files which breaks the job?

[kellino]

We have a few design parameters which includes NOT using VTL. We deployed AWS File Gateway which worked great until files started getting glacerized as per our S3 Lifecycle policy.

We understand that Veeam expects all files in the repository to be online, but this should be avoidable. A few questions:

1) Why is Veeam trying to read files that are over 30 days old on the repository, when we simply want Veeam to continue depositing backing copies on S3.
2) Why does the above failure cause the entire job to stop processing/

We understand that we have the option to use IA for storage but this is 10x the cost.

In 2019 there has to be a way for the Backup Copy jobs to run and not care about the old files that have been archived. Veeam simply needs to read from the on-prem location and copy the backups to the S3 repository -- but without trying to access old/archived files on the S3 repository. How can we achieve this?

Thanks

[backupquestions]

1) Why is Veeam trying to read files that are over 30 days old on the repository, when we simply want Veeam to continue depositing backing copies on S3.
Because those files are still subject to a retention policy in Veeam. A backup copy job with GFS enabled for example, creates some monthly backups, and then you take away 3 of them to glacier all of a sudden... Well, how is Veeam supposed to know where it is in retention policy now? Part of the backup job is to process that retention and remove older files itself. You are pulling the rug out from under it and expecting the code to adapt.

2) Why does the above failure cause the entire job to stop processing?
Same as 1. end of job retention is processed and you are confusing it with removing some of its files. Now it can't process retention policy so it never knows where it is and how many weekly/monthly/yearly to keep etc.

Backup copy jobs are not designed for taking one off "archival" fulls like this. They are always still subject to retention policy.

Some options for you are
1. Take backups to real tapes, they can be archival of course.
2. Use a virtual tape library (you ruled this out in your strategy, not sure why but it was probably not a good decision)
3. Take veeamzip one off backups, maybe not the best solution and I don't think Veeam can write it directly to S3 repo.

Perhaps you should have consulted with Veeam engineers before designing a strategy and setup and expecting the software code to adapt to your wishes.

[kellino]

Thanks for the feedback but let me offer a suggestion.

First off we set retention at a very high level into the future with the hopes that Veeam would not attempt to do anything with these files before our lifecycle policy addresses. Veeam can tell from both the catalog and the directory data that these files aren't even close to the limit and do not need to be read.

Second, it shouldn't be terribly complex to add a checkbox that tells Veeam to NOT manage retention on the target. What we are trying to enable here is the use of cold storage such as Glacier. There are ways to do this with the S3 API but for some reason Veeam doesn't allow the API for either Backup or Backup Copy targets even though its supported in SOBR. Until this is enabled in the product, we are not given many options in which to consume Glacier -- but what could be done is to provide customers with a checkbox that effectively says "Don't have Veeam manage my retention because I'd like to consume Glacier and manage it myself -- just write the files and don't try to read the old stuff".

[veremin]

Backup copy job uses forward forever incremental method, so all files in current backup chain are subject to access and modification.

Currently, you can consume Glacier only via Storage Gateway configured in VTL mode.

Thank you for your feature request, consider your vote counted.