active/synthetic fulls - what about S3 - storage level corruption, health check?

[lando_uk]

So when running scheduled active or synthetic fulls on jobs, how does object storage deal with those when part of a capacity tier?
Do they receive more data on those days and consume more space (£££) or do they always use incremental no matter what the main jobs are configured for?

If forever incremental, how do they guarantee data consistency? Does having the storage-level corruption guard enabled check S3 as well as the local rep?

Apologises if this has been asked a million times.

[Gostev]

Indeed a million times but nevertheless a good time to provide an update on where we're going with this based on the empirical experience.

It's always forever incremental with object storage regardless of everything else.

Health check is currently not there with Capacity Tier because most cloud object storage keep multiple copies of the same object with a hash attached to each copy (part of S3 protocol) and transparently serves you another "good" copy if the "main" copy gets corrupted over time. We did not see issues with object storage returning corrupted objects in Support yet.

I do want to add at least a "health check light" (without checking the object content) to the Capacity Tier which merely ensures all objects are still in place. This is also based on our support experience as this is the issue we saw a few times in the past years. Weird stuff like only 1 object our of millions is magically missing.

Although now that I'm thinking about this, may be we should also add "full scan health checks" too considering the proliferation of on-prem S3-compatible object storage, where we can't expect every storage to store multiple copies of the same object. On-prem storage also makes it actually realistic to perform such "full scans" as the backup data will not be going over the Internet or resulting in massive egress charges some cloud object storage providers have.

[lando_uk]

Hi Gostev,
Thanks for the 1,000,001 explanation.
I guess that because S3 uses per-vm backups, the worse that can happen if something goes wrong with S3 99.999999999% - is that just one VM will be broken, unlike a VMK on local storage where whole lot could be affected.

[lando_uk]

One more question, if for example i'm on a 60 restore point retention and i lower it to 30, would there be excessive API calls (costs) on S3 when it's deleting old restore points?

[Gostev]

One block of one VM, to be precise or at least we haven't seen other scenarios yet.

What do you mean by excessive? They would all have to be issued one way or another, the only difference is whether these calls will be issued across those 30 days (as a part of normal 60 day retention policy processing) or all within a few hours (after you change the retention policy).

[Cragdoo]

In the article about health check for object https://helpcenter.veeam.com/docs/backu ... ml?ver=120 it states

To allow Veeam Backup & Replication perform the health check of data blocks located in the object storage repository, you must:

Configure a proxy appliance located in the object storage at the Mount server step of the New Object Repository wizard.

Does this also apply to Capacity tier in SOBR. Does the health check run against the Capacity tier or the just the Performance Tier?

[veremin]

Currently, there is no health for Capacity Tier, but we are thinking about providing a light health check option for Capacity Tier in one of the next product updates. Thanks!