How much local storage do I have to have?

[vkky2k]

I am new to veeam.

Let’s say that there are total of 200Tb backup data for the entire organization, and we want to keep 35 days retention. To reduce footprint in DataCenter, we want to use AWS S3 as the target to store backup data as much as possible, and use local storage as little as possible.

What is architecture for this purpose?
How much local storage do I have to have?
Do I have to have 200Tb local storage to store the first set or first day of full backup and only then the rest of retention or incremental backups could be stored onto S3?

If experts here can share your thoughts?

Thank you in advance!

[jamcool]

lots of tough questions. I can only share what we have in our setup. Also with S3 storage, you cannot do synthetic full backups which is something we have to have as if we had to do full backups, it would take 3-4 days.

One of our repositories is 338 TB volume. We have several jobs (primary VMWARE and few Windows agents) writing to it with both of Veeam's built in compression and deduplication. The largest job (VMWARE) is showing 285 TB for size. The other jobs are pretty small maybe totaling 30 TB for size. Our jobs are set to keep 14 backups with synthetic full backups on Saturday and daily incremental backups. The "used" backup data is 308 TB but we have 141 TB of free space in the repository. Our backups tend to get between 2-3 compression. This is primary Window servers with some Linux servers but no Exchange or SQL backups.

[vkky2k]

Correction:
200TB is the amount of data need to be backed up for the organization.

[Mildur]

You can use the restore point calculator to get an answer.

I have preconfigured it for you. You need to adjust change and compression to the real values in your organization. It depends on the data type.
I assume, that you will go with refs or xfs FASTClone.

https://rps.dewin.me/?m=1&s=204800&r=35 ... ,0,0,1,0&e

[vkky2k]

First and foremost, among the final amount 537.05Tb, how much data or what are these data can be pushed to AWS S3 and what are those have to be stored in local storage?

Again, to reduce the footprint, we wanted to store backups to S3 as much as possible

[Mildur]

Do you have adjusted the change and compression rate do the real values in your Environment? The final amount could be significant different from my calculation

You can configure „move policy“ for the capacity tier.
This way, you could move the backups from your local storage to the s3 storage as soon a new backup chain has started.

For the s3 storage calculation, you can choose the same amount from my calculation. S3 storage has the same behaviour for storing backups like on refs/xfs. Only changed data will be uploaded. There will not be another „full“ in the s3 storage after the first one.

[vkky2k]

If I understand you correctly, the reasonable amount if local storage I have to prepare is the amount of the first full backups which is 100Tb in this example, after compression, and the rest of all incremental / synthetic full cab be uploaded to S3?

So,
local storage = 100Tb + some work space
S3=437.05Tb

Correct?

[Mildur]

and the rest of all incremental / synthetic full cab be uploaded to S3?

The Move Policy moves only sealed chains to the capacity tier. An active chain will never be "moved" until a new Fullbackup was done. Your local storage will always need to store synthentic fullbackups and their incremental backups.
As soon they reach your configured "Move backups to object storage as they age out of the operational restores window" period, they will be moved to capacity tier and will be deleted on the local storage.
https://helpcenter.veeam.com/docs/backu ... ml?ver=110


With the move Policy and weekly synthentic fulls, you will have always between 7-14 days of restore points on your disk.
If your vbk is 100TB, and the incrementals are 10TB, then you will need around 260TB storage.
But we can tell you, how big your Fullbackup is. And your incrementals. You have to find out that by yourself todo a correct calculation.
https://rps.dewin.me/?m=1&s=204800&r=7& ... ,0,0,1,0&e

Read more about " the move Policy" here:
https://helpcenter.veeam.com/docs/backu ... ml?ver=110

[vkky2k]

“ you will have always between 7-14 days of restore points on your disk.”

Are 7-14 days restore points you suggested, or have to be? To explain my point, is it possible to store 3-7 days restore points on my local disk?

Thank you for being so patient!

[Mildur]

To explain my point, is it possible to store 3-7 days restore points on my local disk?

Sorry, there was a wrong thinking of me. I have to correct my statement.
If you configure move policy period to 0 day, the Restore Points should be moved away (deleted on local storage) directly after a new synthetic full.

This way, only 1-8 restore points should be on the local storage.

If you want to have less restore points, you need todo more synthetic fulls in a week (for example on wednesday and on sunday)

[vkky2k]

I have one more follow-up, and then I am done.

For 35 retention, in the list of calculations, why would you list 41 days instead of 35 days?

Thank you!

[veremin]

It's because of how forward incremental method with periodic full backups works. Thanks!

[vkky2k]

Based on the conversation, it sounds like I still have to purchase 170Tb minimum local storage in my case.

To further reduce the footprint on-compus, is there anyway I can specify the size as the threshold as the point of starting tiering to S3?

For instance, I only have 50TB local storage and once the backup data reached to this threshold, then data is going to start to move to S3.


Thank you for your further help!

[Mildur]

Yes, you can do that with the override option.
But only sealed chains will be moved. You need the storage for at least 1 full backup and all incrementals of that chain. 50 TB local backup storage is not enough for your case.

To override behavior of moving old backups, click Override, select the Move oldest backup files sooner if scale-out backup repository is reaching capacity check box and define a threshold in percent to force data transfer if a scale-out backup repository has reached the specified threshold.

https://helpcenter.veeam.com/docs/backu ... ml?ver=110

[vkky2k]

Right, that would be 170Tb minimum local storage in my case.

[Mildur]

Yes, that could work with reFS or xfs.
Calculate your storage to have at least two backup chains with full and their increments. Veeam needs to be create a new fullbackup before the old chain is sealed and can be moved. I correct my previous answer, you need at least 2 fullbackups. But with xfs and refs, not the entire space will be needed.

[steamrick]

Agreed on having enough space for at least two separate fullbackups.
If you compressed fullbackup is around 100TB I'd want 250~300TB to work with. Enough for two active full and a couple incrementals.

Having to delete your old backup chain before you can build a new one (as might be necessary if the old one is failing the health check or similar) is about as fun as gargling dishwater.

[grigatkis]

You want at least two active full backups worth of storage, with some spare for incrementals.

Backups can and do occasionally get data corruption (always have health check enabled) and if you can avoid having to delete old backup chains before doing a new active full you'll be much happier.

[skyforger]

I assume that you are aware the Backing up to AWS S3 Scale Out Tier costs, for that kind of amount data the monthly (at leas first month?) costs would be over 12000$ am I right?