AWS S3 Requests

[stuemek]

Hello,

we configured Veeam 10 to offload our backups immediately to S3 IA storage with immutability for 28 days, the retention policy is set to keep 28 days and one yearly backup. The initial backup with 2,1TB took about 5 days due to poor upload capacity. Our daily incremental is about 60GB each and once a week it will perform a synthetic full backup.

Yesterday I checked our AWS billing and it was at ~4,3 million requests for SIA storage. Today the SIA requests didn't change and the requests for standard S3 storage suddenly jumped to ~4,3 million.

Do you guys have any idea why that happened? The object storage is still configured to use SIA storage.

Thanks in advance!

[wishr]

Hi Stuemek,

I think it might be a bug on the AWS billing side if no changes to the object storage configuration have been made. I would suggest raising a ticket with AWS and you may do the same with our technical support in parallel, just in case.

Thanks

[stuemek]

Hi wishr,

if I remember correctly, I changed the retention policy and the immutable policy from 31d to 28d yesterday. Does that have an imapct on the requests? I wouldn't have thought so.

Another question: I am considering changing from weekly synthetic to forever incremental to save on storage. As I can see, there is a bug in the current version offloading the full backup every single day. If I set the backup schedule only from Monday-Saturday and set the synthetic backup to Sunday (where actually no backup happens), will it work as forever incremental without affecting the bug?

[wishr]

Stuemek,

When you have an S3 IA storage class chosen as a target B&R will not interact with normal S3 so there should not be any requests made to IA. It's hard to say right now what is causing this, especially without looking at the billing data so you may want to get in touch with our engineers so they could have a deeper look into the situation.

Regarding the second question, issue #7 only affects Copy Mode of Capacity Tier and not the Move Mode. What mode do you use?

By the way, If you configure the backup job the way you mentioned a synthetic full backup will be created on Sunday and the data from it will be transferred to capacity tier, according to its configuration. In this case, you will not be affected by the issue - thanks to the synthetic full backup you created. With no active periodic full backup or periodic synthetic full backup in the backup chain, the issue will persist.

Thanks

[stuemek]

I am using Copy Mode.

Are you sure a synthetic backup will be created on Sunday if I do not schedule a backup for that day in the Schedule Options but only in the Advanced Storage Options?

[wishr]

Yes, please refer to the documentation I linked above which states: "Veeam Backup & Replication automatically triggers a backup job session to create a synthetic full backup, even if a regular backup job session is not scheduled on this day. " This setting controls the schedule for synthetic full backups.

Thanks

[stuemek]

OK, thanks for the link.

How about if I select the option to create a monthly active full but only on the first Friday on March for example. Will that prevent the bug from occuring until the hotfix is released?

[wishr]

Any way you prefer. You just need to enable either Active full or Synthetic full in the job settings to workaround the issue while we are working on a direct solution.

Thanks

[stuemek]

Just an upate: It happened again yesterday. The requests in standard Tier went up another ~4 million! I opened a ticket on AWS and will do so on Veeam Support, too. This is getting really expensive...

[veremin]

For now I would put the capacity extent in maintenance mode to prevent further offload activity to it, open a ticket with our support team and let them find the root cause.

Kindly, post your ticket number here, once you have one.

Thanks!

[stuemek]

Already did that. Here's my Ticket ID: #04089724

[veremin]

We're about to publish the cumulative patch for this and several others top issues, so, you should be provided with it within your existing support ticket, as soon as it's released. Thanks!

[wishr]

Hi Stuemek,

As far as I see you have closed the ticket. I'd recommend reopening it to stay updated on the cumulative patch availability.

Thanks

[stuemek]

Hi,

we have moved to Wasabi Cloud Storage to get rid of the requests issue and save on costs in the future, that's why we closed the ticket. Thanks anyway!

[veremin]

That's another way to go Glad to hear you're all set now. Thanks!