Discussions related to using object storage as a backup target.
Post Reply
stuemek
Novice
Posts: 7
Liked: 3 times
Joined: Jul 29, 2019 7:01 am
Contact:

AWS S3 Requests

Post by stuemek »

Hello,

we configured Veeam 10 to offload our backups immediately to S3 IA storage with immutability for 28 days, the retention policy is set to keep 28 days and one yearly backup. The initial backup with 2,1TB took about 5 days due to poor upload capacity. Our daily incremental is about 60GB each and once a week it will perform a synthetic full backup.

Yesterday I checked our AWS billing and it was at ~4,3 million requests for SIA storage. Today the SIA requests didn't change and the requests for standard S3 storage suddenly jumped to ~4,3 million.

Do you guys have any idea why that happened? The object storage is still configured to use SIA storage.

Thanks in advance!
wishr
Veteran
Posts: 3077
Liked: 453 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: AWS S3 Requests

Post by wishr »

Hi Stuemek,

I think it might be a bug on the AWS billing side if no changes to the object storage configuration have been made. I would suggest raising a ticket with AWS and you may do the same with our technical support in parallel, just in case.

Thanks
stuemek
Novice
Posts: 7
Liked: 3 times
Joined: Jul 29, 2019 7:01 am
Contact:

Re: AWS S3 Requests

Post by stuemek »

Hi wishr,

if I remember correctly, I changed the retention policy and the immutable policy from 31d to 28d yesterday. Does that have an imapct on the requests? I wouldn't have thought so.

Another question: I am considering changing from weekly synthetic to forever incremental to save on storage. As I can see, there is a bug in the current version offloading the full backup every single day. If I set the backup schedule only from Monday-Saturday and set the synthetic backup to Sunday (where actually no backup happens), will it work as forever incremental without affecting the bug?
wishr
Veteran
Posts: 3077
Liked: 453 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: AWS S3 Requests

Post by wishr »

Stuemek,

When you have an S3 IA storage class chosen as a target B&R will not interact with normal S3 so there should not be any requests made to IA. It's hard to say right now what is causing this, especially without looking at the billing data so you may want to get in touch with our engineers so they could have a deeper look into the situation.

Regarding the second question, issue #7 only affects Copy Mode of Capacity Tier and not the Move Mode. What mode do you use?

By the way, If you configure the backup job the way you mentioned a synthetic full backup will be created on Sunday and the data from it will be transferred to capacity tier, according to its configuration. In this case, you will not be affected by the issue - thanks to the synthetic full backup you created. With no active periodic full backup or periodic synthetic full backup in the backup chain, the issue will persist.

Thanks
stuemek
Novice
Posts: 7
Liked: 3 times
Joined: Jul 29, 2019 7:01 am
Contact:

Re: AWS S3 Requests

Post by stuemek »

I am using Copy Mode.

Are you sure a synthetic backup will be created on Sunday if I do not schedule a backup for that day in the Schedule Options but only in the Advanced Storage Options?
wishr
Veteran
Posts: 3077
Liked: 453 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: AWS S3 Requests

Post by wishr » 1 person likes this post

Yes, please refer to the documentation I linked above which states: "Veeam Backup & Replication automatically triggers a backup job session to create a synthetic full backup, even if a regular backup job session is not scheduled on this day. " This setting controls the schedule for synthetic full backups.

Thanks
stuemek
Novice
Posts: 7
Liked: 3 times
Joined: Jul 29, 2019 7:01 am
Contact:

Re: AWS S3 Requests

Post by stuemek »

OK, thanks for the link.

How about if I select the option to create a monthly active full but only on the first Friday on March for example. Will that prevent the bug from occuring until the hotfix is released?
wishr
Veteran
Posts: 3077
Liked: 453 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: AWS S3 Requests

Post by wishr »

Any way you prefer. You just need to enable either Active full or Synthetic full in the job settings to workaround the issue while we are working on a direct solution.

Thanks
stuemek
Novice
Posts: 7
Liked: 3 times
Joined: Jul 29, 2019 7:01 am
Contact:

Re: AWS S3 Requests

Post by stuemek »

Just an upate: It happened again yesterday. The requests in standard Tier went up another ~4 million! I opened a ticket on AWS and will do so on Veeam Support, too. This is getting really expensive...
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: AWS S3 Requests

Post by veremin »

For now I would put the capacity extent in maintenance mode to prevent further offload activity to it, open a ticket with our support team and let them find the root cause.

Kindly, post your ticket number here, once you have one.

Thanks!
stuemek
Novice
Posts: 7
Liked: 3 times
Joined: Jul 29, 2019 7:01 am
Contact:

Re: AWS S3 Requests

Post by stuemek » 1 person likes this post

Already did that. Here's my Ticket ID: #04089724
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: AWS S3 Requests

Post by veremin »

We're about to publish the cumulative patch for this and several others top issues, so, you should be provided with it within your existing support ticket, as soon as it's released. Thanks!
wishr
Veteran
Posts: 3077
Liked: 453 times
Joined: Aug 07, 2018 3:11 pm
Full Name: Fedor Maslov
Contact:

Re: AWS S3 Requests

Post by wishr »

Hi Stuemek,

As far as I see you have closed the ticket. I'd recommend reopening it to stay updated on the cumulative patch availability.

Thanks
stuemek
Novice
Posts: 7
Liked: 3 times
Joined: Jul 29, 2019 7:01 am
Contact:

Re: AWS S3 Requests

Post by stuemek » 2 people like this post

Hi,

we have moved to Wasabi Cloud Storage to get rid of the requests issue and save on costs in the future, that's why we closed the ticket. Thanks anyway!
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: AWS S3 Requests

Post by veremin »

That's another way to go :) Glad to hear you're all set now. Thanks!
Post Reply

Who is online

Users browsing this forum: Stupots and 14 guests