Discussions related to using object storage as a backup target.
Post Reply
Asahi
Expert
Posts: 134
Liked: 7 times
Joined: Jun 03, 2016 5:44 am
Full Name: Iio Asahi
Location: Japan
Contact:

About Glacier's support at Capacity Tier

Post by Asahi »

Hi ,

Please tell me about future support for SOBR 's Capacity Tier.
Does Veeam have plans to support Glacier at Capacity Tier in the future?

Kind Regards,
Asahi,
Climb Inc.
HannesK
Product Manager
Posts: 14287
Liked: 2877 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: About Glacier's support at Capacity Tier

Post by HannesK »

Hello Asahi,
yes, there are plans. We postponed that from U4 because of API changes.

Best regards,
Hannes
Asahi
Expert
Posts: 134
Liked: 7 times
Joined: Jun 03, 2016 5:44 am
Full Name: Iio Asahi
Location: Japan
Contact:

Re: About Glacier's support at Capacity Tier

Post by Asahi »

Hi Hannes,

Thank you for reply.

OK, I understood that there is a support plan for Glacier.
When will it be supported?

Kind Regards,
Asahi,
Climb Inc.
HannesK
Product Manager
Posts: 14287
Liked: 2877 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: About Glacier's support at Capacity Tier

Post by HannesK »

Hi Asahi,
we can't speak about timelines right now, it is too early.

Thanks for your understanding,
Hannes
Asahi
Expert
Posts: 134
Liked: 7 times
Joined: Jun 03, 2016 5:44 am
Full Name: Iio Asahi
Location: Japan
Contact:

Re: About Glacier's support at Capacity Tier

Post by Asahi »

Hi Hannes,

OK, I look forward to future updates.

Kind Regards,
Asahi,
Climb Inc.
gregverbarendse
Novice
Posts: 3
Liked: never
Joined: Jun 22, 2018 3:46 pm
Full Name: Greg Verbarendse
Contact:

Is AWS S3 Glacier compatible with Capacity Tier

Post by gregverbarendse »

I've searched the forum and haven't seen this exact question asked and answered yet so apologize if this is already out there.

I'm relatively new to Veeam, and very new to AWS, and I'm trying to understand if AWS S3 Glacier is compatible with the Capacity Tier or if it has to be S3 Standard? Is there any difference between S3 Glacier and regular Glacier? It looks like what used to be just Glacier in AWS is now S3 Glacier but not sure if that changes the compatibility with Veeam.

If S3 Glacier is not compatible with the Capacity Tier can I use the VTL option to go straight to Glacier or does that also need to hit the S3 Standard bucket first?

Thanks in advance for any feedback.

Greg
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Is AWS S3 Glacier compatible with Capacity Tier

Post by veremin » 1 person likes this post

No, it is not, as it uses rather different architecture and thus requires different approach to support it. Also, the use case behind Glacier is a different - "write once read never", which current Capacity Tier has not been designed for.

As to using it in VTL mode, you can use this option to go straight to Glacier (backup or file to tape jobs), however, keep data retrieval and removal prices in mind (those might hit you hard). Kindly, check these community forums and VTL approach has been discussed previously.

Thanks!
sfirmes
Veeam Software
Posts: 225
Liked: 117 times
Joined: Jul 24, 2018 8:38 pm
Full Name: Stephen Firmes
Contact:

Re: Is AWS S3 Glacier compatible with Capacity Tier

Post by sfirmes »

S3 Glacier isn't a target for the capacity tier.

This is where you can find what is supported and the current list is copied:
https://helpcenter.veeam.com/docs/backu ... ml?ver=100
S3-Compatible devices
Amazon S3
Microsoft Azure Blob Storage
Microsoft Azure Data Box devices
IBM Cloud Object Storage
Senior Solutions Architect, Product Management - Alliances @ Veeam Software
ober72
Veeam Vanguard
Posts: 698
Liked: 136 times
Joined: Jan 24, 2014 4:10 pm
Full Name: Geoff Burke
Contact:

Re: Is AWS S3 Glacier compatible with Capacity Tier

Post by ober72 »

Just a quick note for AWS VTL, make sure you export your tapes once done. When they are loaded in Veeam AWS treats them as being in S3 storage $$$$$$$

On the AWS console after you export from the Veeam console you will see that the tape has been archived to Glacier or Deep Glacier depending on what you chose.

cheers
Geoff Burke
VMCA2022, VMCE2023, CKA, CKAD
Veeam Vanguard, Veeam Legend
dalbertson
Veeam Software
Posts: 492
Liked: 175 times
Joined: Jul 21, 2015 12:38 pm
Full Name: Dustin Albertson
Contact:

Re: Is AWS S3 Glacier compatible with Capacity Tier

Post by dalbertson » 3 people like this post

@ober72 AWS only charges for S3 if you have moved data onto the tape. If the tape is just loaded into the system there are no charges.

@gregverbarendse Here is a link to a guide i wrote on deploying the gateway https://www.veeam.com/wp-using-aws-vtl- ... guide.html
Dustin Albertson | Director of Product Management - Cloud & Applications | Veeam Product Management, Alliances
ober72
Veeam Vanguard
Posts: 698
Liked: 136 times
Joined: Jan 24, 2014 4:10 pm
Full Name: Geoff Burke
Contact:

Re: Is AWS S3 Glacier compatible with Capacity Tier

Post by ober72 »

Hi Dustin,

Yes sorry that is a very good point :). Sorry for the confusion.

cheers
Geoff Burke
VMCA2022, VMCE2023, CKA, CKAD
Veeam Vanguard, Veeam Legend
gregverbarendse
Novice
Posts: 3
Liked: never
Joined: Jun 22, 2018 3:46 pm
Full Name: Greg Verbarendse
Contact:

Re: Is AWS S3 Glacier compatible with Capacity Tier

Post by gregverbarendse »

Thank you all for the comments, they were helpful. I do have a follow up question though. S3 storage has an individual file size limit of 5 TB, what happens with backup files that are larger than 5TB destined for a SOBR configured to copy or move to an S3 Capacity Tier? I have a few large file servers where the full backups are larger than 5 TB.

@dalbertson Thanks for the link, I did find that guide this morning so I will be doing some reading later.
dalbertson
Veeam Software
Posts: 492
Liked: 175 times
Joined: Jul 21, 2015 12:38 pm
Full Name: Dustin Albertson
Contact:

Re: Is AWS S3 Glacier compatible with Capacity Tier

Post by dalbertson » 1 person likes this post

The VTL gateway doesn’t support tapes over 2.5tb. So if a job is over that then it’s spread over multiple tapes automatically.

As far as cloud tier...here is another paper i wrote ;) https://www.veeam.com/wp-cloud-tier-deep-dive.html

Configuration settings chosen for the backup job using SOBR with Capacity Tier as the Target define the type of backup chain as it gets stored on Scale-Out Backup Repository and further processed by Capacity Tier’s Offload job.

In addition to the type of backup chain configured, selected configuration of Storage Optimization and Compression settings define block size for the backup data, which in turn, defines the number of blocks consumed on object storage as the backup chains are offloaded to the Capacity Tier.

Local Target (16 TB + backup files): 4096 KB data blocks
Local Target: 1024 KB data blocks
LAN Target: 512 KB data blocks
WAN Target: 256 KB data blocks

The smaller the Block Size the more calls we make to the Object Storage to upload the data which will impact costing.
Dustin Albertson | Director of Product Management - Cloud & Applications | Veeam Product Management, Alliances
sfirmes
Veeam Software
Posts: 225
Liked: 117 times
Joined: Jul 24, 2018 8:38 pm
Full Name: Stephen Firmes
Contact:

Re: Is AWS S3 Glacier compatible with Capacity Tier

Post by sfirmes »

We write to the object storage using the block size you defined for for the backup job. Our default and best practice is 1mb blocks (local target). Our maximum block size is 4mb which is below their limit.
Senior Solutions Architect, Product Management - Alliances @ Veeam Software
ober72
Veeam Vanguard
Posts: 698
Liked: 136 times
Joined: Jan 24, 2014 4:10 pm
Full Name: Geoff Burke
Contact:

Re: Is AWS S3 Glacier compatible with Capacity Tier

Post by ober72 »

dalbertson wrote: Mar 05, 2020 10:11 pm The VTL gateway doesn’t support tapes over 2.5tb. So if a job is over that then it’s spread over multiple tapes automatically.

As far as cloud tier...here is another paper i wrote ;) https://www.veeam.com/wp-cloud-tier-deep-dive.html

Configuration settings chosen for the backup job using SOBR with Capacity Tier as the Target define the type of backup chain as it gets stored on Scale-Out Backup Repository and further processed by Capacity Tier’s Offload job.

In addition to the type of backup chain configured, selected configuration of Storage Optimization and Compression settings define block size for the backup data, which in turn, defines the number of blocks consumed on object storage as the backup chains are offloaded to the Capacity Tier.

Local Target (16 TB + backup files): 4096 KB data blocks
Local Target: 1024 KB data blocks
LAN Target: 512 KB data blocks
WAN Target: 256 KB data blocks

The smaller the Block Size the more calls we make to the Object Storage to upload the data which will impact costing.
Hi Dustin,

I created 5TB tapes and it worked but are you saying that I should redo with 2.5TB tapes because it is a veeam restriction?

Thanks
Geoff Burke
VMCA2022, VMCE2023, CKA, CKAD
Veeam Vanguard, Veeam Legend
ober72
Veeam Vanguard
Posts: 698
Liked: 136 times
Joined: Jan 24, 2014 4:10 pm
Full Name: Geoff Burke
Contact:

Re: Is AWS S3 Glacier compatible with Capacity Tier

Post by ober72 »

AWS say min is 100gb and max 5TB but again I might have missed something in the veeam docs

https://aws.amazon.com/storagegateway/faqs/

Q: How much data can I store on a virtual tape?

A: The minimum size and maximum size of a virtual tape you can create on a tape gateway is 100 GiB and 5 TiB respectively. Please note, you only pay for the amount of data stored on each tape, and not for the size of the tape.
Geoff Burke
VMCA2022, VMCE2023, CKA, CKAD
Veeam Vanguard, Veeam Legend
dalbertson
Veeam Software
Posts: 492
Liked: 175 times
Joined: Jul 21, 2015 12:38 pm
Full Name: Dustin Albertson
Contact:

Re: Is AWS S3 Glacier compatible with Capacity Tier

Post by dalbertson »

@ober72 No...i forgot that recently AWS raised the VTL tape size from 2.5 to 5tb. This was not a Veeam limit as it was just the max size of a tape you could create in AWS.....you are good the way you are.

Start writing tape jobs and enjoy.....btw are you using glacier pools or deep archive pools? just curious.
Dustin Albertson | Director of Product Management - Cloud & Applications | Veeam Product Management, Alliances
ober72
Veeam Vanguard
Posts: 698
Liked: 136 times
Joined: Jan 24, 2014 4:10 pm
Full Name: Geoff Burke
Contact:

Re: Is AWS S3 Glacier compatible with Capacity Tier

Post by ober72 »

Hi Dustin,

We have our in house tape libraries but have the AWS for extras and some extra redundancy so we are using both. Again it depends on RTO :) as always. I think deep glacier is 12 days retrieval so we keep that in mind.
As an aside I really like the whole VTL techology. I setup an opensource https://www.quadstor.com/virtual-tape-library.html one that can emulated our $$$$ production one and went nuts experimenting so I felt more confident when doing the real deal.

cheers
Geoff Burke
VMCA2022, VMCE2023, CKA, CKAD
Veeam Vanguard, Veeam Legend
dalbertson
Veeam Software
Posts: 492
Liked: 175 times
Joined: Jul 21, 2015 12:38 pm
Full Name: Dustin Albertson
Contact:

Re: Is AWS S3 Glacier compatible with Capacity Tier

Post by dalbertson »

That is cool Geoff. I had done some testing with that product a while back as well and thought it was nice. By this point i think i have tested about every VTL device out there. Not sure how i became the VTL guy but i partly blame @tsightler

How do you like it so far?
Dustin Albertson | Director of Product Management - Cloud & Applications | Veeam Product Management, Alliances
ober72
Veeam Vanguard
Posts: 698
Liked: 136 times
Joined: Jan 24, 2014 4:10 pm
Full Name: Geoff Burke
Contact:

Re: Is AWS S3 Glacier compatible with Capacity Tier

Post by ober72 »

Works really well. The whole tape medial pool organization is flexible and convenient. It is much easier than dealing with physical tapes and then having to organized to be sent off to secure offsite locations etc.

cheers
Geoff Burke
VMCA2022, VMCE2023, CKA, CKAD
Veeam Vanguard, Veeam Legend
tsightler
VP, Product Management
Posts: 6009
Liked: 2843 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Is AWS S3 Glacier compatible with Capacity Tier

Post by tsightler » 2 people like this post

dalbertson wrote: Mar 05, 2020 11:35 pm That is cool Geoff. I had done some testing with that product a while back as well and thought it was nice. By this point i think i have tested about every VTL device out there. Not sure how i became the VTL guy but i partly blame @tsightler
That's OK @dalbertson I'll take the blame as long as you have to do the work. :D

I think the VTL is a great way to leverage deep archive solutions. As @ober72 notes most of the complexity of tape was in the management of the physical media itself which VTL completely eliminates. I'm glad you're finding this approach useful.
gregverbarendse
Novice
Posts: 3
Liked: never
Joined: Jun 22, 2018 3:46 pm
Full Name: Greg Verbarendse
Contact:

Re: Is AWS S3 Glacier compatible with Capacity Tier

Post by gregverbarendse »

dalbertson wrote: Mar 05, 2020 10:11 pm The VTL gateway doesn’t support tapes over 2.5tb. So if a job is over that then it’s spread over multiple tapes automatically.

As far as cloud tier...here is another paper i wrote ;) https://www.veeam.com/wp-cloud-tier-deep-dive.html

Configuration settings chosen for the backup job using SOBR with Capacity Tier as the Target define the type of backup chain as it gets stored on Scale-Out Backup Repository and further processed by Capacity Tier’s Offload job.

In addition to the type of backup chain configured, selected configuration of Storage Optimization and Compression settings define block size for the backup data, which in turn, defines the number of blocks consumed on object storage as the backup chains are offloaded to the Capacity Tier.

Local Target (16 TB + backup files): 4096 KB data blocks
Local Target: 1024 KB data blocks
LAN Target: 512 KB data blocks
WAN Target: 256 KB data blocks

The smaller the Block Size the more calls we make to the Object Storage to upload the data which will impact costing.
Okay, so the actual size of the .vbk files isn't an issue then because data is stored at the block level in the Cloud Tier? Sorry for the newb questions!

Thanks,
Greg
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Is AWS S3 Glacier compatible with Capacity Tier

Post by Gostev » 1 person likes this post

Yes, as we offload data to object storage in the forever-incremental manner (often, this approach is also called "source-side deduplication"). Thanks!
Asahi
Expert
Posts: 134
Liked: 7 times
Joined: Jun 03, 2016 5:44 am
Full Name: Iio Asahi
Location: Japan
Contact:

[MERGED] About Glacier's support at Capacity Tier

Post by Asahi »

Hello Hannes and VeeamTeam,

How far is Amazon Glacier's support plan currently?

Kind Regards,
Asahi,
Climb inc.
HannesK
Product Manager
Posts: 14287
Liked: 2877 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: About Glacier's support at Capacity Tier

Post by HannesK »

Hello,
we are working on it, but it is still to early to talk about a timeline.

Just because I'm interested: what retention time to you plan for backups in Glacier?

Best regards,
Hannes
Asahi
Expert
Posts: 134
Liked: 7 times
Joined: Jun 03, 2016 5:44 am
Full Name: Iio Asahi
Location: Japan
Contact:

Re: Is AWS S3 Glacier compatible with Capacity Tier

Post by Asahi »

Hello Hannes,

Thank you for information.
I was relieved to hear that we are still working on Glacier support.

I would like to keep the data for about 2 years.

Kind Regards,
Asahi,
Climb Inc.
klin8251
Influencer
Posts: 12
Liked: 3 times
Joined: Mar 05, 2010 7:46 pm
Full Name: Peter Klingler
Contact:

Re: Is AWS S3 Glacier compatible with Capacity Tier

Post by klin8251 »

Will glacier support be part of a SOBR? Do you think it will be added in the next major release? What are the challenges of adding Glacier support?
dalbertson
Veeam Software
Posts: 492
Liked: 175 times
Joined: Jul 21, 2015 12:38 pm
Full Name: Dustin Albertson
Contact:

Re: Is AWS S3 Glacier compatible with Capacity Tier

Post by dalbertson »

It is not currently supported and there is no ETA that i can give as to when support will be there. But it is something we are investigating as mentioned above.

Can you please define your use case and how you would like to use it.
Dustin Albertson | Director of Product Management - Cloud & Applications | Veeam Product Management, Alliances
HannesK
Product Manager
Posts: 14287
Liked: 2877 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: Is AWS S3 Glacier compatible with Capacity Tier

Post by HannesK »

What are the challenges of adding Glacier support?
the challenge is that it requires a massive change of the current "optimized for object storage" approach.

The small objects we use for S3 are great for granular restore and to optimize for space usage. Even instant VM recovery from S3 is technically possible. :-)

If we would just throw all these small objects to glacier, it would cost customers a lot of money on PUT costs. We want to deliver a product that does not have such massive side effects... also please keep in mind, that Glacier can only save money with long term retention (90 / 180 days for glacier / deep archive). Anything short term would cost a fortune in Glacier for the early access fees
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Is AWS S3 Glacier compatible with Capacity Tier

Post by Gostev »

This made me remember how someone got burned by Glacier yesterday ;)
Post Reply

Who is online

Users browsing this forum: No registered users and 7 guests