Is AWS S3 Glacier compatible with Capacity Tier

[Asahi]

Hi ,

Please tell me about future support for SOBR 's Capacity Tier.
Does Veeam have plans to support Glacier at Capacity Tier in the future?

Kind Regards,
Asahi,
Climb Inc.

[HannesK]

Hello Asahi,
yes, there are plans. We postponed that from U4 because of API changes.

Best regards,
Hannes

[Asahi]

Hi Hannes,

Thank you for reply.

OK, I understood that there is a support plan for Glacier.
When will it be supported?

Kind Regards,
Asahi,
Climb Inc.

[HannesK]

Hi Asahi,
we can't speak about timelines right now, it is too early.

Thanks for your understanding,
Hannes

[Asahi]

Hi Hannes,

OK, I look forward to future updates.

Kind Regards,
Asahi,
Climb Inc.

[gregverbarendse]

I've searched the forum and haven't seen this exact question asked and answered yet so apologize if this is already out there.

I'm relatively new to Veeam, and very new to AWS, and I'm trying to understand if AWS S3 Glacier is compatible with the Capacity Tier or if it has to be S3 Standard? Is there any difference between S3 Glacier and regular Glacier? It looks like what used to be just Glacier in AWS is now S3 Glacier but not sure if that changes the compatibility with Veeam.

If S3 Glacier is not compatible with the Capacity Tier can I use the VTL option to go straight to Glacier or does that also need to hit the S3 Standard bucket first?

Thanks in advance for any feedback.

Greg

[veremin]

No, it is not, as it uses rather different architecture and thus requires different approach to support it. Also, the use case behind Glacier is a different - "write once read never", which current Capacity Tier has not been designed for.

As to using it in VTL mode, you can use this option to go straight to Glacier (backup or file to tape jobs), however, keep data retrieval and removal prices in mind (those might hit you hard). Kindly, check these community forums and VTL approach has been discussed previously.

Thanks!

[sfirmes]

S3 Glacier isn't a target for the capacity tier.

This is where you can find what is supported and the current list is copied:
https://helpcenter.veeam.com/docs/backu ... ml?ver=100
S3-Compatible devices
Amazon S3
Microsoft Azure Blob Storage
Microsoft Azure Data Box devices
IBM Cloud Object Storage

[ober72]

Just a quick note for AWS VTL, make sure you export your tapes once done. When they are loaded in Veeam AWS treats them as being in S3 storage $$$$$$$

On the AWS console after you export from the Veeam console you will see that the tape has been archived to Glacier or Deep Glacier depending on what you chose.

cheers

[dalbertson]

@ober72 AWS only charges for S3 if you have moved data onto the tape. If the tape is just loaded into the system there are no charges.

@gregverbarendse Here is a link to a guide i wrote on deploying the gateway https://www.veeam.com/wp-using-aws-vtl- ... guide.html

[ober72]

Hi Dustin,

Yes sorry that is a very good point . Sorry for the confusion.

cheers

[gregverbarendse]

Thank you all for the comments, they were helpful. I do have a follow up question though. S3 storage has an individual file size limit of 5 TB, what happens with backup files that are larger than 5TB destined for a SOBR configured to copy or move to an S3 Capacity Tier? I have a few large file servers where the full backups are larger than 5 TB.

@dalbertson Thanks for the link, I did find that guide this morning so I will be doing some reading later.

[dalbertson]

The VTL gateway doesn’t support tapes over 2.5tb. So if a job is over that then it’s spread over multiple tapes automatically.

As far as cloud tier...here is another paper i wrote https://www.veeam.com/wp-cloud-tier-deep-dive.html

Configuration settings chosen for the backup job using SOBR with Capacity Tier as the Target define the type of backup chain as it gets stored on Scale-Out Backup Repository and further processed by Capacity Tier’s Offload job.

In addition to the type of backup chain configured, selected configuration of Storage Optimization and Compression settings define block size for the backup data, which in turn, defines the number of blocks consumed on object storage as the backup chains are offloaded to the Capacity Tier.

Local Target (16 TB + backup files): 4096 KB data blocks
Local Target: 1024 KB data blocks
LAN Target: 512 KB data blocks
WAN Target: 256 KB data blocks

The smaller the Block Size the more calls we make to the Object Storage to upload the data which will impact costing.

[sfirmes]

We write to the object storage using the block size you defined for for the backup job. Our default and best practice is 1mb blocks (local target). Our maximum block size is 4mb which is below their limit.

[ober72]

The VTL gateway doesn’t support tapes over 2.5tb. So if a job is over that then it’s spread over multiple tapes automatically.

As far as cloud tier...here is another paper i wrote https://www.veeam.com/wp-cloud-tier-deep-dive.html

Configuration settings chosen for the backup job using SOBR with Capacity Tier as the Target define the type of backup chain as it gets stored on Scale-Out Backup Repository and further processed by Capacity Tier’s Offload job.

In addition to the type of backup chain configured, selected configuration of Storage Optimization and Compression settings define block size for the backup data, which in turn, defines the number of blocks consumed on object storage as the backup chains are offloaded to the Capacity Tier.

Local Target (16 TB + backup files): 4096 KB data blocks
Local Target: 1024 KB data blocks
LAN Target: 512 KB data blocks
WAN Target: 256 KB data blocks

The smaller the Block Size the more calls we make to the Object Storage to upload the data which will impact costing.

Hi Dustin,

I created 5TB tapes and it worked but are you saying that I should redo with 2.5TB tapes because it is a veeam restriction?

Thanks

[ober72]

AWS say min is 100gb and max 5TB but again I might have missed something in the veeam docs

https://aws.amazon.com/storagegateway/faqs/

Q: How much data can I store on a virtual tape?

A: The minimum size and maximum size of a virtual tape you can create on a tape gateway is 100 GiB and 5 TiB respectively. Please note, you only pay for the amount of data stored on each tape, and not for the size of the tape.

[dalbertson]

@ober72 No...i forgot that recently AWS raised the VTL tape size from 2.5 to 5tb. This was not a Veeam limit as it was just the max size of a tape you could create in AWS.....you are good the way you are.

Start writing tape jobs and enjoy.....btw are you using glacier pools or deep archive pools? just curious.

[ober72]

Hi Dustin,

We have our in house tape libraries but have the AWS for extras and some extra redundancy so we are using both. Again it depends on RTO as always. I think deep glacier is 12 days retrieval so we keep that in mind.
As an aside I really like the whole VTL techology. I setup an opensource https://www.quadstor.com/virtual-tape-library.html one that can emulated our $$$$ production one and went nuts experimenting so I felt more confident when doing the real deal.

cheers

[dalbertson]

That is cool Geoff. I had done some testing with that product a while back as well and thought it was nice. By this point i think i have tested about every VTL device out there. Not sure how i became the VTL guy but i partly blame @tsightler

How do you like it so far?

[ober72]

Works really well. The whole tape medial pool organization is flexible and convenient. It is much easier than dealing with physical tapes and then having to organized to be sent off to secure offsite locations etc.

cheers

[tsightler]

That is cool Geoff. I had done some testing with that product a while back as well and thought it was nice. By this point i think i have tested about every VTL device out there. Not sure how i became the VTL guy but i partly blame @tsightler

That's OK @dalbertson I'll take the blame as long as you have to do the work.

I think the VTL is a great way to leverage deep archive solutions. As @ober72 notes most of the complexity of tape was in the management of the physical media itself which VTL completely eliminates. I'm glad you're finding this approach useful.

[gregverbarendse]

The VTL gateway doesn’t support tapes over 2.5tb. So if a job is over that then it’s spread over multiple tapes automatically.

As far as cloud tier...here is another paper i wrote https://www.veeam.com/wp-cloud-tier-deep-dive.html

Configuration settings chosen for the backup job using SOBR with Capacity Tier as the Target define the type of backup chain as it gets stored on Scale-Out Backup Repository and further processed by Capacity Tier’s Offload job.

In addition to the type of backup chain configured, selected configuration of Storage Optimization and Compression settings define block size for the backup data, which in turn, defines the number of blocks consumed on object storage as the backup chains are offloaded to the Capacity Tier.

Local Target (16 TB + backup files): 4096 KB data blocks
Local Target: 1024 KB data blocks
LAN Target: 512 KB data blocks
WAN Target: 256 KB data blocks

The smaller the Block Size the more calls we make to the Object Storage to upload the data which will impact costing.

Okay, so the actual size of the .vbk files isn't an issue then because data is stored at the block level in the Cloud Tier? Sorry for the newb questions!

Thanks,
Greg

[Gostev]

Yes, as we offload data to object storage in the forever-incremental manner (often, this approach is also called "source-side deduplication"). Thanks!

[Asahi]

Hello Hannes and VeeamTeam,

How far is Amazon Glacier's support plan currently?

Kind Regards,
Asahi,
Climb inc.

[HannesK]

Hello,
we are working on it, but it is still to early to talk about a timeline.

Just because I'm interested: what retention time to you plan for backups in Glacier?

Best regards,
Hannes

[Asahi]

Hello Hannes,

Thank you for information.
I was relieved to hear that we are still working on Glacier support.

I would like to keep the data for about 2 years.

Kind Regards,
Asahi,
Climb Inc.

[klin8251]

Will glacier support be part of a SOBR? Do you think it will be added in the next major release? What are the challenges of adding Glacier support?

[dalbertson]

It is not currently supported and there is no ETA that i can give as to when support will be there. But it is something we are investigating as mentioned above.

Can you please define your use case and how you would like to use it.

[HannesK]

What are the challenges of adding Glacier support?

the challenge is that it requires a massive change of the current "optimized for object storage" approach.

The small objects we use for S3 are great for granular restore and to optimize for space usage. Even instant VM recovery from S3 is technically possible.

If we would just throw all these small objects to glacier, it would cost customers a lot of money on PUT costs. We want to deliver a product that does not have such massive side effects... also please keep in mind, that Glacier can only save money with long term retention (90 / 180 days for glacier / deep archive). Anything short term would cost a fortune in Glacier for the early access fees

[Gostev]

This made me remember how someone got burned by Glacier yesterday