Discussions related to using object storage as a backup target.
Post Reply
FSW Ant
Influencer
Posts: 11
Liked: never
Joined: Jun 24, 2021 5:28 pm
Full Name: Anthony
Contact:

Cloud archival storage for revolving 90 day time window - Whats the most cost effective option?

Post by FSW Ant »

Hello! This is my first post. I'm looking for some advice to try to find the best solution to achieve our goals. We're a Veeam Enterprise licensed organization and we are looking to change our infrastructure and backup designs to accommodate our goals but I can't seem to figure out the most cost effective solution to getting us there. We know we're planning on using the Archive Tier to send older backups to the cloud with a retention policy, but everything seems to be much more expensive than it should be due to read/write costs and deletion policies. Here is what we are trying to do, and I'll include some diagrams to help explain a little better.

We want to have the most recent single weeks backup stored onsite at our on-premise datacenter, with all backups older than that transitioning to the cloud each week after the current weeks backup expires. We know we need local backup storage for 2 total weeks worth as the next weeks backup will need to complete before moving the prior week to the cloud. When the prior week is moved to the cloud, the oldest cloud stored backup would be deleted and this would continue to rotate for a 90 day (12 week) cycle changing each week. So when The backups run this week, last weeks backup would be moved to a cloud based storage, such as Glacier Deep archive or similar, and then "week 13" in the cloud would be "expired out and removed". The problem is, we don't want to keep 6 months worth, just 3 months. This is due to the size of our backups which we can estimate at about 80TB for the single weeks synthetic fulls of all our jobs added together. Retaining 6 months worth is not intelligent for us as anything beyond 90 days isn't considered viable/useful. The other thing is, we don't want to take "Monthly" backups because then you are stuck with having missing gaps between the months of the things that change on a daily/weekly basis. So that is why we want to expire out the backups and upload new ones weekly.

https://ibb.co/jHBf4wP
https://ibb.co/CB5LP6S
https://ibb.co/HNYqVBz

If someone can give some advice/recommendations on what the best way to achieve our goals would be, we would appreciate it. Thanks!
HannesK
Product Manager
Posts: 14287
Liked: 2877 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: Cloud archival storage for revolving 90 day time window - Whats the most cost effective option?

Post by HannesK »

Hello,
and welcome to the forums.

If you want to use archive tier, then you need to change to universal licensing or enterprise plus edition first. On the other hand: with only 12 weeks retention, archive tier is probably more expensive than capacity tier. That's expected and the reason why the default settings don't even allow archive tier with such low retentions.

If I look at your pictures, there seem to be many misunderstandings how Veeam stores data in object storage. I recommend to check out the sticky forum FAQ. It covers also archive tier post338749.html#p338749

The cost topic is also covered in the FAQ. Meaning to look at alternative object storage providers (Backblaze and Wasabi are popular from what I see on the forums).

Best regards,
Hannes
FSW Ant
Influencer
Posts: 11
Liked: never
Joined: Jun 24, 2021 5:28 pm
Full Name: Anthony
Contact:

Re: Cloud archival storage for revolving 90 day time window - Whats the most cost effective option?

Post by FSW Ant »

Thank you, Hannes. We have Enterprise Plus already so no worries there. The diagrams I made were more a quick effort to convey the general concept of what our goal is. I've previously looked into both Wasabi and Backblaze, but was trying to figure out a way to do it with the buckets that Veeam was sort of "designed for" (Amazon and Azure). Even considering using Capacity Tier it ends up being too expensive for us. The last time we looked into this, no matter what option we went with the cost was over $20,000 per year which is just not cost effective for us. At that price point it makes more sense for me to just buy a cheap SAN and put it in one of my remote sites to back up to. Granted, I understand all the trade off's that makes, and I really -want- to archive older backups to the cloud so I'm trying to find a solution that makes it more attractive than buying more on-premise hardware to do it..
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Cloud archival storage for revolving 90 day time window - Whats the most cost effective option?

Post by Gostev » 1 person likes this post

Veeam wasn't really "designed for Amazon or Azure". They are important partners, however it's in our DNA to always give customers a choice, so we spend much time working closely with the other significant vendors too. Because some customers want premium offerings while other are looking for "the most cost effective" option - and no single provider can deliver that.

Also, please don't think about those other cloud object storage providers as some second-class citizens, as for example Wasabi consumption by Veeam customers is comparable to hyperscalers despite us not featuring them directly in the UI.
FSW Ant
Influencer
Posts: 11
Liked: never
Joined: Jun 24, 2021 5:28 pm
Full Name: Anthony
Contact:

Re: Cloud archival storage for revolving 90 day time window - Whats the most cost effective option?

Post by FSW Ant »

Right, I was generally talking about the partners. I don't consider other cloud storage providers inferior as long as using them is still considered a fully supported configuration from Veeam's standpoint, which it sounds like it is.

That being said, I've done some more data analysis on our side to get a better idea of where were are at for storage usage, and backup size as well as rate of date change (incremental sizes). Here is where we currently are at, which appears to be smaller than I remembered since I was able to do some major cleanup in the past year.

Total live data: Approx. ~46TB (about ~30TB of this is backed up by a different method since that data is on a iSCSI connected drive and Veeam doesn't back that up).
Total size of Veeam Synthetic Fulls for 1 week (excluding the ~30TB mentioned above): ~16TB
Total Daily incremental size for the Veeam backups currently is ~1.2TB of daily changes for a total weekly incremental total of ~7TB.

So if I'm doing my math right, and this is the part that I would need help with to confirm. I plan to at some point try to find a way to get the ~30TB that isn't backed up into Veeam yet incorporated into it.

So by my math, I would have 46TB for the initial full, plus 90 days worth of "forever" incrementals at ~1.2-~1.5TB per day to equal somewhere between 154TB-181TB of total cloud storage needed for our 90 day retention goal. Does this sound accurate or am I calculating this incorrectly?

Thanks in advance

Ant
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Cloud archival storage for revolving 90 day time window - Whats the most cost effective option?

Post by Gostev »

Yeah you're perfectly correct.

Please only double check that your full backups do indeed take only 16TB with 46TB of live data, as 3x compression ratio is uncommon. The vast majority of our customers are seeing around 2x. But perhaps your production data is somewhat special simply compresses better.

Assuming all your data is correct, your object storage needs are:
16TB + 1.5TBx90days = 151TB
FSW Ant
Influencer
Posts: 11
Liked: never
Joined: Jun 24, 2021 5:28 pm
Full Name: Anthony
Contact:

Re: Cloud archival storage for revolving 90 day time window - Whats the most cost effective option?

Post by FSW Ant »

not exactly, the 46TB number is the total size of all our Veeam Synthetic fulls (16TB) + (approximately 30TB of data that Veeam can't backup because its on an iSCSI drive and when choosing to backup that server in Veeam, it simply doesn't see those drives). I perform manual backups of that 30TB of data since Veeam cannot do it. So the 46TB is the total size of a single point backup for the whole organization across the various backup solutions. The actual live data is larger and we are indeed getting compression. I should have clarified that when I mentioned total live data as that was meant to say total single point backup for whole org data.

so the backup size would definitely be 46TB + 1.5TBx90days=181TB. Likely a little less though since 1.5TB daily change is being generous for now. Either way though, that tells me what I need to know for planning. It looks like we could do Wasabi for approximately $1,100.00 per month, which is much more attractive than when we looked at cloud backup previously.

So should I get approval to continue down the Wasabi path, would I set that up as a capacity tier, archive tier, SOBR, etc?
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Cloud archival storage for revolving 90 day time window - Whats the most cost effective option?

Post by Gostev »

You would set it up as a Capacity Tier.

By they way, Wasabi will offer you something that no storage box in a remote site possibly can, no matter how much you pay for it: immutable backups sitting in a 3rd party data center. This protects your data not only against ransomware and hackers, but even against malicious insiders. This could be a big "selling point" for your management potentially, as previously this level of protection required a very expensive process of continuously shipping tapes to some Iron Mountain location.
FSW Ant
Influencer
Posts: 11
Liked: never
Joined: Jun 24, 2021 5:28 pm
Full Name: Anthony
Contact:

Re: Cloud archival storage for revolving 90 day time window - Whats the most cost effective option?

Post by FSW Ant »

I saw that when I was reviewing the release info for V11 actually. I think thats great, and is definitely a selling point to mention to the decision makers. Currently I run V10 and I don't know if we'll be able to go to V11 for a while though as we have about 8-10 Linux physical servers that are running one of the older RHEL 6 2.6.32 releases. Those servers aren't slated to be replaced just yet so we are in a bit of a holding pattern. We were able to get the agent to install when we did the V10 upgrade with our Linux admin manually installing it.

I also like the fact that the V11 release adds an actual Powershell Module vs just the snap-in that was used previously. I haven't checked into it but hopefully the new module allows PS commands to get backup job status info for agent based backups. I've been waiting for that for 3 versions now haha.
Mildur
Product Manager
Posts: 8549
Liked: 2223 times
Joined: May 13, 2017 4:51 pm
Full Name: Fabian K.
Location: Switzerland
Contact:

Re: Cloud archival storage for revolving 90 day time window - Whats the most cost effective option?

Post by Mildur » 1 person likes this post

V10 can also use the immutability Feature from wasabi. You don‘t need V11 for that. :)
Product Management Analyst @ Veeam Software
HannesK
Product Manager
Posts: 14287
Liked: 2877 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: Cloud archival storage for revolving 90 day time window - Whats the most cost effective option?

Post by HannesK »

approximately 30TB of data that Veeam can't backup because its on an iSCSI drive and when choosing to backup that server in Veeam, it simply doesn't see those drives
by the way: physical machines can be backed up with Veeam agents. I assume that you tried to back up that iSCSI drive with a VMware snapshot... which is impossible because it's not a VMDK. It has nothing do to with Veeam. It's "hypervisor based backup" vs. "agent-based backup on top of an operating system inside a machine".
FSW Ant
Influencer
Posts: 11
Liked: never
Joined: Jun 24, 2021 5:28 pm
Full Name: Anthony
Contact:

Re: Cloud archival storage for revolving 90 day time window - Whats the most cost effective option?

Post by FSW Ant »

Yes, we already back up several physical machines via agents. And yes, we found that the VMWare snapshot doesn't back up those iSCSI drives because VMWare is not aware that those drives exist. I've been reviewing options online and the suggestion I found was that even though that server is a VM on VMWare with iSCSI drives mapped inside Windows, we would be able to instead use the physical Veeam agent for that VM and it would then backup all volumes since its not trying to back up at the hypervisor level. So that may be the quick and simple way around that issue.

I definitely am curious though, does V11 bring the capability for powershell to gather job status info from agent based backups finally?
Mildur
Product Manager
Posts: 8549
Liked: 2223 times
Joined: May 13, 2017 4:51 pm
Full Name: Fabian K.
Location: Switzerland
Contact:

Re: Cloud archival storage for revolving 90 day time window - Whats the most cost effective option?

Post by Mildur »

I definitely am curious though, does V11 bring the capability for powershell to gather job status info from agent based backups finally?
This was already possible with V10, as far I know:

https://helpcenter.veeam.com/docs/backu ... ml?ver=110
Product Management Analyst @ Veeam Software
Post Reply

Who is online

Users browsing this forum: No registered users and 8 guests