Discussions related to using object storage as a backup target.
Post Reply
tyler.jurgens
Veeam Legend
Posts: 418
Liked: 244 times
Joined: Apr 11, 2023 1:18 pm
Full Name: Tyler Jurgens
Contact:

S3-integrated Storage - Block Size

Post by tyler.jurgens »

With S3-integrated storage (Eg: minio) the storage tells Veeam what block size to use (in this case, Minio recommends a 4 MB block).

However, when I check the job settings on a new backup job, using the defaults, the block size is set to 1 MB.

Which block size will be used on the backup job? The size the Veeam job reflects, or the size the storage system recommends?

Assuming the block size used is the block size the storage system recommends and the settings in the veeam job are ignored: If I wanted to override the block size, how is that done?
Tyler Jurgens
Veeam Legend x3 | vExpert ** | VMCE | VCP 2020 | Tanzu Vanguard | VUG Canada Leader | VMUG Calgary Leader
Blog: https://explosive.cloud
Twitter: @Tyler_Jurgens BlueSky: @explosive.cloud
sfirmes
Veeam Software
Posts: 312
Liked: 147 times
Joined: Jul 24, 2018 8:38 pm
Full Name: Stephen Firmes
Contact:

Re: S3-integrated Storage - Block Size

Post by sfirmes »

@tjurgens-s2d The block size that you specify in the backup job is what is used when putting objects to object storage. The object storage vendor may have their best practice size recommendations and if you choose to use them that is where you would select the block size to be used.
Steve Firmes | Senior Solutions Architect, Product Management - Alliances @ Veeam Software
tyler.jurgens
Veeam Legend
Posts: 418
Liked: 244 times
Joined: Apr 11, 2023 1:18 pm
Full Name: Tyler Jurgens
Contact:

Re: S3-integrated Storage - Block Size

Post by tyler.jurgens »

On Luca's blog post (https://www.virtualtothecore.com/a-firs ... am-sosapi/) it looks like it might be planned capability to force a default block size. I'm guessing that's not in place yet?
Tyler Jurgens
Veeam Legend x3 | vExpert ** | VMCE | VCP 2020 | Tanzu Vanguard | VUG Canada Leader | VMUG Calgary Leader
Blog: https://explosive.cloud
Twitter: @Tyler_Jurgens BlueSky: @explosive.cloud
Gostev
Chief Product Officer
Posts: 31977
Liked: 7441 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: S3-integrated Storage - Block Size

Post by Gostev »

No, this has not been implemented yet. Also, some people at Veeam argue if it's even a good idea to let storage vendors control this :)

For example, changing the default 1MB block to 4MB will approximately double your disk space consumption in the long run. This is obviously great for a storage vendor as they get to sell 2x more nodes, but not so great for your wallet or data center footprint.
tyler.jurgens
Veeam Legend
Posts: 418
Liked: 244 times
Joined: Apr 11, 2023 1:18 pm
Full Name: Tyler Jurgens
Contact:

Re: S3-integrated Storage - Block Size

Post by tyler.jurgens »

As with anything, let it be configurable. If the storage vendor's SOSAPI implementation forces 4 MB blocks, let the Veeam client override it. Eg: If I create a job against a S3-integrated bucket, the block size defaults to the SOSAPI block size. If I change it, it uses the block size I set.

For our Minio implementation, we'd take 4 MB blocks over hundreds of millions of tiny blocks negatively impacting our performance. We have one client backing up 45 TB using 223 million objects! It has made my leadership team want to avoid Veeam, while I've been trying my best

The storage size issue only really gets amplified if the clients retain long term incremental backups. Yes, it takes more space, but its a tradeoff we (at least in our specific situation) need to make. Given each GFS point already is effectively free, its not so bad.
Tyler Jurgens
Veeam Legend x3 | vExpert ** | VMCE | VCP 2020 | Tanzu Vanguard | VUG Canada Leader | VMUG Calgary Leader
Blog: https://explosive.cloud
Twitter: @Tyler_Jurgens BlueSky: @explosive.cloud
sfirmes
Veeam Software
Posts: 312
Liked: 147 times
Joined: Jul 24, 2018 8:38 pm
Full Name: Stephen Firmes
Contact:

Re: S3-integrated Storage - Block Size

Post by sfirmes »

Also when the SOSAPI starts to use the object size as noted in the blog, it will be at the bucket level and every job pointed to that bucket will use that object size. As @Gostev mentioned, you can see significant storage growth when increasing the object size from the 1MB default/best practice value. Due to that, you should try sticking to changing the object size on a workload/job level and monitor the effect of the change.
Steve Firmes | Senior Solutions Architect, Product Management - Alliances @ Veeam Software
tyler.jurgens
Veeam Legend
Posts: 418
Liked: 244 times
Joined: Apr 11, 2023 1:18 pm
Full Name: Tyler Jurgens
Contact:

Re: S3-integrated Storage - Block Size

Post by tyler.jurgens »

That's all well and good, and we've been changing block sizes on individual backup jobs. However, when you're talking many customers, each with many jobs, changing these all takes a significant amount of time.
I agree that we need to change and monitor, which we have. It just doesn't scale well and is prone to human error.

Also concerned with how Capacity/Archive tier block sizes get set. For a direct to S3 backup job, its straight forward - change the block size on the job, run an active full and call it a day. Backup Copy jobs or Capacity/Archive tier jobs are more difficult - one has to change the primary backup job block size setting.

Unless the SOSAPI knows enough to change the block size on any jobs that either have a backup copy or are part of a SOBR with offloading to S3, forcing the block size on backup jobs alone isn't going to have a major impact, as I suspect most customers will use S3 as a backup copy target, or SOBR offload target. I may be wrong here though, Veeam may have more clear metrics on how S3 is being used.
Tyler Jurgens
Veeam Legend x3 | vExpert ** | VMCE | VCP 2020 | Tanzu Vanguard | VUG Canada Leader | VMUG Calgary Leader
Blog: https://explosive.cloud
Twitter: @Tyler_Jurgens BlueSky: @explosive.cloud
k00laid
Veeam Vanguard
Posts: 228
Liked: 56 times
Joined: Jan 13, 2011 5:42 pm
Full Name: Jim Jones
Location: Hurricane, WV
Contact:

Re: S3-integrated Storage - Block Size

Post by k00laid »

@gostev @sfirmes, I hate to resurrect this topic but I'm trying to understand the "why" of the storage increase when changing from a 1 MB to 4 MB block size. Is it just a matter of creating a higher liklihood of "misses" as deduplication is applied through block generation because 4x as much data has to match?
Jim Jones, Sr. Product Infrastructure Architect @iland / @1111systems, Veeam Vanguard
Gostev
Chief Product Officer
Posts: 31977
Liked: 7441 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: S3-integrated Storage - Block Size

Post by Gostev »

Because incremental backups are block level so when a given data block had say 200KB of data changed in it, in the first case incremental backup will need to store only 1MB of data (which will include unchanged data that surrounds the changed 200KB), but in the second case 4 times more data will need to be stored (4MB block).
emachabert
Veeam Vanguard
Posts: 395
Liked: 169 times
Joined: Nov 17, 2010 11:42 am
Full Name: Eric Machabert
Location: France
Contact:

Re: S3-integrated Storage - Block Size

Post by emachabert »

This is in fact no so trivial when you have huge amount of data, metadata operations surrounding the data block (like retention information) are something to take into account.

We are dealing with perfomance (job duration) issues when you do immutable backup copy to an S3 object storage with mutli TB VMs (which are included in job with regular VM..). You can spend hours (like 5 to 10 hours) where veeam will not transfert any data but just storm the storage with millions of putObjectRetention api calls since there are so many objects to put the retention information on. We are currently in an extensive analysis of the pattern, fine tunning of Veeam internals and object storage internals with the help of both vendor support teams to mitigate this side effect.
At least this is interesting as deep diving the thing makes you understand how everything works under the hood.
Veeamizing your IT since 2009/ Veeam Vanguard 2015 - 2023
tyler.jurgens
Veeam Legend
Posts: 418
Liked: 244 times
Joined: Apr 11, 2023 1:18 pm
Full Name: Tyler Jurgens
Contact:

Re: S3-integrated Storage - Block Size

Post by tyler.jurgens »

We have had the same challenges @emachabert. The small objects Veeam creates are definitely a challenge for any HDD based Minio deployment. Sure, we could solve those by switching to NVMe drives, but the cost of that is prohibitive.

Our best solution has been to increase block size to 4 MB (or even 8 MB, which is unlocked with a regedit). I would prefer if Veeam could allow us to set this value on Backup Copy Jobs and SOBR Offload settings, but as of yet that feature request has not been implemented. I understand the desire by Veeam to keep object sizes small, but it makes things very challenging for us.

We've been very honest with customers when telling them to go to 4 MB blocks and what the consequences are (increased size, fewer api calls, etc). Haven't had push back on it yet.
Tyler Jurgens
Veeam Legend x3 | vExpert ** | VMCE | VCP 2020 | Tanzu Vanguard | VUG Canada Leader | VMUG Calgary Leader
Blog: https://explosive.cloud
Twitter: @Tyler_Jurgens BlueSky: @explosive.cloud
sfirmes
Veeam Software
Posts: 312
Liked: 147 times
Joined: Jul 24, 2018 8:38 pm
Full Name: Stephen Firmes
Contact:

Re: S3-integrated Storage - Block Size

Post by sfirmes » 2 people like this post

One thing to note is MinIO used NVMe drives in their Veeam Ready testing: https://www.veeam.com/sys1047

Have you contacted MinIO to see if they have any recommendations on what performance metrics you can expect with an hdd setup?

Thanks

Steve
Steve Firmes | Senior Solutions Architect, Product Management - Alliances @ Veeam Software
edh
Service Provider
Posts: 345
Liked: 102 times
Joined: Nov 02, 2020 2:48 pm
Full Name: Manuel Rios
Location: Madrid, Spain
Contact:

Re: S3-integrated Storage - Block Size

Post by edh » 1 person likes this post

@emachabert How many disk did you deployed in your minio? Is it distributed deployment? We deployed 1PB Minio last week and we're getting 8Gbps in writing performance
The most problem we encounter that Minio lacks of support even with their slack channel and the documentation is incomplete when you want to enable Nginx cache features.
Service Provider | VMCE
emachabert
Veeam Vanguard
Posts: 395
Liked: 169 times
Joined: Nov 17, 2010 11:42 am
Full Name: Eric Machabert
Location: France
Contact:

Re: S3-integrated Storage - Block Size

Post by emachabert »

We are not using MinIO.
This is a multi PB, geo distributed object storage.
We have no issues on throughput either, we reach insane numbers as each node has 50gb/s front and 50gb/s back.

The issue is about the number of objects within one single bucket. When you reach a billion of objects on which you have to put retention metadata you need to be careful. Issues arise when using object lock.

Using SOBR with multiple buckets to spread the load is one trick that makes things smoother.
Veeamizing your IT since 2009/ Veeam Vanguard 2015 - 2023
tyler.jurgens
Veeam Legend
Posts: 418
Liked: 244 times
Joined: Apr 11, 2023 1:18 pm
Full Name: Tyler Jurgens
Contact:

Re: S3-integrated Storage - Block Size

Post by tyler.jurgens »

sfirmes wrote: Sep 02, 2024 1:26 pm One thing to note is MinIO used NVMe drives in their Veeam Ready testing: https://www.veeam.com/sys1047

Have you contacted MinIO to see if they have any recommendations on what performance metrics you can expect with an hdd setup?

Thanks

Steve
We understand the performance metrics we can expect with our Minio deployment. Hence the desire for larger blocks, specifically on backup copy jobs or SOBR offload jobs. We have over 4 PB of Minio deployed (in a single cluster across multiple pools) and can ingest at multiple Gbps.

I feel we are caught between a rock and a hard place. NVMe is cost prohibitive (although how amazing would that be!). Veeam uses small objects by default. A happy medium would be for Veeam to allow us to change the Block Size on a Backup Copy Job (or a SOBR Offload) and allow us to set an object size that our Object Storage (in this case Minio) is happier with, fully understanding that there is tradeoffs with backup sizes as a result. We don't always want to change block sizes on the initial backup job, because it may have been in place for years, or the customer's storage can't handle new full backups for the size change to take effect. Whatever the case is, its not always easy to put in place. Nor do we want to use a backup job for direct to S3 because it adds additional load to the production infrastructure. Also, despite my own personal desires, we don't only ingest backups from one backup product (it gets incredibly interesting when you dig into how different vendors deal with backups - from one vendor making one object per block, to another vendor making one object per drive that gets backed up).

It will make a huge difference for those of us stuck between a rock and a hard place.

Also, our issue is slightly different than @emachabert's - we don't have an issue with retention metadata, since that's not handled in any database for Minio.
Tyler Jurgens
Veeam Legend x3 | vExpert ** | VMCE | VCP 2020 | Tanzu Vanguard | VUG Canada Leader | VMUG Calgary Leader
Blog: https://explosive.cloud
Twitter: @Tyler_Jurgens BlueSky: @explosive.cloud
tyler.jurgens
Veeam Legend
Posts: 418
Liked: 244 times
Joined: Apr 11, 2023 1:18 pm
Full Name: Tyler Jurgens
Contact:

Re: S3-integrated Storage - Block Size

Post by tyler.jurgens »

edh wrote: Sep 02, 2024 1:40 pm @emachabert How many disk did you deployed in your minio? Is it distributed deployment? We deployed 1PB Minio last week and we're getting 8Gbps in writing performance
The most problem we encounter that Minio lacks of support even with their slack channel and the documentation is incomplete when you want to enable Nginx cache features.
We didn't go the NGINX route, we use HAProxy instead and are very happy with it. That said, I don't believe Minio recommends caching in either NGINX or HAProxy.
Tyler Jurgens
Veeam Legend x3 | vExpert ** | VMCE | VCP 2020 | Tanzu Vanguard | VUG Canada Leader | VMUG Calgary Leader
Blog: https://explosive.cloud
Twitter: @Tyler_Jurgens BlueSky: @explosive.cloud
edh
Service Provider
Posts: 345
Liked: 102 times
Joined: Nov 02, 2020 2:48 pm
Full Name: Manuel Rios
Location: Madrid, Spain
Contact:

Re: S3-integrated Storage - Block Size

Post by edh »

Service Provider | VMCE
emachabert
Veeam Vanguard
Posts: 395
Liked: 169 times
Joined: Nov 17, 2010 11:42 am
Full Name: Eric Machabert
Location: France
Contact:

Re: S3-integrated Storage - Block Size

Post by emachabert »

We observe two things that are demanding for the object storage when you are in a clustered multi node and multi site setup that needs synchronization over large number of operations:
- delete operations wich are POST operation grouped by default in batch of 1k by Veeam. If you receive millions of delete you will produce millions of small operation that need to be synchronized accross nodes (no matter what the metadata storage is). This is demanding (and we have more than 25TB of nvme for clustered dedicated metadata space)
- putObjectRetenion calls that are also sent in batch after the data transfert thus producing the same effect of smal operation storm

By the way we are using both nginx and haproxy.
Nginx are front line and act as reverse proxies with different feature in filtering, measuring and spreading the TLS compute over multiple servers.
Ha proxy are second line loadbalancer handling fast loadbalancing with healthchecks and transparent retrying without notifying the clients, also handling end to end TLS using multiple haproxy servers.

IMO caching in front of the storage has no benefit in a backup use case... if you serve frequently accessed data with your object storage then why not configuring per bucket caching at nginx level but I wouldnt do this for a backup use case.
Veeamizing your IT since 2009/ Veeam Vanguard 2015 - 2023
karsten123
Service Provider
Posts: 512
Liked: 126 times
Joined: Apr 03, 2019 6:53 am
Full Name: Karsten Meja
Contact:

Re: S3-integrated Storage - Block Size

Post by karsten123 »

Eric, what is the product we talking about?
tyler.jurgens
Veeam Legend
Posts: 418
Liked: 244 times
Joined: Apr 11, 2023 1:18 pm
Full Name: Tyler Jurgens
Contact:

Re: S3-integrated Storage - Block Size

Post by tyler.jurgens »

Thanks for the link Manuel, interesting read. That said, I'll echo Eric's statement, its not going to help for use as a backup repository.

Now I'm super curious about Eric's Nginx & HAProxy deployment. My guess is he's using a Ceph cluster for S3.
Tyler Jurgens
Veeam Legend x3 | vExpert ** | VMCE | VCP 2020 | Tanzu Vanguard | VUG Canada Leader | VMUG Calgary Leader
Blog: https://explosive.cloud
Twitter: @Tyler_Jurgens BlueSky: @explosive.cloud
emachabert
Veeam Vanguard
Posts: 395
Liked: 169 times
Joined: Nov 17, 2010 11:42 am
Full Name: Eric Machabert
Location: France
Contact:

Re: S3-integrated Storage - Block Size

Post by emachabert »

No, it is not Ceph, it is a stretched RING over multiple datacenter, used with multiple protocols (file and object).
Veeamizing your IT since 2009/ Veeam Vanguard 2015 - 2023
AlexandreD
Service Provider
Posts: 51
Liked: 2 times
Joined: Jan 22, 2019 4:21 pm
Full Name: ALEXANDRE D
Location: Reims, France
Contact:

Re: S3-integrated Storage - Block Size

Post by AlexandreD »

emachabert wrote: Aug 30, 2024 8:08 am This is in fact no so trivial when you have huge amount of data, metadata operations surrounding the data block (like retention information) are something to take into account.

We are dealing with perfomance (job duration) issues when you do immutable backup copy to an S3 object storage with mutli TB VMs (which are included in job with regular VM..). You can spend hours (like 5 to 10 hours) where veeam will not transfert any data but just storm the storage with millions of putObjectRetention api calls since there are so many objects to put the retention information on. We are currently in an extensive analysis of the pattern, fine tunning of Veeam internals and object storage internals with the help of both vendor support teams to mitigate this side effect.
At least this is interesting as deep diving the thing makes you understand how everything works under the hood.
Hello, same performance "issues" here.
We have install a S3-compatible object storage in a second datacenter, and send all backup copy to it.
raw performance/throughput are good, around 50 Gbps ( ~5GB/s) so if we do a active full, storage ingest are quit good.
But daily, on a 12 hours windows for backup copy, there is few traffic, jobs are doing something in background, but there is no transfered data during hours. All VMs are stuck for a while at 99% status, at "finalizing" step.
Often we have error about gateway server (direct mode in bucket S3) not available, or the mount server are consuming all it CPU (12 vCPU @3Ghz).
So i try some change, create SOBR with multiple S3 bucket, create one dedicated mount server by bucket. I don't find lot of best practices for big environnement
https://bp.veeam.com/vbr/2_Design_Struc ... bject.html
so i try some things, delete, retry other things.

Have you made any progress in your analyses? Have you found any ways to improve?

thank you

Alexandre
sfirmes
Veeam Software
Posts: 312
Liked: 147 times
Joined: Jul 24, 2018 8:38 pm
Full Name: Stephen Firmes
Contact:

Re: S3-integrated Storage - Block Size

Post by sfirmes »

@AlexandreD have you opened a support case with Veeam and/or the object storage provider? If so, could you please provide the case#(s) so that we can look into this further. Also, are you running VBR v12.2 or higher? In VBR v12.2 we made some significant design changes with regards to how we handle object deletions. Here is a snippet from the VBR v12.2 What's New document:
V12.2 is a recommended upgrade to all object storage users due to the multiple under-the-hood
improvements and optimizations. The new version should put significantly less stress on object storage by
reducing the frequency of certain API calls by up to 10 times, which also reduces costs for cloud object
storage providers that do charge for API calls. We’ve also reduced gateway server memory consumption
and enhanced our code’s reliability in several corner cases observed in customer support.

Background checkpoint removal: Checkpoint removal operations have been decoupled from
the backup job and will now run as a system session after backup jobs and offload processes finish
and there are no conflicting tasks. Checkpoint removal has the lowest priority for the task scheduler.
In addition, this process is triggered daily at 3 a.m. to perform the same activity for backup no longer
associated with any jobs. This change will remove additional load from object storage when it is already
extremely busy accepting incoming backups and prevent backup jobs from appearing to “hang” at
the end while processing checkpoint removal.

Intelligent extent selection: Unstructured data backups pointed to SOBRs will now consider running
task count and the available free space during extent selection. This more intelligent allocation will
improve load distribution across SOBR extents compared to just selecting them solely based on
available free space.
In VBR v12.3 we made additional enhancements to the deletion feature introduced in v12.2 and we also added "Automatic Bucket Provisioning" which can also help underperforming object storage platforms. Here is a snippet about these enhancements and you can find the full details on these in the VBR v12.3 What's New Document:
Automatic bucket provisioning — Many S3-compatible object storage solutions use dedicated metadata
databases for each bucket, so creating multiple buckets helps to spread the load. While a scale-out backup
repository supports adding multiple buckets to Performance and Capacity tiers, there has been no solution
for customers using regular object storage repositories, which routinely causes performance issues as
buckets grow large, due to the metadata database becoming too large to promptly serve S3 API requests.
To address this issue, newly created standalone object storage repositories can automatically provision
a new bucket for every given number of workloads protected. You can also enable this setting for existing
repositories, however new backup placement rules will apply to newly protected workloads only. Managing
buckets automatically requires additional CreateBucket and DeleteBucket permissions on object storage,
whose presence will be validated by the Backup Repository wizard if the corresponding option is selected.
While we expect most object storage to benefit from this new capability, some advanced storage solutions
perform similar load-balancing natively under the hood. Please check with your storage vendor whether
they recommend enabling this option.

Background checkpoint removal enhancements — The reporting around background checkpoint
removal activities added in version 12.2 resulted in many upgraded customers receiving errors caused by
object storage infrastructure issues they were not aware of. V12.3 approaches this problem from a few
angles. First, it significantly increases timeouts on operations that were common root causes for object
storage connection failures simply due to object storage being too slow to respond, such as certificate
retrieval. Second, the failure reports now offer more insights into object storage issues encountered and
possible underlying causes. Finally, in light of checkpoint removal being a retriable problem as opposed
to a critical backup issue, V12.3 now lowers the severity of this event from Error to Warning and adds
an ability for users to make it an information event by creating ObjectStorageCheckpointRemovalSeverity
(DWORD, 0 — info, 1 — warning, 2 — error) registry value under the HKLM\SOFTWARE\Veeam\Veeam
Backup and Replication key on the backup server.
Hope this helps.
Steve Firmes | Senior Solutions Architect, Product Management - Alliances @ Veeam Software
edh
Service Provider
Posts: 345
Liked: 102 times
Joined: Nov 02, 2020 2:48 pm
Full Name: Manuel Rios
Location: Madrid, Spain
Contact:

Re: S3-integrated Storage - Block Size

Post by edh »

Just our two cents, after several weeks investing time in Object storage on-prem Distributed Cluster(+1PB) blablabla, we roll back. As service provider you cant manage the block size of the customer, at least in backup copy jobs. I saw customers sending objects of 256K , 1MB , 4MB **1. In our experience the biggest the best. Less objects less S3 calls, less GETs/HEADS/PUT/DELETES. Every S3 software implemented the S3 "protocol" as they want, some S3 solution got deferred deletes, other use PostSQL for metadata others rocksdb etc... and let you configure at gateway if a object is less than XX KB put in SSD places etc..
We didnt found any improvement about Automatic Bucket Provisioning, maybe in CEPH that metadata is per bucket "sharding", but not in Minio that works with FS. The only way that this can be improved is that repo vcc send minimum block size to vcc clients as its done for other options. But I think this will not be done because after reading several threads in this forum. With NVME / QLC SSD you will not have any problem with S3, just with HDD S3 backend. "HyperStorage" providers solve that with custom S3 software. Even they got some problems to deal within.

And with inmutable backups is even works worst because if you trace what is your S3 object storage doing the op called PutObjectRetention takes "hours" for refresh. The only thing I've done missing immutability in XFS is that it allows it to be done without GFS or Full periodic.

And remember , most of the current S3 Solutions use DIRECT_IO for their writes... your performace will be degraded on heavy loads.

PD: Take care about CMR o SMR disk...
**1: Check custom registry for enable 8MB Block Size
Service Provider | VMCE
tyler.jurgens
Veeam Legend
Posts: 418
Liked: 244 times
Joined: Apr 11, 2023 1:18 pm
Full Name: Tyler Jurgens
Contact:

Re: S3-integrated Storage - Block Size

Post by tyler.jurgens » 1 person likes this post

at least in backup copy jobs.
Unfortunately Veeam currently has no way of adjusting block size for Backup Copy or Capacity Tier jobs directly. Block size at this time must be adjusted on the backup job itself.

The best we have found to mitigate the Veeam impact on our Minio deployment (outside of using NVMe drives, which aren't cost effective) is to adjust the block size as high as possible on the client. Often this requires us to create (or request the client to create) *new* backup jobs targeting our S3 directly with the 8 MB block size. Stored size is less of a concern to us than performant backups. This results in larger objects which are easier on HDD drives to read/write, and easier to shard across the pools. It also means far fewer PutObjectRetention calls. One customer's backups went from *days* to hours just by increasing the block size.

I've asked Veeam to allow Backup Copy Jobs to adjust the block size. I seem to recall seeing a post by Anton saying its not possible to change the block size on a copy job. Alternatively though, instead of changing the block size, perhaps Veeam could look to 'bundle' blocks together into one large object. For example, 64 blocks become one large object rather than 64 individual objects. However, this could mean only a single block in that bundle requires immutability but the entire object needs to be stored for the length of the immutability period on that one block. Size would get inflated quite a bit, but again, might be a worthy tradeoff. Perhaps some intelligence could be added as well - if Veeam would intelligently bundle these 64 blocks into one object based on immutability requirements (eg: one set of 64 blocks need to be immutable for 1 month - vs 64 other blocks needing to be immutable for 1 week), it might mitigate the size inflation. This could solve two problems at once - object size impacting performance and decreasing the number of API calls required to make those objects immutable. Ultimately, it would present other challenges of course. So as said above - its a tradeoff, but one we'd be interested in exploring. Bundled object size could also be configurable as well - maybe one company is ok with a bundle of 8, whereas another company would prefer a bundle of 128.
Tyler Jurgens
Veeam Legend x3 | vExpert ** | VMCE | VCP 2020 | Tanzu Vanguard | VUG Canada Leader | VMUG Calgary Leader
Blog: https://explosive.cloud
Twitter: @Tyler_Jurgens BlueSky: @explosive.cloud
Post Reply

Who is online

Users browsing this forum: claudio.fortuna, Semrush [Bot] and 10 guests