kubimike wrote:[tsightler]but it also reduces the benefit, since the new "chain" is still 100% dependent on blocks from the prior chain as it's sharing all of those blocks.
That being said, if all the VBKs are related somehow, when veeam deletes the oldest restore point (due to retention) that would make your statement false ?
But that's entirely what block clone is, each VBK is sharing any unchanged blocks from the prior VBK, when you delete the older VBK, that just means the blocks aren't shared anymore. Let's try a super simple example, three VBKs, each containing only 4 blocks, labled A-D:
On Monday the backup runs and creates the first VBK with the first version of each block:
Mon VBK = A1 B1 C1 D1
On Tuesday the backup runs and creates an VIB, but only block A has changed so the VIB consist only of the new block:
Tue VIB = A2
But the job was configured to create a synthetic full so what the system does is create a VBK and "block clones" the most recent version of this block into the VBK so you end up with a VBK like this:
Tues VBK = A2 B1 C1 D1
Note that this new VBK is simply referencing the same blocks as in the Monday VBK and the Tuesday VIB. Once the synthetic full is built, the system deletes the Tuesday VIB file, but that doesn't delete block A2 because the synthetically built VBK is still referencing block A2. Note that both the VIB and the VBK reference the very same block, so deleting the file freed no space from the actual disk, the exact same content remain on disk.
This continues for Wed-Friday, with different blocks changing each day:
Wed VBK = A3 B1 C1 D2
Thu VBK = A4 B1 C2 D2
Fri VBK = A5 B1 C2 D3
So now lets imagine it's time to delete the VBK from Monday because I've hit my retention. The system delete the Monday VBK, which contained blocks A1, B1, C1, D1, so what can actually be deleted? The Tuesday VBK still references blocks B1, C1, and D1, so those blocks cannot be freed, regardless of the fact that the file has been deleted. Only block A1 can actually be freed, because the other backup files still depend on some of those blocks.
In just this super simple example there are 14 block references which have to be tracked and processed by ReFS during delete. On the other hand, imagine if this was just regular incremental:
Mon VBK = A1 B1 C1 D1
Tue VIB = A2
Wed VIB = A3 D2
Thu VIB = A4 C2
Fri VIB = A5 D3
Note that this method has exactly the same blocks on disk, but exactly zero cloned blocks for the same amount of space. When retention hits on Friday the only thing that happens is that block A2 from the Tues VIB is cloned into the VBK, and then the VIB is deleted, much lighter and faster than having a bunch of blocks cloned to create a synthetic full. This is why I believe that we don't see the problem with customers that are running forever forward or reverse incremental modes, at least so far I haven't found one.
So assuming all of that made any sense at all, lets apply some scale to it! Let's assume you have a 5TB backup file. Assuming you are using normal Veeam settings for storage optimization (local block size and optimal compression), the average Veeam block is about 512KB (assuming 2:1 compression, could be slightly smaller if you get better compression). That means a 5TB backup file contains about 10,000,000 blocks! If you have 100 VBK files all made with block clone, and assuming your change rates are similar to observed averages to this point, probably 60% of your blocks are cloned across all 100 VBK files. Imagine the accounting that ReFS has to do to update it's block reference counts for all of those blocks.
On the other hand, if you ran weekly synthetic fulls, you'd use exactly the same amount of space but, at most, you'd have 15 VBKs on disk with cloned blocks, drastically reducing the reference block count, also, since it's only weekly, the amount of blocks cloned between files would be less as well because a lot more blocks change during a week than during a day.
Note that I'm not saying 100% that this would eliminate the issue in your case, obviously we have people even with weekly synthetic fulls that have the problem, but it's certainly providing additional stress to ReFS having to keep up with the extra accounting for so many block cloned files.