Primary ReFS Backup Repository Space Savings?

[DaStivi]

Hello there i've read the Blogpost from Luca Dell'Oca (http://www.virtualtothecore.com/en/wind ... epository/)

and he states following:

NOTE: there is NO space saving on primary backups during transform operations, as the same block is always written once, and it’s only moved from the incremental backup file to the full one. Transform operations in primary jobs are about time saving, not space saving.

when i understand that correctly there is only speed benefit from refs on primary backups? i thought and this is what Veeam promised there is space saving on synthetic fulls too?

last week i've upgraded to Veeam 9.5, than did a inplace upgrade from my 2012r2 server... all fine ... then i created a new ReFS BackupRepository and copied all the old Veeam Backupfiles to this Repository.... in the past i did just forward Inc. forever, as synthetic fulls wheren not applicable because of space issues... every week a syn. full with almost 3TB is too much... so the refs "Spaceless full backup technology" promised a huge step forward....

what i saw now over the weekend, that my 7TB Repository where getting filled completely and backups started too fail... today in the morning i checked what happend, and what i saw first was a backup server with without any ram... some process eaten all memory available at least 32gb... every task worked terribly slow on the server... i was not able to start the vbr console but i've managed to delete some old backup files and after some time i get dropped from the server and couldn't connect for some time ... i had to to something different in the time but after that i managed to loggon back and was greeted with dirty shutdown warning.. the machine bsod'ed (maybe because out-of-ram)

Abnormal Program Termination (Windows bug check, STOP: 0x0000013C (0xFFFFA405955E8080, 0x00000000000000E8, 0x0000000000000000, 0x0000000000000000))

anyway... i'm up and running now.. have space left on the backup repository and tried a controlled synthetic full run now...

i see the "fast clone" full backup creation and yes as luca stated, no space savings... the fullbackup does have the fullsize on the disk, what is very disappointing!


all the space is used:


i'm left now with the simple question: why does "space savings" not work in primary backups? as this would have really changed a lot too!

[dellock6]

Hi,
sorry if the post has created this small confusion, the main difference is between incremental backups (IO savings) versus synthetic operations (IO "AND" space savings). So, it's correct that also primary backups can give you space savings if you do synthetic fulls, exactly like GFS retention on a backup copy job. Sorry for the confusion, I'm editing the post to make it more clear. Thanks for the heads-up.

[DaStivi]

ok then i understood it correctly... the question is now why does my syn. full does not leverage space savings... ?!

edit:
i did a active full now... but i have to wait till tomorrow to try a synthetic full as Veeam does not do a syn. full now after the active full (or the synfull from testing before...)

edit2:
btw. should the refs integration work with "perVM" Backupfiles too?

[vClintWyckoff]

I even myself had to read Luca's post a few times, but it is 100% accurate. Space savings AND Time Savings will be experienced on the Synthetic operations.

So for me to clarify, for example....

Where you need to look for the disk space savings is within the Used Space on the ReFS volume. Synthetic Fulls do not consume space on ReFS volumes, because they are created by referencing the disk blocks already stored on the disk as a part of full and incremental backup files which were previously created. So a good test of this is to have a primary job create a few synthetic fulls or create a backup copy job that creates a few GFS restore points. Then look at how little the used space on ReFS volumes changes after each new synthetic VBK is created.

So you shouldn't be confused by the fact that Windows Explorer shows the actual size of these VBKs, they're literally normal files BUT they're sharing their blocks with existing files. However, you'll note that the total size of all backup files just does not add up to Used Space on the volume.



Also, to clarify - copied VBK / VIB files to a new ReFS volume will not benefit from the FastMerge / Block Clone API. What happens here is that the BlockClone API requires that the file blocks be aligned to the 4k or 64k boundary. So either an Active Full or Synthetic Full are required to get the benefits of the new integration.

[DaStivi]

The issue here was that even the sum on volme information added up to full space...

The file information, as far as I understood it should also make notice od that... At least with the "size on disk" information... As with when using dedup...

But I will see tomorrow after ah active full... Maybe changing backuprepo and using perVM file broke the mechanisms...

[dellock6]

Stefan, as said before, just moving Veeam files into the new ReFS repository is not enough, as blocks need to be aligned to 4k or 64k cluster size to leverage the ReFS blockclone API. The native OS file explorer doesn't align writes, so the file deployed in the new volume are not aligned. You need a full to align the block and start consuming the api with the following incrementals.

In regards to the images you posted, you need to open the properties of the root volume to check the overall disk space used, files in the file system will always show their file size with no space reduction. The 50GB full will always be shown as 50GB regardless the gfs spaceless full backup. Check the root of the volume V: in your example.

[DaStivi]

Ok i unterstand that... As said i did a active full now... But have to wait for tomorrow as now no new synthetic full can be created, only normal increments gets created...

Anyway, even when existing files are copied and of course no space reduction is done from windows explorer, wouldn't it be possible to "activate" the block clone API (alignment) at least with the next syn.full? What I described in my opening post I thought that would happen but instead I created "normal" synfulls that ate all remaining space...

[tsightler]

Synthetic full is enough to activate block clone, but you can't clone blocks from the old, unaligned chain to the new, aligned chain, so the first creation of a synthetic on a ReFS enabled repository using backups copied from another repository will require the same amount of space as a traditional synthetic. After that, new synthetics should be able to share blocks with previous synthetics. In your screenshot above it definitely looks like the VBK from 11-14 is sharing blocks with the VBK from 11-12, although it's impossible to tell completely with the screenshot provided as you won't see any savings on the file properties.

[DaStivi]

some files looks like to have leveraged the ReFS API now:



i'm however not totally sure what files... furthermore i haven't done a full, or syn.full on my production store with almost 4TB, that filled the store entirely.

at least the fastcloning API does its job very well

[futureweb]

I am wondering if there is a way to see the acutal file or folder size that is used?

With DEDUP you can see acutal file size and file size, but with REFS, unfortunatly this difference was missed.
As you mentioned, only looking at the drive size itself shows you the real size, so not sure if this is something MS is missing,
or any other possibility to view it per file or folder?

We see quite amazing savings. 13,5 TB and about 7 TB used, while we have 1,3 TB of Endpoint backups,
where I am not even sure if it leverages any savings here, as it's only 1 full and many inc's, or if any fast cloning is happening,
when inc's of endpoints are merged.

Anyway, synthetic full take about 27 minutes on 5,1 TB, which is 3,2 GB per second which sounds pretty good to me. (having it weekly and inc's daily).


Thanks
Patrick

[millardjk]

It's too bad that Microsoft didn't think the fastclone feature could use some explicit UI reporting for space savings. Having to pull up the volume root to see the actual space consumed versus all the folders for the logical file use is...so 1992.

[nsumner]

What I am wondering is, is there ANY way that I can take advantage of block clone on old files? Any way at all! I'm out of space and want to use block clone to solve the problem but I must be able to use it for my archive!

[Mike Resseler]

Noach,

What do you mean with your old files? Are you saying the files that are already copied to a ReFS repository but are not aligned? (And please be aware that there are indeed space savings as you can see from above but again, it is not a deduplication technology so space savings really can be very dependent)

[nsumner]

Hi what I mean is I currently have backups on a NTFS disk. I want to create a new REFS disk and move the files from the NTFS to the REFS and wipe out the NTFS (and ultimately add those drives to the REFS).

Right now I'm going to need to do things in a very backwards manner (move to NTFS with Dedup). Move to a larger REFS after eliminating the main NTFS drive (I'm buying more HDs so I can create another RAID array during the move). But even after all is said and done I will have what is nearly 50TB of data taking up a large amount of space it shouldn't!

[DaStivi]

What I am wondering is, is there ANY way that I can take advantage of block clone on old files? Any way at all! I'm out of space and want to use block clone to solve the problem but I must be able to use it for my archive!

i understand that like you have your vbks and copied them to ah new refs repo... you can not take advantage of refs fastclone instantly... you have to run a syn. full at least... that takes the fullamount of space for the vbk.... but after that fastclone on next syn full is used!

so you need at least 2times the vbk space.... than you can take advantage of refs!

[Mike Resseler]

Can you work part by part? Doing a migration per backup job (for example)? From the moment that migrated backup job is using the block cloning API (Don't forget simply copying will not align the blocks automatically!)

[nsumner]

I understand the fastclone will only take effect after the first full backup. But right now I have 50TB say of backups. They are monthly and probably 90% the same so let us just say for arguments sake that it actually would require 10TB if all of those backups had gone to REFS starting 5 months ago.

I want to get my current 50TB down to 10TB! But simply copying files doesn't actually take advantage of fast clone. How can I take advantage of the fast clone to get things working normally.

[nsumner]

Can you work part by part? Doing a migration per backup job (for example)? From the moment that migrated backup job is using the block cloning API (Don't forget simply copying will not align the blocks automatically!)

Mike how do I do a migration per copy job? Will that align the blocks? I can do it and would love to do it! But I'm not 100% clear on how you want me to do it.

BTW Does the server Veeam is on have to be Windows 2016 or only the server with the storage attached to it?

[futureweb]

Only the Server with Storage attached to it needs to be Windows 2016. We have it this way, and works fine. Veeam is still on 2012.

[nsumner]

Thanks, that was my plan. Do you know how to a migration copy?

[foggy]

Mike how do I do a migration per copy job? Will that align the blocks? I can do it and would love to do it! But I'm not 100% clear on how you want me to do it.

You need to trigger full backup on each job one by one. After the first full, the job will start utilizing fastclone and subsequent fulls will be "spaceless".