Fast Clone not working after changing from NFS Repository to Linux Repository

[mark99]

hello, I'm trying to run this feature in a test lab:
veeam community 12.2.0.334 + truenas 24.10rc2 zfs raidz2 + a <file share> computer.

To begin with, I set up the environment the way it was set up in the production environment: Truenas NFS Share and Veeam NFS Share, gateway auto. we are not satisfied with the speed of merging copies

I installed veeam linux agent on truenas with zfs,
- checked the pool flag feature@block_cloning and zfs_bclone_enabled - both in the enabled state,
- created a file /etc/veeam/EnableZFSBlockCloning
- adding a new backup repository specified the same folder that is connected via nfs,
- turning on the xfs fastclone check mark, a warning appeared that openzfs is in preview mode (has the file system been determined?),
- enabled the import of existing backups

then I reassigned the backup repository in backup task to the new one I just created, and deleted the old one (nfs share)
however, fastclone remained disabled: when performing the transform, the merge occurs as before through copying

what else do I need to check to enable this functionality?
do I need to recreate the zfs pool or delete old archived copies?
what should I pay attention to in the logs, what could be the problem?

[david.domask]

Hi Mark, welcome to the forums.

Just to confirm, the repository itself is a Linux Server, but the directory used for backups is actually a TrueNAS mounted NFS share?

As far as I know, this won't work, as explained in this post.

A simple test would be just to cd to the mounted NFS directory from the linux box, copy some larger file there (maybe copy a VBK), then try:

cp --reflink=always file.vbk file-copy.vbk

Likely it will fail as I'm fairly certain the XFS reflink won't work over NFS like this.

[mark99]

Hi David.
I've probably described it badly.
initial configuration: nfs share, gateway auto (transform/merges by veeam server over nfs)

I have reconfigured to <linux server item> configuration + backup repository to this linux server without using nfs.
I checked the "use fast cloning xfs box", which is only available when adding a repository on a linux server
but copies are still merged through copying (in this case it is fast, since its size is very small in the test lab)

cp --reflink=always seems to work fine

there is another option specifically with truenas: ixsystems implemented fastclone support in their samba package, but it was in their repository the other day, and I haven't had time to check it yet ( 1 , 2 )

[david.domask]

Got it, thanks for the clarification, indeed I just misunderstood

Quick question, looks like it's a Forever Forward Incremental chain, correct? (i.e., no Active or Synthetic Full backups?)

When reconfiguring to the current configuration, did you start a new job or map to the already existing backups? Likely you are missing this requirement. While you didn't move the backups, effectively it's the same, and you need to either schedule a Compact Full or start an Active Full, and then the merges should use fast clone.

If you have a smaller VM, you can make a test job pointing to the linux repository, use Point based retention (step 4.), set the retention to something low like 1 or 2. Ensure that there are no full backpus scheduled, and then just run the job a few times until a merge happens, and you should see Fast Clone is used. If that's the case, then it's working, and very likely you just need to run Compact or Active full on the existing job.

[mark99]

Yes, that's right, a forever forward incremental chain imported from the old nfs repository.
Most likely, yes, I didn't do it. Thank you. I'll try it

I do not yet know what will be faster to do if truenas is slow enough and there may not be enough space in it for another active full copy. does defrag and compact overwrite all copies? otherwise, it doesn't matter, I'll go read the documentation again

[david.domask]

Compact full will do the following:

1. Make a new .temp file that is the target of the Compact Full operation
2. Write the compacted/defragged data from the original VBK into the .temp file
3. If all is good with the Compact process, the original VBK is removed, and the .temp file is renamed to .VBK and set as the full backup for the backup chain

So indeed, for a short period of time, you'll need space for an additional Active Full on the repository, but the original VBK is only removed _after_ the compact was successfully performed.

[mark99]

David, did I understand correctly that compact full only affects the VBK file of the primary copy, leaving the rest of the chain of VIB files unchanged and available for recovery after completion of the defrag&compact?
Accordingly, I need (relative to the screenshot) ~ 340GB for a short period of time to use fastclone in the future?

Thanks

[david.domask]

> David, did I understand correctly that compact full only affects the VBK file of the primary copy,

Correct, the process is described here in more detail.

> Accordingly, I need (relative to the screenshot) ~ 340GB for a short period of time to use fastclone in the future?

Correct, as mentioned in the User Guide. (3rd bullet point)

[mark99]

Hi David,
I am currently trying to enable fastclone on another installation
Initial configuration: veeam 12.2.0.334 + truenas on zfs, veeam agent installed, repository is nfs via gateway agent

I did the following:
1. checked zfs_bclone_enabled and feature@block_cloning - enabled
2. created file /etc/veeam/EnableZFSBlockCloning
3. added new repository <linux server>, chosen some directory, enabled xfs fast clone, import existing backups
4. changed the destination repository to a new linux server on existing jobs
5. I allocated a separate small job (3 vms) for the experiment, which consisted of Forever Forward Incremental chain (30 restore points)
5.1. I started a compact full task for her, which ended successfully
6. then I start the job again, reducing the number of restore points, and the merge happens through copying. fastclone is inactive.

where could I have made a mistake and which log should I look at?
thanks

[david.domask]

Hi Mark,

Unfortunately, can't quite say what would be preventing fast clone use for this backup chain.

May I ask you to create a temporary job with similar retention configuration and try a fresh backup and testing the merge? Perhaps create a small "dummy" vm quick and back that up (to save time/space). I suspect it may be some conditions aren't met still for the original backups, and this would help to confirm that.

[mark99]

Hi David,
I created a temporary new job with a similar storage configuration, three small VMs and 2 restore points, launched it 3 times and fastclone is active on merge.

The difference with that task is that Active Full was not performed for it.
But compact full was successfully completed for that task. I've re-read the documentation section on fastclone and it looks like it should work...
The disk size for one of the VMs was changed after Compact full




I'm comparing Transform.Target.log is on the storage server and seems to have found the difference:
New job merge starts with:

Code: Select all

Transform.Patch
(EString) BackupFileLink = ...
...
(EBoolean) VirtualSyntheticEnabled = true

Old job merge starts with:

Code: Select all

Transform.Patch
(EString) BackupFileLink = ...
...
(EBoolean) VirtualSyntheticEnabled = false

I went to look at the job transform log on the server, and I think I found a problem:

Code: Select all

Warning (3)    Linux synthetic is not available. Storage ... block alignment size 4096 is not a multiple of repository cluster size 1048576

A little higher on new job:

Code: Select all

... command: 'Invoke: DataTransfer.RestoreText
... output: <VCPCommandArgs><Item key="Text" type="EString" 
     value="...BlockAlignmentSize=~=1048576~= ... blockSize=~=1048576~= ..."

On old job:

Code: Select all

... command: 'Invoke: DataTransfer.RestoreText
... output: <VCPCommandArgs><Item key="Text" type="EString" 
     value="...BlockAlignmentSize=~=4096~= ... blockSize=~=1048576~= ..."

New backups in the old task are recorded with BlockAlignmentSize=1M.
I searched in the log when the change occurred and came to the conclusion that the 4K alignment block was used while the storage was connected via NFS.

It turns out, in addition to the fact that I need to make Compact full, I still need to wait until all the incremental copies recorded over NFS merge into a full copy, and fastclone will work only after that (probably...)

Do I understand correctly?
Perhaps there are any other ways to convert existing copies to a different alignment block size?
... Although, it doesn't make sense, it will take as long as merging the copies to the full.
We are now going to commission a new storage system, taking into account the amount of time it takes to overwrite everything, it will be easier to reconfigure backups from scratch starting with Active Full to it, and then simply delete all copies on the current storage.

In any case, I am grateful for your help in finding the reasons for this behavior.

Perhaps the documentation does not explicitly say about "Align backup file data blocks", since in both repositories (nfs and linux server) it was turned on, that the alignment block should be the same

[david.domask]

Hi Mark,

>Warning (3) Linux synthetic is not available. Storage ... block alignment size 4096 is not a multiple of repository cluster size 1048576

Yes, that would probably do it in this case. Really great research here, and indeed, I would say that's likely what's causing it.

An Active Full will likely help here, but if I got you correctly, you'll just handle this on the storage refresh, which makes sense.

Glad that indeed it was working, and that your work found the root cause. Glad to have been of some assistance, and indeed, I think we can expand on this a bit in the User Guide to be more clear and return more clear messages in the UI about why Fast Clone didn't get used in cases like this.

[mark99]

On the job for which I was doing Compact Full and all the copies that were written over NFS were merged into full, and fast clone started working.

The next question for me was how much VBK fragmentation affects merge speed without using fast clone: I highlighted a job where VBK is about 7TB, incremental copies ~300gb. Initially, the merge took place at a speed of ~30-40mb/s for 3-5 hours, then after compact full the speed increased to 100-150mb/s and took only 40 minutes.
Thus, the effect of fragmentation may well reduce the merge rate by 10 times.

I started to study this topic, since the interval of incremental copies is 1 day and at some point the merge did not keep up until the next interval. The situation was aggravated by the retention policy set in days, in total: skipping (or starting later) the task means starting the merge of two copies that will definitely not be in time. It accumulated like a snowball until I realized what was going on.

The NFS server in TrueNAS turned out to be quite slow and unstable, it hung for 20 minutes when merging several tasks at the same time.

Also, I found another strange thing: if a task stops in the middle of the process with a data transfer error and retry is performed for it, then the resulting size of the incremental copy may grow multiples for some reason. Those copies, which in fact should have occupied about 200-300gb, after 2-3 retries begin to occupy 800-900 GB.
David, do you have any thoughts on why this might be happening?

These were interesting experiments, and maybe they will help someone.