fastclone slow - server 2016

[parneye]

Hi

I've read through the veeam-backup-replication-f2/refs-4k-hor ... 40629.html and veeam-backup-replication-f2/synthetic-f ... 51567.html articles.

- Veeam is 9.5 Update 4 (and also was this version when the job was created)
- The OS is Server 2016 and was "not" up to date when the ReFS and Job was made - it is now, but the fastclone is still very slow (24 hours).

- 2019-11 Servicing Stack Update for Windows Server 2016 for x64-based Systems (KB4520724) (Successfully Installed)
- 2019-12 Cumulative Update for Windows Server 2016 for x64-based Systems (KB4530689) (Successfully Installed)

- Repo is 64K, 10TB.
- Backup VBK size is 2TB.

Is there anything else I can try - or do I need to blow away the backup and start again?

Thanks

[Gostev]

This is quite strange... basically, your ReFS volume is too small to experience such a significant ReFS performance issues with any Windows version whatsover, patched or not.

So I would actually look at other things first:
- Do you have enough RAM on the server?
- Are you sure fast cloning is actually used?
- Storage issues?
- Antivirus?
- etc.

Please add support case ID for this issue, as requested when you click New Topic.

[wa15]

For what it’s worth, we were having similar issues but removing antivirus (Symantec AND the built Windows defender) seemed to help. May be worth a shot.

[parneye]

- I believe RAM is sufficient. It has 10GB, we can easily increase it to see if it makes a difference..
- In the Veeam log, it says: Synthetic full backup created successfully [fast clone] 22:49:46
- Storage Performance is good.
- Antivirus is installed - I have not tested disabling

I will go with your suggestions of disabling AV and see if that helps.. For what it's worth, we have a very similar setup (albeit Windows 2019), that runs perfect..

[parneye]

please see previous message for the answers to the questions asked..
i have now also tested disabling AV, and also monitored the RAM / CPU.
AV made no difference, and during the fast clone CPU is low (~ 5 - 20%), and RAM is sitting on 79% used.

any other ideas?

[Gostev]

The amount of RAM is likely the culprit. Assuming you have all-in-one Veeam install on a single server, you're below minimal system requirements. Remember you need to add up RAM requirements for each backup infrastructure component installed on the server.

Also, you did not add support case ID for this issue according to my earlier request. Keep in mind that topics on technical issues without support case ID I eventually removed by moderators, as explained when you click New Topic.

[parneye]

Hi Gostev.. okay. apologies regarding Case ID as I have not created a case yet (i thought i'd reach out in the forums first) - i took the bottom of your message as a default signature or something similar.. the RAM has been upped to 16GB with the same speed of clone.. RAM Used is sitting on 81% used - so not too sure how it is the issue. i can up it to 32GB to completely rule it out, but as far as I can see, it's not the cause.

[parneye]

hi Gostev.. please see note above as response.. in addition, i have gone ahead and created support id - 03948836

[Gostev]

Good. Looks like the amount of RAM may still be an issue. But please work with our support for further troubleshooting over a Webex, as trying to guess what is wrong with your server over forum posts is not a very productive approach

[parneye]

Hi Gostev, I managed to fix my problem by playing around with a few settings - it ended up being a shaper on the repository. i had it set to 30MB/s, which for some reason effected the speed of the fast clone.. after disabling the "Limit read and write data rates" on the repository, it resolved the issue (fast clone now takes 15 minutes instead of 20 hours)...
IMO there's a calculation where it averages out the size of the VBK over time and doesn't allow the clone to go past that point as it is "throttling it" (although the logs don't say it is being throttled).. i have not tried re-enabling it to see if the issue returns.
Also, with a basic calculation (2.5TB / 30MB/s) it comes out to 23 hours which is pretty close to how long it was taking..
Not too sure if this would be considered a bug to resolve, as I would like the datastore throttled to 30MB/s.
I'll update my support ticket.

[Gostev]

Thanks for the follow up! By pacing writes, you're helping with RAM load by system cache, so looks like this resolved your lack of RAM issue.

[parneye]

Hi Gostev.. i think you may have read my last post the other way around..
it was by turning off the throttle is the thing that fixed fastclone... unless I misinterpreted what you said..
In this case RAM is not the issue, as one of my tests i changed RAM to 32GB (sitting on 30% used) and the problem still persisted.
It was only by "disabling" the throttle on the datastore is what fixed it.
At 10GB the server was pretty tight, but RAM used was sitting around 80% - so even in that case I believe it would be okay.

it appears fastclone takes in to account the throttle on the datastore, in my opinion a bit incorrectly, because even though you could argue creating a 2TB file in 10 minutes is 3000 MB/s - that's not what is happening in reality... unless something else weird is going on that I have missed.