How to handle 20TB+ copy jobs to cloud? (cloud-side processing takes forever!)

[BackItUp2020]

I've had multiple Veeam tickets, but in the end, the processing of data on the cloud provided side seems to be the part that takes the longest time and is pushing our RPOs out of whack.

Example:
File server VM has multiple disks, but one 17TB disk.
File server back up locally - 31 minutes, 1.7TB "read", 6.8GB "transferred".
We have a dedicated circuit to our cloud provider.
Copy job sits on this large disk for over 17 hours, coping very little data in the end. The target processing, computations, etc. are the hold up.

What is recommended for large serves like this? I can only think of maybe CDP, but even that is quite an architecture change from where we are now.

I'd love to just get our file server into the cloud natively, but until then...

[tyler.jurgens]

Really hard to know as we don't have visibility into your cloud partner's configuration. I'm assuming you are using Backup Copy jobs to a Cloud Connect repository.

If the provider has S3 storage, you could try going direct to S3.
If you are going to Cloud Connect storage, is the provider using S3 repositories on the back end, or some form of XFS/ReFS repositories?
Are the jobs (going to Cloud Connect storage) being held up at all? Are they starting immediately?

You could try to do non CDP replication, but again your provider would have to support replication, not just offsite backup.

I'm not sure there really are any "recommendations" but I'd suggest this is more of a conversation between you and your provider, rather than you and the RnD forums. Bring Veeam support into it if you can to identify speed bottlenecks.

Every provider can architect their environment differently, which can result in very different speeds for customers. Talk to them and see what you can do with them to get you to the RPO/RTO you want.

[BackItUp2020]

Thanks - I forgot that non-CDP replication is an option. I'll have to check w/ our provider too to see which options are available.

On the back-end of the, I believe it is XFS repo as well.

From prior Veeam and provider cases, its mostly the computational aspects of the jobs on the cloud side. Determining which blocks have changed on the source side, copy those over and then applying those retention settings. It seems there should be a better way to do that step. Sort of like CBT but on the copy job side so a read of a 17TB disk doesn't take 16 hours to determine and eventually copy a few GB.

[tyler.jurgens]

We've had customers push very large backups (as a Backup Copy Job) to us, and beyond the initial active full that is created, it doesn't take long to push a small incremental upgrade to our repositories.

Talk to your SP and Veeam support. Something isn't right there if its taking 16 hours to process a few GB.

[BackItUp2020]

I'm in talks now (again) and will see what comes of it. Job bottleneck is the target side (97%) for HOURS.

Letssss gooooo!!

[tyler.jurgens]

Have you tried extending your retention policy - to eliminate any merge operations from happening? See if that's the problem?

Just guessing at this point, but with XFS repositories we've seen far better success with merge operations than with ReFS. Best to keep chatting with your SP as well as Veeam support.

[BackItUp2020]

I haven't, but from what I recall if I were to reseed the job, it would (of course) take time to copy, but finish promptly. Subsequent jobs would be quicky too until I hit that retention period and that's when things take a dive.

If I were to do what you are suggesting I think it would improve the job completion time until I reach the new retention period where I would see the same thing occur.

Locally, when I went from ReFS to XFS, everything got better. Faster jobs, much less disk space - it was great! Hope to hear some finding from our cloud provider side.

Thanks!

[BackItUp2020]

My apologies for the poor grammar there. I think it's too late for me to edit

[BackItUp2020]

UPDATE: After another support case, I'm in the process of validating, but the conclusion is that the large file is spending most of its time being decrypted in order to be opened on the cloud side. YUCK. We can't do unencrypted backups (but we will in order to test the theory), so not sure where we'll go from here.

[AlphaOmega]

I find this very thread very interessting, could you please point out what the cloud target really is technically?

[BackItUp2020]

I'm not sure exactly what you are looking for, but I do not have insight to the target repo, unfortunately. Next follow up call, I will see what I can find out.