WAN Backup Copy Job very slow

[mintyevertonian]

Hi all

I set a backup copy job to run every 7 days. I did the intial sync on site and shipped the backup to an offsite location

I am running this job now which should be 20-30GB in size (increment) but it is taking forever to start transferring


The job is taking forever to perform these tasks

"change digests are not available"
"calculating disk digest"
"Digests are missing on source loading them from target"


Any ideas what these are?

Thanks

[Vitaliy S.]

Hi Fintan,

Have you performed seeding as described here? Did you use WAN accelerators or you've run your backup copy job in a direct mode?

Thanks!

[mintyevertonian]

The initial backup copy was done on site as a direct copy

Then I transferred the NAS box offsite and changed the job to use WAN accelerators .

It picks up the latest jobs locally but it is taking forever to transfer

Am i missing something regarding seeding?

[foggy]

After mapping backup copy job to existing backup, digests calculation is required to understand which blocks should be transferred to the target. Currently, digests calculation is indeed not very fast and can take considerable time, depending on the backup size (roughly 1 minute is required to calculate digests for 1GB of data, which results in ~17 hours to calculate digests for 1TB disk, for example) + additional time is required to transfer digests to the source over the slow network.

[asatnik11]

foggy,

What is the bottleneck that leads to the slow digest calculation? Is it CPU, Memory, Repository I/O or Source/Target WAN Cache I/O? I'd love to know because ~10-15MB/s digest calculation is incredibly painful on a 7TB backup copy job.

[Gostev]

Assuming you have 7.0 R2 and multi-core CPU on target WAN accelerator, than the bottleneck is most likely your backup storage speed. Prior to R2, CPU was typically the common bottleneck, as the digest calculation process only used a single core.

[tfloor]

Do you have a proxy on the target side? Is it being properly used? Sounds like probably not. You might have to manually select the proxy.

Maybe i have the same problem. Using Backup Copy jobs. The calculating disk digest takes a long time.
Disk (750GB) on 47% and running already 9:20 hours.

Is that normal?
Here a summary:

8-12-2013 23:16:38 :: Changed extents are not available.
8-12-2013 23:16:38 :: Digests are missing on source, loading them from target
8-12-2013 23:25:10 :: 41,4 MB transferred over network, 65,6 MB obtained from WAN Accelerator cache
8-12-2013 23:25:29 :: Hard disk 2 (750,0 GB)
8-12-2013 23:25:29 :: Calculating disk 2 digests

Is a backup copy job also using the proxy on the other side. I can't configure that in my opinion. Only wan acc.

Any advice here is appreciated, because the copy jobs for all my vm's are taking longer than the interval of the normal backup jobs. So that results in not completed cycles.

[foggy]

Tristan, I've re-merged your post to a more appropriate discussion, since the original thread you've posted into talks about replication, not backup copy jobs (hence, question about target proxy). Explanation given above should justify the timing for you.

[tfloor]

Thanks for putting it in the right thread.

"After mapping backup copy job to existing backup, digests calculation is required to understand which blocks should be transferred to the target. Currently, digests calculation is indeed not very fast and can take considerable time, depending on the backup size (roughly 1 minute is required to calculate digests for 1GB of data, which results in ~17 hours to calculate digests for 1TB disk, for example) + additional time is required to transfer digests to the source over the slow network."......

I hope this will not be needed each run?, is it only the first run after seeding?

And i still wondering what's the bottleneck exactly.
9-12-2013 10:16:39 :: Hard disk 2 (750,0 GB) 27,0 GB read at 5MB/s Zzzzzz.....
The hardware is all physical Multicore 4 or 6 cores. And the WAN is c.a 400MBIT

[foggy]

I hope this will not be needed each run?, is it only the first run after seeding?

Correct, digests need to be calculated after job mapping.

And i still wondering what's the bottleneck exactly.
9-12-2013 10:16:39 :: Hard disk 2 (750,0 GB) 27,0 GB read at 5MB/s Zzzzzz.....
The hardware is all physical Multicore 4 or 6 cores. And the WAN is c.a 400MBIT

Not sure you need WAN acceleration on this kind of link at all. It was specifically designed for limited bandwidth (100Mbit or less) and could affect performance on faster links.

Bottleneck should be displayed in the job session log.

[tfloor]

Sorry, WAN is 100 MBIT. I'm definitly trying to get a gigabit line here to the offsite location, but that will take some months.
The bottleneck at the moment: Target WAN

Is it also possible to remove the link between the backup and the backup copy job. Because the copy job will not finish before the real backup copy is starting again. I want to have the backup copy job complete for once

[foggy]

This is expected, since WAN accelerators are effectively trading disk I/O for WAN bandwidth savings.

[tfloor]

This is expected, since WAN accelerators are effectively trading disk I/O for WAN bandwidth savings.

Is it also possible to remove the link between the backup and the backup copy job. Because the copy job will not finish before the real backup copy is starting again. I want to have the backup copy job complete for once (before new restore points are coming). Or do i have to disable the backup job temporarily

[foggy]

What backup mode are you using? If it is a reversed incremental mode, then you can disable the backup job to allow the copy job to complete. This is not required with forward incremental.

[tfloor]

What backup mode are you using? If it is a reversed incremental mode, then you can disable the backup job to allow the copy job to complete. This is not required with forward incremental.

I'm using formward incremental.
But in my idea, if the backup copy job is still running and the normal backup copy job is started, then at the end the backup copy job will re-rerun trying to get the latest restore point. It is triggered after backup job is finished isn't it?

[foggy]

Not exactly. Backup copy job is continuous and copies the latest VM state from the backups available at the moment when new synchronization interval starts (or waits for the new restore point to be created, if linked to the backup job and the latest state is already copied). Only once restore point is copied during each sync interval.

[tfloor]

Not exactly. Backup copy job is continuous and copies the latest VM state from the backups available at the moment when new synchronization interval starts (or waits for the new restore point to be created, if linked to the backup job and the latest state is already copied). Only once restore point is copied during each sync interval.

My interval is 7 days, but for so far i haven't seen a complete one yet.
But i will keep an eye on it and take a look next morning how it goes.

[nsimao]

After mapping backup copy job to existing backup, digests calculation is required to understand which blocks should be transferred to the target. Currently, digests calculation is indeed not very fast and can take considerable time, depending on the backup size (roughly 1 minute is required to calculate digests for 1GB of data, which results in ~17 hours to calculate digests for 1TB disk, for example) + additional time is required to transfer digests to the source over the slow network.

We have a potentially large customer in Australia (lots of branch sites) who is finding that his digests are taking more than a week for 800GB of data, which after looking closer is due to his target storage performance. They have a QNAP NAS (TS-1679U-RP) with Seagate enterprise SATA drives (ST33000650NS) and it's really struggling to do the digest. Is there any way around the digest process?

What backup mode are you using? If it is a reversed incremental mode, then you can disable the backup job to allow the copy job to complete. This is not required with forward incremental.

I have asked them to change over to forward incremental with Active fulls monthly to ensure were giving the digest enough time to complete.

I'm also looking at alternative options, as I can see this being an issue moving forward for them with a large number of branch sites. To avoid the digest, a remote offsite backup job from DR could work, but they would lose all the benefits that backup copy offers. Telling them to upgrade their target storage is most likely what will need to happen to accomodate for all sites.

[foggy]

Nelson, actually digests need to be calculated during the initial job run only, right after job mapping, so its a one-time operation that should not affect duration of backup copy further. However, low-end storage indeed is not a good choice in terms of performance if we're talking about such an I/O intensive operation as transform (which is performed during each backup copy synchronization cycle).

[sfens]

foggy,

Is there a way to "seed the digests"? If digests need to be copied across a WAN connection, can I just trick Veeam into thinking that it's performing the initial backup copy job run at DR while in fact running it locally?

Here's what I'm thinking:

Production: 192.168.1.0
DR: 192.168.2.0
Connected via VPN.

Disable VPN connection
Create temporary subnet at production site - 192.168.2.100
Create backup repository 192.168.2.100
Create the backup copy job
Run backup copy job
Remove temporary subnet 192.168.2.100 at production site
Move backup repository (Synology in my case) to DR site
Enable VPN
Run backup copy job

Using this method, there should be no need to do any mapping, and the initial backup copy should go pretty quickly.

Would I also need to create the target wan accelerator in this 192.168.2.x temporary subnet and then move it to DR? The target wan accelerator could either be a VM that I move or a workstation or laptop.

[Vitaliy S.]

Hi Seth,

Yes, you can run the backup copy job locally and then transfer all the content to the target site with the components you've used. Target WAN accelerator should also be moved, since it contains global cache for your backup copy jobs.

Be aware that digest calculation does not require much traffic to be transferred over the WAN link, it just needs to read the entire data and then compare the VM virtual disk blocks.

Thanks!

[sfens]

Is it accurate to say that I would not see much of a speed increase by moving the target repository and wan accelerator locally? Would I need to use my elaborate workaround to avoid mapping the backup? Is the digest calculation necessary the first time using WAN Acceleration, or any time a backup is mapped?

[Vitaliy S.]

If you run backup copy job locally using WAN accelerators locally and then transferring all the components, including backup repository, then backup mapping will not be needed, but digest creation will still be done.

See this online help center web page for more info on the digest calculation > http://helpcenter.veeam.com/backup/70/v ... sub=digest