Changing backup copy transfer modes

[y1008946]

Hi, I have a backup copy job which I wanted to see how it would perform in direct mode.

The job ran in the same time or slightly longer, so I changed it back to wan accelerator mode.

The job is now running but it is doing more than usual

Digests are missing on source, loading them from target

When I read the guides it said there was no problem switching between modes, have I got it wrong?

Thanks

[Vitaliy S.]

Hello,

There are no issues in switching copy modes for the backup copy job. When you switch to WAN accelerator enabled mode, your job will start to populate cache for optimized data transfer and calculate digests on source. BTW, what bandwidth do you have between your sites? And how much data do you transfer with backup copy job?

Thank you!

[y1008946]

Thanks Vitaly, when I seeded the job some time ago, the first run after took 4 days... after the first run it worked fine in WAN accelerator mode taking about 16 hours a day. The bottleneck was the source WAN and someone in support mentioned that we should try running in direct mode and that it wouldn't cause any problems and I could switch back after.

Direct mode took the same time so yesterday, so I switched it back to WAN accelerator mode so It took up less bandwidth. It seems to be running the same as it did the first run after seeding.

Our upload speed is 5Mbps and the download speed out our offsite is 8mbps

Roughly just over 30GB of data was transferred over the network when run in direct mode.

Many Thanks

[foggy]

Everything looks expected in your case. You can switch between direct and WAN accelerated mode, but on the first pass after turning WAN acceleration, the target global cache will be rebuilt (like it is after seeding).

[y1008946]

ah ok, it wasn't explained like that to me by support, I will just have to wait for the next 80 hours!

Is there any improvements on the speed of this process in v8?

Many Thanks

[foggy]

Yes, there will some improvements in this area.

[y1008946]

Great, thanks.

Last question, the interval time is set at 1 day and its going to take a lot longer. Its currently halfway through processing a VM.

When it has finished processing the VM should I change the interval time to say 7 days and then press sync now?

Thanks

[veremin]

If you want to make backup copy job create a restore point at the target location once a week, then, you can set the said schedule. Thanks.

[y1008946]

Hi, after the VM had finished I changed the interval time and pressed sync now.

When it got to the old interval time though it expired.

At the time it expired it was calculating disk digests.

When the copy interval started again it hasn't calculated the disks, it just says 'Changed extents are not available' and it seems to be transferring a large amount of data.

It has read 22GB in 4 hours and sent 8GB of data, but it is saying it is only 12% through processing the VM.

Why is trying to send so much data?

Many thanks

[veremin]

First of all, I wouldn't be worried about mentioned percents, as currently they are calculated not that precisely. In other words, indicated 12% doesn't necessarily mean that backup console would send 66,7 GB (8*100/12). Also, after WAN Accelerators have been enabled, some portion of information will be stored in cache and it has to be filled with something. Therefore, the first cycle following WAN Accelerators' turning on might be marked with increased amount of sent data. Thanks.

[y1008946]

Thanks for your reply, Its at 48% at the moment after 34 hours.

It has read 225.9 GB and transferred 56 GB.

In WAN accelerator mode usually it transfers 3 GB for the 10 VMs.

In direct mode it transferred 40 GB for the 10 VMs.

I'm not sure why it is transferring so much, the other VMs ran fine in the job, and this particular VM was doing the same (calculating digests), but when it failed calculating the digests it seems like it is wanting to transfer the whole thing rather than the changes? (sorry for my basic explanation!)

I don't think there is a lot I can do other than to wait another 36 hours for it to finish processing the VM and hope it has no problems with the others remaining in the job.

Many thanks

[veremin]

For me it seems that after WAN Accelerators have been re-enabled, backup console is trying to populate cache with data. As the result, the amount of sent data has been increased, and the whole process is taking longer than usual. Let's see what happens, when initial cycle is finished. Thanks.

[y1008946]

Ok, I'll monitor it the next few days and report back!

Thanks

[y1008946]

Hi, after 65 hours it has transferred 100GB of a 400GB disk.

It has just finished processing disk 1 and it has calculated the disk 2 digests.

I think that the cause of it needing to transfer so much is that the interval expired while calculating the disk digests (it was at 5% of calculating disk digests)

Is there anyway this could be tested in the lab?

Thanks v much

[veremin]

As I've said, it's expected to have increased amount of sent data after WAN Accelerators re-enabling. However, let us know if the amount of transferred data keeps being significant during subsequent runs. Thanks.

[y1008946]

Thanks, I do agree that it does take longer and processes more data after enabling the WAN accelerators. The first run after seeding processed 2.2TB, whereas it was processing around 150GB on subsequent runs The amount transferred stayed around the same 5GB.

A couple of weeks ago I seeded this job, after the first run the stats were:

Processed: 2.2TB
Read: 1.5TB
Transferred: 5GB

For the disk of the VM which we have been discussing, it read 400GB and transferred 1GB


After switching back to WAN Accelerator Mode and after the interval expired during calculating disk digests, the stats will be:

Processed: 2.2TB
Read: 1.5TB
Transferred: 105GB

For the disk of the VM which we have been discussing, it read 400GB and transferred 100GB


All of the other VMs and the second disk of the VM we have been discussing, have run as expected and very similar to the first run after seeding. Processing close to the size of the VM but transferring small amounts of data.

I am not a Veeam expert, I agree that it should have processed that much data, but don't agree that it would have transferred that much data if the interval hadn't expired during digest calculations.


Thanks

[veremin]

A couple of weeks ago I seeded this job, after the first run the stats were:

How the exact procedure looked like? You performed a local direct backup copy job, then, copied resulting data to target location, mapped the backup copy job to it and enabled WAN Accelerator role? Or you performed a local WAN Accelerated backup copy job, then, copied both resulting data and global cache to target location and enabled WAN Accelerator role? Thanks.

[y1008946]

I performed a local direct backup copy job, then, copied resulting data to target location, mapped the backup copy job to it and enabled WAN Accelerator role.

[veremin]

Got it. Then, it seams that because digest calculation had been interrupted by interval expiration, the backup copy job wasn't able to use digests during the subsequent run and target cache became the primary source of deduplication for it. Depending on circumstances, this might indeed result in increased amount of transferred data. Thanks.

[y1008946]

ok thanks, if that situation happens again is there anything that could be done?

Would there be any reset which could be performed to get it back to calculate the digests.

The job is running again smoothly but it would just be handy to know.

Thanks v much

[veremin]

ok thanks, if that situation happens again is there anything that could be done?

I doubt that there is a workaround for that at the moment. However, we will take a closer look at that and see whether it can be fixed in the upcoming release. Thanks.

[y1008946]

Thanks very much, let me know if you want to see any logs

[veremin]

Can you upload your logs to some public available source and PM me a link? The logs we're interested in include: jobs, agents, WAN Accelerators. Thanks.

[y1008946]

Hi, thanks for helping with this, I'm sorry I didn't reply, I had a week off from work and when I got chance to upload the logs it was too late.

This problem has happened again, but not through changing the modes. At the moment a VM is 3 complete after 3 hours and it has send over 10gb

There has been some problems with our ISP and the connection has dropped 3 times for a few minutes over the past 3 days. And unlikely it has been while processing the same VM in the backup copy job.

The first time, the connection had dropped for a few hours, the job queued but the interval expired after the vm had been processing for about 4 hours and was 5% from completing

The next day the vm had been processing for about 6 hours and was at 93% when the connection dropped. The interval expired before the job could get back to the VM

Yesterday the VM was calculating digests when the connection dropped for a short time. It calculated them again, but the job interval expired as it was still processing (it had been running for over 9 hours)

I will upload the job log and pm it to you. If you want to do a webex that is ok.

The problem I find is that it can be difficult to recover from connection drop outs. If the connection drops for a sort period, Veeam seems to forget where it last was and then it is difficult to catch up

Many thanks