Calculating Digests taking forever all of a sudden...

[jbarrow.viracoribt]

What could be going on?

My jobs ran fine for the last few days. I paused them to set up a few backup jobs. Now when I resumed them the calculating digests step is taking forever, like 60 minutes for a 50GB hard drive, this single job would take days at this rate. What could I be missing?

FYI, my Prod site and DR site are on the same FC network. I was running my jobs through a single proxy at the Prod site which is using Direct SAN access.

[Vitaliy S.]

Hi Jonathan,

Where is replica metadata stored for your replication jobs? What VMs do you have in the job? Exchange, SQL servers?

Thanks!

[jbarrow.viracoribt]

The replica metadata was on my Proxy at the DR site from previous jobs. I tried changing the job so that the metadata is kept on the PROD PROXY but it does not seem to make a difference.

Here is my layout, for replication jobs, the DR PROXY wasn't configured as part of the job before since both proxies are on the same FC network.

PROD SAN > fibre channel > PROD PROXY > ethernet > DR PROXY > fibre channel > DR SAN > E: Drive (RDM)

[jbarrow.viracoribt]

So I changed my replication job to the suggested config using a physical proxy at both sites (which was slower in backup times in my initial testing a week ago).

Now when I see the calculating digests task, I see the DR server with a good amount of ethernet traffic coming from the DR site ESX hosts where the replicas are held. It sounds as if it's retrieving data from the existing replicas during this process, but why is that going over ethernet and not fibre channel?

Here's the traffic going from one of my hosts at my DR site to my physical proxy at my DR site.



I'm now at 10 minutes into the first calculating digests task and it's taken 10 minutes to get 50% through a 40GB hard drive. That can't be right, can it?

[kjc3303]

It sounds as if it's retrieving data from the existing replicas during this process, but why is that going over ethernet and not fibre channel?

You DR proxy is using NBD (Network Mode) so will be retrieving data from you DR San over Ethernet

[jbarrow.viracoribt]

You DR proxy is using NBD (Network Mode) so will be retrieving data from you DR San over Ethernet

But my DR Proxy is configured for SAN only, why would it be pulling data over Ethernet?

[kjc3303]

Well I was going to suggest unticking the failover to network mode box, but I see that you already have. If there was an issue with your SAN access it should fail not go to nbd mode, I have seen and tested this many time on my original setup when I was having issues with SAN access.

One for the veeam guys I think

[tsightler]

Have you always been using a physical proxy, even when you only had one proxy? That's probably why the calculating digests was so slow to start with, because the target side was using NBD mode which pulls data over the network. I believe you had another thread on the forums where I mentioned that the only way to do FC to FC replication is to use a virtual proxy with Hotadd and present the physical LUNs to the same ESXi hosts (I apologize in advance if I'm confusing you with another forum user).

I'm also interested in why the system is performing a "calculating digests" at all. The "calculating digests" step is normally only performed if Veeam believes that the replica state is not consistent with the current jobs metadata. You stated that the jobs were running, and you only paused them to "setup some backup jobs", this shouldn't trigger the "calculating digests" step. Note that once this step finishes, the jobs will run normally and will not need to perform this step during future job runs unless other changes are made to the job configs or the VMs.

[jbarrow.viracoribt]

Have you always been using a physical proxy, even when you only had one proxy? That's probably why the calculating digests was so slow to start with, because the target side was using NBD mode which pulls data over the network. I believe you had another thread on the forums where I mentioned that the only way to do FC to FC replication is to use a virtual proxy with Hotadd and present the physical LUNs to the same ESXi hosts (I apologize in advance if I'm confusing you with another forum user).

I'm also interested in why the system is performing a "calculating digests" at all. The "calculating digests" step is normally only performed if Veeam believes that the replica state is not consistent with the current jobs metadata. You stated that the jobs were running, and you only paused them to "setup some backup jobs", this shouldn't trigger the "calculating digests" step. Note that once this step finishes, the jobs will run normally and will not need to perform this step during future job runs unless other changes are made to the job configs or the VMs.

I think that may have been my other post but I was confused as to what you were suggesting. Interesting about the FC to FC communications. I guess I need to read up on hotadd because that sounds faster if we can get off the eternet network.

[tsightler]

Unfortunately hotadd has it's own potential drawbacks. Specifically, with some storage systems and configurations it causes a significant increase in the amount of I/O performed on the target storage system (I have yet to be able to determine the exact configurations that cause this, so only by testing can you know if your setup is impacted). This can actually cause the hotadd to be slower than using network mode because the target storage system becomes the bottleneck.

Your current configuration should offer good combination of performance and network bandwidth savings. Yes, calculating digests won't be as fast as it could be, but this should not be a common operations and is not part of a normal job cycle. You normal replications will be incremental, and with compression and dedupe will likely send only minimal data over the network.