Comprehensive data protection for all workloads
Post Reply
jbarrow.viracoribt
Expert
Posts: 184
Liked: 18 times
Joined: Feb 15, 2013 9:31 pm
Full Name: Jonathan Barrow
Contact:

Calculating Digests taking forever all of a sudden...

Post by jbarrow.viracoribt » Feb 21, 2013 4:53 am

What could be going on?

My jobs ran fine for the last few days. I paused them to set up a few backup jobs. Now when I resumed them the calculating digests step is taking forever, like 60 minutes for a 50GB hard drive, this single job would take days at this rate. What could I be missing?

FYI, my Prod site and DR site are on the same FC network. I was running my jobs through a single proxy at the Prod site which is using Direct SAN access.

Vitaliy S.
Product Manager
Posts: 22435
Liked: 1442 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Calculating Digests taking forever all of a sudden...

Post by Vitaliy S. » Feb 21, 2013 8:22 am

Hi Jonathan,

Where is replica metadata stored for your replication jobs? What VMs do you have in the job? Exchange, SQL servers?

Thanks!

jbarrow.viracoribt
Expert
Posts: 184
Liked: 18 times
Joined: Feb 15, 2013 9:31 pm
Full Name: Jonathan Barrow
Contact:

Re: Calculating Digests taking forever all of a sudden...

Post by jbarrow.viracoribt » Feb 21, 2013 1:40 pm

The replica metadata was on my Proxy at the DR site from previous jobs. I tried changing the job so that the metadata is kept on the PROD PROXY but it does not seem to make a difference.

Here is my layout, for replication jobs, the DR PROXY wasn't configured as part of the job before since both proxies are on the same FC network.

PROD SAN > fibre channel > PROD PROXY > ethernet > DR PROXY > fibre channel > DR SAN > E: Drive (RDM)

jbarrow.viracoribt
Expert
Posts: 184
Liked: 18 times
Joined: Feb 15, 2013 9:31 pm
Full Name: Jonathan Barrow
Contact:

Re: Calculating Digests taking forever all of a sudden...

Post by jbarrow.viracoribt » Feb 21, 2013 4:56 pm

So I changed my replication job to the suggested config using a physical proxy at both sites (which was slower in backup times in my initial testing a week ago).

Now when I see the calculating digests task, I see the DR server with a good amount of ethernet traffic coming from the DR site ESX hosts where the replicas are held. It sounds as if it's retrieving data from the existing replicas during this process, but why is that going over ethernet and not fibre channel?

Here's the traffic going from one of my hosts at my DR site to my physical proxy at my DR site.
Image
Image

I'm now at 10 minutes into the first calculating digests task and it's taken 10 minutes to get 50% through a 40GB hard drive. That can't be right, can it?

kjc3303
Expert
Posts: 167
Liked: 24 times
Joined: Dec 02, 2010 12:25 pm
Full Name: Kevin Clarke
Location: Cheshire
Contact:

Re: Calculating Digests taking forever all of a sudden...

Post by kjc3303 » Feb 21, 2013 5:13 pm

It sounds as if it's retrieving data from the existing replicas during this process, but why is that going over ethernet and not fibre channel?
You DR proxy is using NBD (Network Mode) so will be retrieving data from you DR San over Ethernet

jbarrow.viracoribt
Expert
Posts: 184
Liked: 18 times
Joined: Feb 15, 2013 9:31 pm
Full Name: Jonathan Barrow
Contact:

Re: Calculating Digests taking forever all of a sudden...

Post by jbarrow.viracoribt » Feb 21, 2013 5:19 pm

kjc3303 wrote: You DR proxy is using NBD (Network Mode) so will be retrieving data from you DR San over Ethernet
But my DR Proxy is configured for SAN only, why would it be pulling data over Ethernet?

Image

kjc3303
Expert
Posts: 167
Liked: 24 times
Joined: Dec 02, 2010 12:25 pm
Full Name: Kevin Clarke
Location: Cheshire
Contact:

Re: Calculating Digests taking forever all of a sudden...

Post by kjc3303 » Feb 21, 2013 5:23 pm

Well I was going to suggest unticking the failover to network mode box, but I see that you already have. If there was an issue with your SAN access it should fail not go to nbd mode, I have seen and tested this many time on my original setup when I was having issues with SAN access.

One for the veeam guys I think

tsightler
VP, Product Management
Posts: 5306
Liked: 2160 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Calculating Digests taking forever all of a sudden...

Post by tsightler » Feb 21, 2013 5:27 pm

Have you always been using a physical proxy, even when you only had one proxy? That's probably why the calculating digests was so slow to start with, because the target side was using NBD mode which pulls data over the network. I believe you had another thread on the forums where I mentioned that the only way to do FC to FC replication is to use a virtual proxy with Hotadd and present the physical LUNs to the same ESXi hosts (I apologize in advance if I'm confusing you with another forum user).

I'm also interested in why the system is performing a "calculating digests" at all. The "calculating digests" step is normally only performed if Veeam believes that the replica state is not consistent with the current jobs metadata. You stated that the jobs were running, and you only paused them to "setup some backup jobs", this shouldn't trigger the "calculating digests" step. Note that once this step finishes, the jobs will run normally and will not need to perform this step during future job runs unless other changes are made to the job configs or the VMs.

jbarrow.viracoribt
Expert
Posts: 184
Liked: 18 times
Joined: Feb 15, 2013 9:31 pm
Full Name: Jonathan Barrow
Contact:

Re: Calculating Digests taking forever all of a sudden...

Post by jbarrow.viracoribt » Feb 21, 2013 5:30 pm

tsightler wrote:Have you always been using a physical proxy, even when you only had one proxy? That's probably why the calculating digests was so slow to start with, because the target side was using NBD mode which pulls data over the network. I believe you had another thread on the forums where I mentioned that the only way to do FC to FC replication is to use a virtual proxy with Hotadd and present the physical LUNs to the same ESXi hosts (I apologize in advance if I'm confusing you with another forum user).

I'm also interested in why the system is performing a "calculating digests" at all. The "calculating digests" step is normally only performed if Veeam believes that the replica state is not consistent with the current jobs metadata. You stated that the jobs were running, and you only paused them to "setup some backup jobs", this shouldn't trigger the "calculating digests" step. Note that once this step finishes, the jobs will run normally and will not need to perform this step during future job runs unless other changes are made to the job configs or the VMs.
I think that may have been my other post but I was confused as to what you were suggesting. Interesting about the FC to FC communications. I guess I need to read up on hotadd because that sounds faster if we can get off the eternet network.

tsightler
VP, Product Management
Posts: 5306
Liked: 2160 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Calculating Digests taking forever all of a sudden...

Post by tsightler » Feb 21, 2013 5:37 pm

Unfortunately hotadd has it's own potential drawbacks. Specifically, with some storage systems and configurations it causes a significant increase in the amount of I/O performed on the target storage system (I have yet to be able to determine the exact configurations that cause this, so only by testing can you know if your setup is impacted). This can actually cause the hotadd to be slower than using network mode because the target storage system becomes the bottleneck.

Your current configuration should offer good combination of performance and network bandwidth savings. Yes, calculating digests won't be as fast as it could be, but this should not be a common operations and is not part of a normal job cycle. You normal replications will be incremental, and with compression and dedupe will likely send only minimal data over the network.

Post Reply

Who is online

Users browsing this forum: No registered users and 36 guests