Host-based backup of VMware vSphere VMs.
Post Reply
dvdrbc
Influencer
Posts: 12
Liked: 3 times
Joined: Jun 17, 2015 2:43 am
Full Name: David Rubic
Contact:

Calculating digests issue

Post by dvdrbc »

Hi Guys,

I have a Linux VM that I replicate hourly. 2 nights in a row now the replication has taken 4 hours+ to replicate locally, it gets held up for a very long time on calculating digests of the hard drive.

I am replicating locally. My B&R Server/Proxy is in my head office (10.0.5.0 VLAN) and I replicate to the DR ESXi hosts in the building across the road connected via fibre on a different subnet (10.0.6.0 VLAN)

All the other VM's I replicate hourly function fine, yet this one has just started giving me troubles and I can't afford to be waiting 3 hours at the minimum. We are using V9 with the latest patch applied.

Can anyone offer any insight?

Image
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Calculating digests issue

Post by foggy »

David, haven't you recently re-sized any of the source VM disks?
dvdrbc
Influencer
Posts: 12
Liked: 3 times
Joined: Jun 17, 2015 2:43 am
Full Name: David Rubic
Contact:

Re: Calculating digests issue

Post by dvdrbc »

Hi Jim, no we haven't. I've noticed another 2 VM's having this issue also now
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Calculating digests issue

Post by Vitaliy S. »

David, maybe you have re-attached these disks recently? The reason for digest recalculation can be any modification of the source disk, which seems to happen in your case. Let us know what the subsequent job run will be, digest recalculation should not be happening.

On a side note, If you want to track changes for the virtual infrastructure configuration, please take a look at these Veeam ONE reports > VMware Configuration Tracking Report Pack
dvdrbc
Influencer
Posts: 12
Liked: 3 times
Joined: Jun 17, 2015 2:43 am
Full Name: David Rubic
Contact:

Re: Calculating digests issue

Post by dvdrbc »

Nope, no disk reattaching, there has been no changes in our virtual environment for at least 3-5 months, and prior to upgrading to V9 I wasn't having this issue (could be completely unrelated though, so im not blaming the upgrade)

Also, the issue is happening on multiple VM's every single night some taking 45 mins minimum to calculate, some taking up to 5 hours

I'll take a look at the link
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Calculating digests issue

Post by Vitaliy S. »

If nothing pops up after quick investigation, then ask our support team to look through backup job logs. There should contain some relevant information as well.
readie
Expert
Posts: 158
Liked: 30 times
Joined: Dec 05, 2010 9:29 am
Full Name: Bob Eadie
Contact:

Re: Calculating digests issue

Post by readie »

After moving to a new VC (and running the migration tool from support) and remapping our VMs, I knew to expect a 'calculating disk digests', but these seem painfully slow, even to the extent that I am wondering whether a new full replication might be quicker.
We have two VMs which are each about 3 TB, made up of 6 x 512GB VHDs. Each of these 512GB is taking between 4 and 7 hours to calculate disk digests - which means the whole 3TB is going to take at least 24 hours . . .
Is this expected, or should I raise a support call?
Bob Eadie
Computer Manager at Bedford School, UK (since 1999).
Veeam user since 2009.
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Calculating digests issue

Post by Shestakov »

Hello Bob,
The migration tool doesn`t help with replication jobs so digests need to be recalculated what indeed takes significant amount of time.
So yes, that`s an expected behavior.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Calculating digests issue

Post by foggy »

Digests calculation depends on the data size. I remember previously it took roughly 1 minute to calculate digests for 1GB of data, though this was improved in one of the latest versions. But still, considering the size of your VMs, I'd say you're seeing an expected performance.
readie
Expert
Posts: 158
Liked: 30 times
Joined: Dec 05, 2010 9:29 am
Full Name: Bob Eadie
Contact:

Re: Calculating digests issue

Post by readie »

OK thanks both, I will be patient. I might do an experiment to find out if starting again with a fresh full might be quicker.
Bob Eadie
Computer Manager at Bedford School, UK (since 1999).
Veeam user since 2009.
Shestakov
Veteran
Posts: 7328
Liked: 781 times
Joined: May 21, 2014 11:03 am
Full Name: Nikita Shestakov
Location: Prague
Contact:

Re: Calculating digests issue

Post by Shestakov »

It would be quicker if you created a new job since recalculation takes more time in most cases.
acire
Novice
Posts: 3
Liked: 1 time
Joined: Aug 20, 2018 8:50 am
Full Name: Aidan IRE
Contact:

Re: Calculating digests issue

Post by acire »

Hi just checking did you ever resolve this issue? I have had issues with this problem and i think you are correct it is related to an upgrade to v9. I have 2 companies pretty much same setup same number same type of vms (mix of linux and windows)
Each site has on site reps, on site backups and offsite reps.
The site with v8 rarely gives any trouble. The V9 site very frequently especially for linux is calculating disgests and so the jobs take longer, on top of this the first rep of the on site the next morning has "cannot find dirty block infomation in previous restore point, cbt will not be used", it then calulcated digests again, so a job that should take 5 to 10 min, is 2 to 3 hours. I have logged another call with veeam but while researching issue came across your post

Thks
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Calculating digests issue

Post by foggy »

Hi Aidan, we cannot check the solution to David's issue as he didn't share the case ID, so please continue investigating with your support engineer.
bdufour
Expert
Posts: 206
Liked: 41 times
Joined: Nov 01, 2017 8:52 pm
Full Name: blake dufour
Contact:

Re: Calculating digests issue

Post by bdufour »

what kind of disk do u have provisioned?
acire
Novice
Posts: 3
Liked: 1 time
Joined: Aug 20, 2018 8:50 am
Full Name: Aidan IRE
Contact:

Re: Calculating digests issue

Post by acire »

4 disks, thick provisioned lazy zeroed. There is maintenance that goes on during the night, but i disabled the off site rep, once off site disabled, the on site rep and backup run fine. Windows vms no problem with onsite/offsite/backup.
dvdrbc
Influencer
Posts: 12
Liked: 3 times
Joined: Jun 17, 2015 2:43 am
Full Name: David Rubic
Contact:

Re: Calculating digests issue

Post by dvdrbc » 1 person likes this post

foggy wrote:Hi Aidan, we cannot check the solution to David's issue as he didn't share the case ID, so please continue investigating with your support engineer.
Hi Foggy, I believe my case ID was Case # 01891991. We resolved this issue with our engineer.
crackocain
Service Provider
Posts: 240
Liked: 27 times
Joined: Dec 14, 2015 8:20 pm
Full Name: Mehmet Istanbullu
Location: Türkiye
Contact:

Re: Calculating digests issue

Post by crackocain »

Hi everyone

I'm experienced this issue before. I start replicate one VM. Incremental Replication finished successfully about 5 minutes, then i started again. Nothing changed source VM. Then calculating digest and this process take a about 1 hour.

I think if replication cycles too short Veeam or Vmware snapshot API detect changes. I don't want open the case :)
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Calculating digests issue

Post by foggy »

dvdrbc wrote:Hi Foggy, I believe my case ID was Case # 01891991. We resolved this issue with our engineer.
As far as I can see from the case notes, only completely recreating the jobs helped to stop digests calculation.
acire
Novice
Posts: 3
Liked: 1 time
Joined: Aug 20, 2018 8:50 am
Full Name: Aidan IRE
Contact:

Re: Calculating digests issue

Post by acire » 1 person likes this post

Hi just to update on my case 03151774, when i gave the previous case number with this thread assigned engineer said

"Root cause for that case was that vmREFs were identical for 2 different target hosts. Veeam creates a temporal folder for each vm that should look like *repository*\*jobname*\*vmREF* and place replica metadata here. If vmREF is the same for 2 hosts then first metadata will get overwritten at the start of the second one."

So to validate id did below for on site and dr rep vmware host, after going to

1. https://*vcenter_name*/mob , if you're using standalone hosts then it will be ESXi host ip
2. Properites -> content
3. RootFolder
4. Datastore
5. childEntity
6. Datastore (select datastore where VM lies)
7. Check vmREF for VMs at the bottom

The vmref for the replicated vm on each host was the same (1), which caused the digest issue.

On the DR rep job, changed the medadata directory for the job. Since this change all backup/reps working perfectly.

Aidan
Post Reply

Who is online

Users browsing this forum: karsten123, Semrush [Bot] and 81 guests