Host-based backup of VMware vSphere VMs.
Post Reply
JosueM
Expert
Posts: 187
Liked: 12 times
Joined: Sep 01, 2012 2:53 pm
Full Name: Josue Maldonado
Contact:

Processing large vms very slow.

Post by JosueM »

Good day everyone,

WE have some vms that have grown up for us to be a large servers, and it seems each time veeam process jobs for those vms things gets really anonoying. I reported this kinda behavior to support but they claim the problems is vmware not veeam.

So I post here again hoping someoneelse migth help us to fix this.

The attached caption is for large vm that was increased the size of one of its hard drives, after the change the replication job took 29.5 hours to complete and it seems it just moved 102.2 GB this is a local replica with destination host in the same switch 1gbps as production host. Have anyone had this kinda behavior before and got it fixed? any help is really appreciated.

Thanks in advance

Image
HannesK
Product Manager
Posts: 14844
Liked: 3086 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: Processing large vms very slow.

Post by HannesK »

Hello,
I reported this kinda behavior to support but they claim the problems is vmware not veeam.
could you please post the case number?

The screenshot looks like that change block tracking (CBT) is not working. The increased disk size you mention seems to be the reason.
1gbps as production host
doing the math, everything looks expected and fine for me. Looks like a hardware issue and not a VMware or Veeam problem :-)

Best regards,
Hannes
JosueM
Expert
Posts: 187
Liked: 12 times
Joined: Sep 01, 2012 2:53 pm
Full Name: Josue Maldonado
Contact:

Re: Processing large vms very slow.

Post by JosueM »

Hello Hannesk,

Case #02576181 was related to something similar, in that case we only wanted to change the ip address of the destination host and after doing that the replication jobs just hanged out cuz started to calculate all the disks. so our solution was set the ip address back to destination host and make a new full replica job instead.

For this case only run after making the disk larger is this one , so I guess next run tomorrow should run "normally" and not take so long.

What do you mean change block tracking (CBT) is not working.?
foggy
Veeam Software
Posts: 21139
Liked: 2141 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Processing large vms very slow.

Post by foggy »

JosueM wrote: Feb 28, 2022 3:13 pm The attached caption is for large vm that was increased the size of one of its hard drives
How was this disk increase performed? If they resized the vmdk, CBT reset with a subsequent full scan is expected.
JosueM
Expert
Posts: 187
Liked: 12 times
Joined: Sep 01, 2012 2:53 pm
Full Name: Josue Maldonado
Contact:

Re: Processing large vms very slow.

Post by JosueM »

hello Foggy,

The disk increased was done first in the vm settings and then expanding the disk from within windows.

Yesterday incremental backup took about 8 hours so it seems something got broken within the replication job. it finish with a warning sign but no additional message display

Image Image
foggy
Veeam Software
Posts: 21139
Liked: 2141 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Processing large vms very slow.

Post by foggy »

Judging by the session log, CBT did work this time, hence the shorter job duration. I believe you will see the warning in the session log if you scroll up on the right screenshot.
JosueM
Expert
Posts: 187
Liked: 12 times
Joined: Sep 01, 2012 2:53 pm
Full Name: Josue Maldonado
Contact:

Re: Processing large vms very slow.

Post by JosueM »

but prior disk size increase, the incremental jobs was taking about half an hour

Image
foggy
Veeam Software
Posts: 21139
Liked: 2141 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Processing large vms very slow.

Post by foggy »

I'd maybe let the job run one more time as CBT has just been retrieved after reset. Also look for the warning message and compare bottleneck stats with those before the disk resize.
JosueM
Expert
Posts: 187
Liked: 12 times
Joined: Sep 01, 2012 2:53 pm
Full Name: Josue Maldonado
Contact:

Re: Processing large vms very slow.

Post by JosueM »

Hello Foggy, yesterday incremental run took 3.5 hours, so it seems we will have to erase replica with all the restore point and do it again from scratch again as always had happended when a minor change occurs.

its kinda frustrating and hard to understand cuz is not able to properly handle these changes as should be.

Image
foggy
Veeam Software
Posts: 21139
Liked: 2141 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Processing large vms very slow.

Post by foggy »

Please check the target proxy settings - it is currently using network instead of hotadd, might be the case.
JosueM
Expert
Posts: 187
Liked: 12 times
Joined: Sep 01, 2012 2:53 pm
Full Name: Josue Maldonado
Contact:

Re: Processing large vms very slow.

Post by JosueM » 1 person likes this post

Hello foggy,

Yesterday run took about 40mins and thats the normal time. thanks for your help.

Image
TheGrove
Influencer
Posts: 14
Liked: never
Joined: Nov 20, 2021 3:35 am
Full Name: Bill Grover
Contact:

Re: Processing large vms very slow.

Post by TheGrove »

I realize I'm late to the discussion, but I have also seen that the replication after increasing a drive size is slow. Since the CBT is gone the replication spends a lot of time calculating a catalog before starting the replication. I found it was quicker to do the following.

Disable the replication job
Expand the drive (VMWare then Windows)
Delete the replica.
Enable the replication job
Run a replication

By doing this it forces the system to rebuild the replica and just copy everything. In the end it is faster than waiting for all of the cataloging and other stuff trying to deal with the size increase.
dbewernick
VeeaMVP
Posts: 680
Liked: 113 times
Joined: Jul 20, 2016 8:02 am
Full Name: David Bewernick
Contact:

Re: Processing large vms very slow.

Post by dbewernick »

@TheGrove please keep in mind that you will loose all the restore points created for the replica. For some these might be part of the DR strategy and a deletion of the replica VM would violate their concept.
TheGrove
Influencer
Posts: 14
Liked: never
Joined: Nov 20, 2021 3:35 am
Full Name: Bill Grover
Contact:

Re: Processing large vms very slow.

Post by TheGrove »

Thanks understood, and it is important to remember. in my specific case it is a tradeoff i am willing to accept. In my case we have 2 clusters a server and a VDI in the same server room. Each has it's own set of servers and SAN. My replication is a select set of servers from the server cluster to the VDI cluster. This is because the server hardware is older than we like (we are working on replacing it) so this is cheap insurance in case i have a failure in the server cluster.
Post Reply

Who is online

Users browsing this forum: Google [Bot] and 33 guests