Comprehensive data protection for all workloads
Post Reply
Yuki
Veeam ProPartner
Posts: 252
Liked: 26 times
Joined: Apr 05, 2011 11:44 pm
Contact:

Very slow replication

Post by Yuki » Dec 17, 2012 9:04 am

Hey all,
We've got an issue with very slow replication from one of the hosts. Here is how our replication looks:

1 Physical Veeam server
2 ESXi1 with VM1 (proxy for all VMs on that host)
3 ESXi2 with VM7 (proxy for itself)
4 ESXi3 - replica target

No shared storage, all local. All proxies have 8 cores assigned to them.


Veem (physical)
|||
|||>ESXi1 (vm1-vm6)--------->
||>>-----------------------------> ESXi3
|>>>ESXi2 (vm7) ------------->


ESXi1 has replicated all VMs in about 2-2.5 hours(about 1.5TB)
ESXi2 which is backing up itself (also about 1.3TB) has done only 2% in an hour +. When i look at the statistics report it shows the bottleneck to be the "source". In the VM7 i can see that all CPUs are at about 80% utilization. The processing rate is showing about 450MBps and network seems to be pushing a lot of traffic, but the processing is abnormally slow. ESXi1 and ESXi2 are identical and connected to the same switch, etc. I can't figure out why the rate is so low. This is the first replica of VM7 from ESXi2 to the ESXi3.

Any suggestions?

dellock6
Veeam Software
Posts: 5734
Liked: 1626 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Very slow replication

Post by dellock6 » Dec 17, 2012 9:28 am

A proxy cannot hotadd itself, and uses network mode. Take a look at the job statistics, probably you will see other VMs beeing replicated with HotAdd, thus they are completed faster.
Also, is ESX3 in another subnet, on across a WAN link? You better place a proxy also at destination to improve performances of all replica jobs, as per the user guide.

Luca.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2019
Veeam VMCE #1

Yuki
Veeam ProPartner
Posts: 252
Liked: 26 times
Joined: Apr 05, 2011 11:44 pm
Contact:

Re: Very slow replication

Post by Yuki » Dec 17, 2012 9:42 am

Hi Luca,
Yes, it can't hot add itself, but that's expected with it being the only host on the server. The rate is still much lower than for example VM1 from ESXi1 was replicating itself.

The target system is sitting on a 1Gbit network connection (LAN) on the same subnet and has a VM with proxy on it. Once we seed all replicas and backups locally - the system will be sent to DR site.

UPDATE:
Ok, i've dug around some more and noticed that the way one of the admins created this VM with several spanned VMDKs. He needed to present a 12TB storage partition to Guest OS, so he created 6x2TB VMDKs and spanned the volume across them. This is due to vSphere VMFS limitation of 2TB. Could this be the reason?

dellock6
Veeam Software
Posts: 5734
Liked: 1626 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Very slow replication

Post by dellock6 » Dec 17, 2012 10:02 am

I'm not aware of any problem involving dynamic disks on the Veeam Proxy itself. Also, multiple disks in the same VM can slow down operations when using hotadd because of the mount/unmount operations itself, but not in network mode. Maybe CBT is not available on this VM? Also, where is located the metadata repository, each proxy has its own? Or is it into the physical Veeam server for both proxies?
Since VM2 only replicates itself, a local metadata repository could help...

Luca.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2019
Veeam VMCE #1

Yuki
Veeam ProPartner
Posts: 252
Liked: 26 times
Joined: Apr 05, 2011 11:44 pm
Contact:

Re: Very slow replication

Post by Yuki » Dec 17, 2012 10:09 am

Metadata repository is located on a NAS present on the network (mapped through the target Proxy but has it's own 1Gbit connection). NAS is not being the bottlneck here as i can see it's utilization and both CPU/Network are at arond 1%.

Another piece that may be useful: VM is a Windows 2012 with 2 volumes (60gb and 12TB). The 12TB volume has Deduplication turned ON and shows 24% efficiency. Any issues with Guest dedupe and Veeam in network mode? I wouldn't think so,but...?

dellock6
Veeam Software
Posts: 5734
Liked: 1626 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Very slow replication

Post by dellock6 » Dec 17, 2012 10:25 am

You said on first post bottleneck is source, maybe the underlying local storage is somehow different from the other server?
Also, have you left automatic choice of proxies, or you configured VM2 for source and the proxy on ESX3 as destination?
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2019
Veeam VMCE #1

Yuki
Veeam ProPartner
Posts: 252
Liked: 26 times
Joined: Apr 05, 2011 11:44 pm
Contact:

Re: Very slow replication

Post by Yuki » Dec 17, 2012 10:37 am

Yes, the bottlneck show in the "Statistics report" within veeam shows "Source" as the bottleneck. The disk subsystem is a SAS-RAID5 volume. it performed very well during a backup operation (not a replica that is being slow).

I had to configure the proxies manually in order to make sure it doesn't try to chose proxy that will be going off-site when backup and replication seeding is completed. However, the system has a number of proxies available including VM7 which is being replicated. Looking at the processes i can see that VM is using itself as the source proxy for replication (Being the only VM on the physical host). The target proxy was also set manually to the VM on the target host.

dellock6
Veeam Software
Posts: 5734
Liked: 1626 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Very slow replication

Post by dellock6 » Dec 17, 2012 11:12 am

Everyhing seems correctly configured. Maybe is better to open a support ticket to let Veeam inspect the logs; and come back here to tell us the results so everyone can benefit from them ;)
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2019
Veeam VMCE #1

christiank@pcs.org
Influencer
Posts: 13
Liked: never
Joined: Dec 19, 2012 8:04 pm
Full Name: Chris
Contact:

Re: Very slow replication

Post by christiank@pcs.org » Dec 19, 2012 11:57 pm

I am experience similar problem to this but it happens when I try to backup VMs. Except for the VEEAM server VM. Did you ever got it resolve? Your solution might help my situation.

Chris

Vitaliy S.
Product Manager
Posts: 23010
Liked: 1560 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Very slow replication

Post by Vitaliy S. » Dec 20, 2012 9:36 am

Hi Chris, I'm afraid Yuki has a little bit different problem, anyway what does the bottleneck statistics show for your backup job?

Yuki
Veeam ProPartner
Posts: 252
Liked: 26 times
Joined: Apr 05, 2011 11:44 pm
Contact:

Re: Very slow replication

Post by Yuki » Dec 20, 2012 11:15 pm

We've opened a case with support and were told we have a corrupt OIB file (not sure what exactly that file is for). And the way to fix this is to remove backups from "backups" in veeam, run Active Full and then import older backups back. That was the original issue. if you are seeing this with Replica - then you may need to remove replica form the "replicas" and in the job settings check the "use low bandwidth" option, then map the problematic VM (the one that has the corrupt OIB) back inside that job.

Right now I'm looking at a job we've initiated with Veeam support while troubleshooting and then tried to cancel it - it has been canceling for over an hour at this point (again related to corruption).

Yuki
Veeam ProPartner
Posts: 252
Liked: 26 times
Joined: Apr 05, 2011 11:44 pm
Contact:

Re: Very slow replication

Post by Yuki » Dec 20, 2012 11:38 pm

Send a follow up e-mail to support asking how to kill the stuck job without rebooting the production server, but didnt receive a response yet. Any suggestions? (it was canceled, but it just sits there doing nothing).

Yuki
Veeam ProPartner
Posts: 252
Liked: 26 times
Joined: Apr 05, 2011 11:44 pm
Contact:

Re: Very slow replication

Post by Yuki » Dec 21, 2012 5:23 am

It looks like we are able to roll backups again without the delay before the backup job starts, but the performance way down. Local proxy with 8 cores is only at 70% CPU utilization, via hot-add and with NAS as the target. We generally see 110MB/s transfer speed as reported by NAS (Veeam statistic's speed is kind of funky, so we don't rely on it). Right now we are seeing only 35Mb/s (Veeam is reporting 70MB/s)

I still have not received a reply to either of my several follow up e-mails. Guessing Veeam doesn't have 24/7 support?

Yuki
Veeam ProPartner
Posts: 252
Liked: 26 times
Joined: Apr 05, 2011 11:44 pm
Contact:

Re: Very slow replication

Post by Yuki » Dec 21, 2012 8:15 am

talked to support, a representative looked at the system but didn't see any obvious misconfiguration. Sent logs and waiting to hear back now.

Gostev
SVP, Product Management
Posts: 24812
Liked: 3573 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Very slow replication

Post by Gostev » Dec 21, 2012 8:21 am

Yuki wrote:Guessing Veeam doesn't have 24/7 support?
We do. But do you have this option (Premium support) purchased?

Yuki
Veeam ProPartner
Posts: 252
Liked: 26 times
Joined: Apr 05, 2011 11:44 pm
Contact:

Re: Very slow replication

Post by Yuki » Dec 21, 2012 8:52 am

Gostev,
i did reach support, i guess person I've corresponded with initially had left for the day so i ended up talking to another tech. :)

Post Reply

Who is online

Users browsing this forum: Baidu [Spider], iL1fe and 45 guests