Very slow replication

[Yuki]

Hey all,
We've got an issue with very slow replication from one of the hosts. Here is how our replication looks:

1 Physical Veeam server
2 ESXi1 with VM1 (proxy for all VMs on that host)
3 ESXi2 with VM7 (proxy for itself)
4 ESXi3 - replica target

No shared storage, all local. All proxies have 8 cores assigned to them.


Veem (physical)
|||
|||>ESXi1 (vm1-vm6)--------->
||>>-----------------------------> ESXi3
|>>>ESXi2 (vm7) ------------->


ESXi1 has replicated all VMs in about 2-2.5 hours(about 1.5TB)
ESXi2 which is backing up itself (also about 1.3TB) has done only 2% in an hour +. When i look at the statistics report it shows the bottleneck to be the "source". In the VM7 i can see that all CPUs are at about 80% utilization. The processing rate is showing about 450MBps and network seems to be pushing a lot of traffic, but the processing is abnormally slow. ESXi1 and ESXi2 are identical and connected to the same switch, etc. I can't figure out why the rate is so low. This is the first replica of VM7 from ESXi2 to the ESXi3.

Any suggestions?

[dellock6]

A proxy cannot hotadd itself, and uses network mode. Take a look at the job statistics, probably you will see other VMs beeing replicated with HotAdd, thus they are completed faster.
Also, is ESX3 in another subnet, on across a WAN link? You better place a proxy also at destination to improve performances of all replica jobs, as per the user guide.

Luca.

[Yuki]

Hi Luca,
Yes, it can't hot add itself, but that's expected with it being the only host on the server. The rate is still much lower than for example VM1 from ESXi1 was replicating itself.

The target system is sitting on a 1Gbit network connection (LAN) on the same subnet and has a VM with proxy on it. Once we seed all replicas and backups locally - the system will be sent to DR site.

UPDATE:
Ok, i've dug around some more and noticed that the way one of the admins created this VM with several spanned VMDKs. He needed to present a 12TB storage partition to Guest OS, so he created 6x2TB VMDKs and spanned the volume across them. This is due to vSphere VMFS limitation of 2TB. Could this be the reason?

[dellock6]

I'm not aware of any problem involving dynamic disks on the Veeam Proxy itself. Also, multiple disks in the same VM can slow down operations when using hotadd because of the mount/unmount operations itself, but not in network mode. Maybe CBT is not available on this VM? Also, where is located the metadata repository, each proxy has its own? Or is it into the physical Veeam server for both proxies?
Since VM2 only replicates itself, a local metadata repository could help...

Luca.

[Yuki]

Metadata repository is located on a NAS present on the network (mapped through the target Proxy but has it's own 1Gbit connection). NAS is not being the bottlneck here as i can see it's utilization and both CPU/Network are at arond 1%.

Another piece that may be useful: VM is a Windows 2012 with 2 volumes (60gb and 12TB). The 12TB volume has Deduplication turned ON and shows 24% efficiency. Any issues with Guest dedupe and Veeam in network mode? I wouldn't think so,but...?

[dellock6]

You said on first post bottleneck is source, maybe the underlying local storage is somehow different from the other server?
Also, have you left automatic choice of proxies, or you configured VM2 for source and the proxy on ESX3 as destination?

[Yuki]

Yes, the bottlneck show in the "Statistics report" within veeam shows "Source" as the bottleneck. The disk subsystem is a SAS-RAID5 volume. it performed very well during a backup operation (not a replica that is being slow).

I had to configure the proxies manually in order to make sure it doesn't try to chose proxy that will be going off-site when backup and replication seeding is completed. However, the system has a number of proxies available including VM7 which is being replicated. Looking at the processes i can see that VM is using itself as the source proxy for replication (Being the only VM on the physical host). The target proxy was also set manually to the VM on the target host.

[dellock6]

Everyhing seems correctly configured. Maybe is better to open a support ticket to let Veeam inspect the logs; and come back here to tell us the results so everyone can benefit from them

[christiank@pcs.org]

I am experience similar problem to this but it happens when I try to backup VMs. Except for the VEEAM server VM. Did you ever got it resolve? Your solution might help my situation.

Chris

[Vitaliy S.]

Hi Chris, I'm afraid Yuki has a little bit different problem, anyway what does the bottleneck statistics show for your backup job?

[Yuki]

We've opened a case with support and were told we have a corrupt OIB file (not sure what exactly that file is for). And the way to fix this is to remove backups from "backups" in veeam, run Active Full and then import older backups back. That was the original issue. if you are seeing this with Replica - then you may need to remove replica form the "replicas" and in the job settings check the "use low bandwidth" option, then map the problematic VM (the one that has the corrupt OIB) back inside that job.

Right now I'm looking at a job we've initiated with Veeam support while troubleshooting and then tried to cancel it - it has been canceling for over an hour at this point (again related to corruption).

[Yuki]

Send a follow up e-mail to support asking how to kill the stuck job without rebooting the production server, but didnt receive a response yet. Any suggestions? (it was canceled, but it just sits there doing nothing).

[Yuki]

It looks like we are able to roll backups again without the delay before the backup job starts, but the performance way down. Local proxy with 8 cores is only at 70% CPU utilization, via hot-add and with NAS as the target. We generally see 110MB/s transfer speed as reported by NAS (Veeam statistic's speed is kind of funky, so we don't rely on it). Right now we are seeing only 35Mb/s (Veeam is reporting 70MB/s)

I still have not received a reply to either of my several follow up e-mails. Guessing Veeam doesn't have 24/7 support?

[Yuki]

talked to support, a representative looked at the system but didn't see any obvious misconfiguration. Sent logs and waiting to hear back now.

[Gostev]

Guessing Veeam doesn't have 24/7 support?

We do. But do you have this option (Premium support) purchased?

[Yuki]

Gostev,
i did reach support, i guess person I've corresponded with initially had left for the day so i ended up talking to another tech.