Slow Replication to DR Site

[KLam]

Hi

Just wanting some help, I am currently experiencing slow replication for one of our main replica job, it is taking up to 10hr - 15hr for it to complete, which isn't feasible as we would like this replica to run every 2hrs.
(The incremental backup of this VM usually run every 2hr, which usually takes a few minutes, after the backup is completed, the replication should then run every 2hr.)

So currently using Veeam 6.5
Our Main Site and the DR site are configured identical (Server etc are spec exactly the same)
Both site have proxy configured and set.
this VM is 2.6TB in size, we have copied the data of this VM to the DR site, perform a restore, once the restore has completed, we then configured it to seed. Replication was then perform and was successful.

So now when the replication runs, it is taking so long, 10hrs - 15hrs, (by looking at the statistic, it looks like the replica is reading all the data within the VM (2.6TB)). Is there any way that I can reduce the time it takes for the replication to complete, as our other VM that we replicate takes 30 sec and at most 10mins (Varies on the amount of data).

all data is replicated to a Datastore at the DR Site
Change block tracking data is enabled
CBT for all protected VMs automatically is also enabled
Application-ware image processing is enabled

Help would be great

thanks, Kevin

[foggy]

Kevin, I wonder what does the bottleneck stats say for this job and do you see the [CBT] tag next to the VM's hard disks processing rate in the job session log, if you select this VM in the list to the left in the realtime stats window?

[KLam]

Hi Foggy

The bottleneck stats is showing source and yes i do see the CBT tags at the end of the VM Hard Disk

"Primary Bottleneck: Source"
"Hard Disk 1 (50.0 GB) 42.0 GB read at 48 MB/s (GBT)"
"Hard Disk 2 (1.9 TB) 1.9 TB read at 63 MB/s (GBT)"
"Hard Disk 3 (500.0 GB) 500.0 GB read at 60 MB/s (GBT)"

However I am getting other error message such as: (no red X though, all Green Ticks)

Unable to establish direct connection to the shared storage (SAN).
Please ensure that:

- Direct SAN connection is not available, failing over to network mode...

- HBA is properly installed in the Veeam Backup server computer, or software iSCSI initiator is configured correctly.
- SAN volume can be seen by operating system in the Windows Disk Management snap-in on the Veeam Backup server.
- Read access is allowed for the Veeam Backup server computer on the corresponding LUN (refer to your SAN documentation).

Kind Regard's

Kevin

[veremin]

Can I ask you to provide us with the information regarding transferred rate, as well? Thanks.

[foggy]

In some rare cases CBT data can get corrupt and trying to reset CBT could be among the steps in addressing this issue.

[KLam]

Hi

Here is the full report of the replication

Code: Select all

01-08-13 4:12:45 PM :: Queued for processing at 8/1/2013 4:12:45 PM
01-08-13 4:12:45 PM :: Required backup infrastructure resources have been assigned
01-08-13 4:12:48 PM :: VM processing started at 8/1/2013 4:12:47 PM
01-08-13 4:12:48 PM :: VM size: 2.4 TB
01-08-13 4:12:48 PM :: Using source proxy VMware Backup Proxy [san]
01-08-13 4:12:54 PM :: Using target proxy 192.168.112.37 [nbd]
01-08-13 4:12:54 PM :: Discovering replica VM
01-08-13 4:12:54 PM :: Preparing replica VM
01-08-13 4:13:39 PM :: Guest processing skipped (VM is powered off)
01-08-13 4:13:39 PM :: Creating VM snapshot
01-08-13 4:13:51 PM :: Processing configuration
01-08-13 4:14:11 PM :: Creating helper snapshot
01-08-13 4:14:17 PM :: Hard Disk 1 (50.0 GB) 42.GB read at 45MB/s (CBT)
01-08-13 4:14:19 PM :: Unable to establish direct connection to the shared storage (SAN).
Please ensure that:
- HBA is properly installed in the Veeam Backup server computer, or software iSCSI initiator is configured correctly.
- SAN volume can be seen by operating system in the Windows Disk Management snap-in on the Veeam Backup server.
- Read access is allowed for the Veeam Backup server computer on the corresponding LUN (refer to your SAN documentation).

01-08-13 4:14:22 PM :: Direct SAN connection is not available, failing over to network mode...
01-08-13 4:30:18 PM :: Hard Disk 2 (1.9 TB) 1.9 TB read at 61MB/s (CBT)
01-08-13 4:30:20 PM :: Unable to establish direct connection to the shared storage (SAN).
Please ensure that:
- HBA is properly installed in the Veeam Backup server computer, or software iSCSI initiator is configured correctly.
- SAN volume can be seen by operating system in the Windows Disk Management snap-in on the Veeam Backup server.
- Read access is allowed for the Veeam Backup server computer on the corresponding LUN (refer to your SAN documentation).

01-08-13 4:30:23 PM :: Direct SAN connection is not available, failing over to network mode...
02-08-13 1:25:16 AM :: Hard Disk 3 (500.0 GB)500.0 GB read at 59MB/s (CBT)
02-08-13 1:25:18 AM :: Unable to establish direct connection to the shared storage (SAN).
Please ensure that:
- HBA is properly installed in the Veeam Backup server computer, or software iSCSI initiator is configured correctly.
- SAN volume can be seen by operating system in the Windows Disk Management snap-in on the Veeam Backup server.
- Read access is allowed for the Veeam Backup server computer on the corresponding LUN (refer to your SAN documentation).

02-08-13 1:25:20 AM :: Direct SAN connection is not available, failing over to network mode...
02-08-13 3:50:24 AM :: Deleting helper snapshot
02-08-13 3:51:21 AM :: Removing VM snapshot
02-08-13 3:51:37 AM :: Swap file blocks skipped: 8.0 GB
02-08-13 3:51:37 AM :: Finalizing
02-08-13 3:51:49 AM :: Network traffic verification detected no corrupted blocks
02-08-13 3:51:49 AM :: Busy: Source 97% > Proxy 16% > Network 2% > Target 0%
02-08-13 3:51:49 AM :: Primary bottleneck: Source
02-08-13 3:51:50 AM :: Processing finished at 8/2/2013 3:51:49 AM

Summary
Duration: 11.39.38
Processing Rate: 61 MB/s
Bottleneck: Source

Data
Processed 2.4 TB(100%)
Read: 2.4TB
Transferred 4.4 GB (558x)

Apologize if i have provided the incorrect information, as I am new to Veeam.

Kind regard's

Kevin

[KLam]

Hi Foggy

Will try your solution and get back to you

Thanks

Kevin

[foggy]

Processed 2.4 TB(100%)
Read: 2.4TB
Transferred 4.4 GB (558x)

Looks like CBT is not actually working and our proprietary change tracking algorithm is being utilized that requires reading of the entire VM disks.

[veremin]

Yep, as Alexander mentioned, even though it was reported that CBT was utilized (“CBT” metrics), it doesn’t seem so (amount of read/transferred). So, it might be worth contacting support directly and letting them investigate such behaviour.

Thanks.

[KLam]

Hi

thanks for all your feedback, I will contact veeam support and see what they can do to help..

thanks heaps.

[veremin]

Hi, Kevin. It will be appreciated, if you can provide us with corresponding support ticket number, so that, we can follow the case, and update the topic once the resolution is found.

Thanks.