Replication Files

[jpaul]

Im getting an error on some of my replication jobs, and am wondering what is causing it. We are replication a customers Data to a remote ESXi host over a 10meg pipe. There are 8 VM's being replicated each night.

here is the error message:
3 of 5 files processed

Total VM size: 135.99 GB
Processed size: 11.46 KB
Processing rate: 0 KB/s
Backup mode: SAN/NBD with changed block tracking
Start time: 11/14/2010 11:00:30 PM
End time: 11/14/2010 11:22:59 PM
Duration: 0:22:28

Replicating file "[SAN_LUN2] Logistics.COMPANYABC.LOCAL/Logistics.COMPANYABC.LOCAL.vmdk"
BackupText failed
Client error: Can not retrieve requested properties.
Soap fault. TimeoutDetail: 'connect failed in tcp_connect()', endpoint: 'https://192.168.225.35:443/sdk'
Failed to backup text locally. Backup: [veeamfs:*:/Logistics.COMPANYABC.LOCAL.vmdk@nfc://conn:192.168.225.35,nfchost:ha-host,stg:4cc9d927-0ea45c56-0bff-005056a24382@VeeamBackup/Logistics(vm-225)/replica.vbkVBK: 'veeamfs:*:/Logistics.COMPANYABC.LOCAL.vmdk@nfc://conn:192.168.225.35,nfchost:ha-host,stg:4cc9d927-0ea45c56-0bff-005056a24382@VeeamBackup/Logistics(vm-225)/replica.vbk'RBK: 'veeamfs:9:/Logistics.COMPANYABC.LOCAL.vmdk@nfc://conn:192.168.225.35,nfchost:ha-host,stg:4cc9d927-0ea45c56-0bff-005056a24382@VeeamBackup/Logistics(vm-225)/2010-11-13T230031.vrb'].

Server error: End of file


oddly, when i checked the backup log today this job ran again and said that it had completed successfully? any ideas?

is this caused because there are multiple jobs running at the same time ? maybe not enough bandwidth or something ?

Thanks Guys,
Justin

[jpaul]

i also get this error on another VM... eventually the job seems to go through though.

2 of 5 files processed

Total VM size: 160.01 GB
Processed size: 5.31 KB
Processing rate: 0 KB/s
Backup mode: SAN/NBD with changed block tracking
Start time: 11/12/2010 11:00:33 PM
End time: 11/12/2010 11:19:35 PM
Duration: 0:19:01

Replicating file "[SAN_LUN1] mail2003/mail2003.nvram"
BackupFile failed
Client error: Can not retrieve requested properties.
Soap fault. TimeoutDetail: 'connect failed in tcp_connect()', endpoint: 'https://192.168.225.35:443/sdk'
Failed to open NFC storage [nfc://conn:192.168.225.35,nfchost:ha-host,stg:4cc9d927-0ea45c56-0bff-005056a24382@VeeamBackup/mail2003(vm-272)/2010-11-11T230027.vrb] for read/write access
Cannot backup file in the service console mode. File: [nfc://conn:10.10.1.15,nfchost:host-9,stg:datastore-11@mail2003/mail2003.nvram]. VBK: [veeamfs:*:/mail2003.nvram@nfc://conn:192.168.225.35,nfchost:ha-host,stg:4cc9d927-0ea45c56-0bff-005056a24382@VeeamBackup/mail2003(vm-272)/replica.vbk]. RBK: [veeamfs:9:/mail2003.nvram@nfc://conn:192.168.225.35,nfchost:ha-host,stg:4cc9d927-0ea45c56-0bff-005056a24382@VeeamBackup/mail2003(vm-272)/2010-11-11T230027.vrb].

Server error: End of file

[Vitaliy S.]

Hello Justin,

How many job do you have running concurrently? It seems like you have intermittent timeouts that were causes by SOAP calls failure. If you have many jobs running at the same time, try reducing this amount to see if that helps.

Thanks!

[mplep]

Hi Vitaliy,

I have ticket #541708 logged which sounds similar?

[jpaul]

there should only be 2-4 running at the same time ... is that too many?

[Alexey D.]

Hello Justin,

Please tell us:
1) Is your 10M connection saturated by any other activity (not related to B&R jobs)? Are you experiencing any problems when connecting to that ESXi with vSphere Client?
2) Are you using vCenter, or just specified a direct connection to that ESXi?
3) What is the version of ESXi?

The point is that ESXi has a rather short pool of connections, and even 2-4 jobs may saturate it, especially if there is any problem with the pipe to your customer's site.

[jpaul]

the connection isnt saturated by anything except the B& R jobs. Im using esxi 4.1 and i do not stay connected to it via the vsphere management interface so i cant say if its causing errors.

[Alexey D.]

Hi Justin, please send all the logs to our support team! Thanks.