Replication error PrepStorageForWrite failed

[singy2002]

Hi,

I am trying to get to the bottom of an issue we are having with the replication tasks.

Setup is as follows.

2 host cluster at primary site (SAN storage)

1 host at secondary site, local storage (DR Site)

1gb Fibre channel between sites.


I had some original jobs setup about a month or so ago, we had to stop these as the customer identified a major issue with the snapshoting during peak hours, basically, when the snapshot was removed at the end fo the process the server is stunned and due to the volume of changes (SQL server) it affected production.
Since then we have now moved the SQL related maintenance plans and backups to be on a different disk that is excluded from the replication task.

In the time between the above job and the new job the only other thing that has changed is the ESX version, they were on vSphere 4.0.0 (build 164*** unpatched) they are now on the update 1 pacthed to 244038.

On saturday we decided to create a new job (deleting the old ones) and start a fresh, the initial replication took about 16 hours with a throughput of 4MB/s (slow but as it was the intial i recall it taking over 10 hours last time)

I have had 1 successfull replication since the initial. All the replication attempts since have come up with various errors.

This being the latest -
Checking destination PrepStorageForWrite failed Client error: No such file or directory POSIX: Failed to create or open file [<datacenterRef>datacenter-2</datacenterRef><hostRef>host-4145</hostRef><datastoreRef>datastore-4148</datastoreRef><datastoreGuid>4b8153ed-b15d9d73-9e46-00215edd374a</datastoreGuid><FullName>[JCBESXDR-Local] VeeamBackup/jcbdc01.jcb-finance.co.uk(vm-101)/replica.vbk</FullName>].

Earlier error -
Removing snapshot The remote name could not be resolved: 'jcbvc' Veeam Backup will attempt to remove snapshot during the next job cycle, but you may consider removing snapshot manually. Possible causes for snapshot removal failure: - Network connectivity issue, or vCenter server is too busy to serve the request - ESX host was unable to process snapshot removal request in a timely manner - Snapshot was already removed by another application PutLocalFile failed Client error: Cannot get service content. Soap fault. No DataDetail: 'get host by name failed in tcp_connect()', endpoint: 'https://jcbvc:443/sdk' Failed to open NFC storage [nfc://conn:jcbvc,nfchost:host-4145,stg:datastore-4148@VeeamBackup/jcbdc01.jcb-finance.co.uk(vm-101)/replica.vbk] for read/write access Server error: End of file

And an earlier one to the above.
Removing snapshot Failed to retrieve "SCSI (0:0) Hard disk 1" disk information. Check VM virtual hardware configuration. GetLocalText failed Client error: Connection limit is exceeded for replica host "jcbvc". Wait until concurrent operations involving this host complete, then try again. Server error: End of file Veeam Backup will attempt to remove snapshot during the next job cycle, but you may consider removing snapshot manually. Possible causes for snapshot removal failure: - Network connectivity issue, or vCenter server is too busy to serve the request - ESX host was unable to process snapshot removal request in a timely manner - Snapshot was already removed by another application BackupFile failed Client error: Search request to '[JCBESXDR-Local]VeeamBackup/jcbdc01.jcb-finance.co.uk(vm-101)/' error. Soap fault. TimeoutDetail: 'connect failed in tcp_connect()', endpoint: 'https://jcbvc:443/sdk' Failed to open NFC storage [nfc://conn:jcbvc,nfchost:host-4145,stg:datastore-4148@VeeamBackup/jcbdc01.jcb-finance.co.uk(vm-101)/2010-05-11T030047.vrb] for read/write access Cannot backup file in the service console mode. File: [nfc://conn:jcbvc,nfchost:host-2299,stg:datastore-15@jcbdc01.jcb-finance.co.uk/jcbdc01.jcb-finance.co.uk.nvram]. VBK: [veeamfs:*:/jcbdc01.jcb-finance.co.uk.nvram@nfc://conn:jcbvc,nfchost:host-4145,stg:datastore-4148@VeeamBackup/jcbdc01.jcb-finance.co.uk(vm-101)/replica.vbk]. RBK: [veeamfs:6:/jcbdc01.jcb-finance.co.uk.nvram@nfc://conn:jcbvc,nfchost:host-4145,stg:datastore-4148@VeeamBackup/jcbdc01.jcb-finance.co.uk(vm-101)/2010-05-11T030047.vrb]. Server error: End of file

The thing about this one i find interesting is the error "Connection limit is exceeded for replica host "jcbvc"

Any thoughts on this?

[singy2002]

Update on this.

I have created a new replica of a different server (only 24gb) and this has worked and continues to work, so the issue is with my two replica tasks of the large servers, due to the size of them I would prefer not to start them over again.

If you can suggest something I can do to remedy the problem I would be grateful.

[Vitaliy S.]

Hello Andrew,

It seems like there is an issue with the number of concurrent connections to the host "jcbvc". Have you already opened a support case with us?
It is best to contact our support directly for assistance with technical issues, as they require full logs research. You can find all the logs at Help|Support Information tab.

Thank you!

[singy2002]

I have logged a call via the website, but thought I may get a quicker response here if someone had enocuntered the error before.

I will wait for tech support thanks.

[lloyd.mills]

Did you find a solution to this issue? I am also having this issue with a replica.

[Vitaliy S.]

Hello Lloyd,

I cannot check this for you since previous poster have not mentioned his support case number,so please contact our support team directly to get a quicker assistance.