Backup jobs and Replication jobs sometimes run concurrently

[withanh]

My Backup job is scheduled to start at 7pm and is usually done by 10:30pm. I have my Replication job scheduled to start at 11:30pm to give it a little buffer, but sometimes the Backup job runs slow for some reason and the two jobs overlap, resulting in a machine possibly not replicating because it is in an active backup.

Reading some other posts got me thinking about how to overcome this and I decided on not scheduling the nightly Replication job but to take the command line and put that in as a Post Job Activity on the Backup job. So in theory my Backup job will run based on the schedule I set, and when it is finished it will kick off the Replication job.

Does anyone have any experience doing things this way? Just wondering if there are any hang-ups to expect. It is going to do the first run in this configuration tonight.

Thanks!

h

[Gostev]

Darhl, daisy chaining jobs is good approach - I have not heard anyone reporting issues with that. Let us know if it worked as expected for you.

[withanh]

Well, the daisy chain worked fine, but I'm still getting errors in my backups. At least they are not related to the machine already being locked by another job. Another thing I noticed is that when daisy chaining, the 2nd job doesn't retry if it fails. Which leads me to what if the originating job fails and goes into a retry, will it try to kick off the 2nd job every time it retries?

My 2 main backup jobs are to backup the iSCSI storage and to backup the internal BladeChassis storage. The backup job then kicks off the replication job for the same datastore.

My BladeChassisStorage replication gave me these two errors: (note this one tried to run for 29 minutes)

Code: Select all

Backing file "nfc://conn:<vcenter_server_name>,nfchost:host-32,stg:datastore-736@<vm1_name>/<vm1_name>-flat.vmdk" Failed to backup file "nfc://conn:<vcenter_server_name>,nfchost:host-32,stg:datastore-736@<vm1_name>/<vm1_name>-flat.vmdk" to "/vmfs/volumes/4a799858-07483c89-2b15-00145e2b33c0/VeeamBackup/<vm1_name>(vm-110)/replica.vbk" Replicate failed Client error: ChannelError: Timeout expired. The timeout period elapsed prior to completion of the operation or the server is not responding.

and (this one tried to run for 54 minutes)

Code: Select all

Backing file "nfc://conn:<vcenter_server_name>,nfchost:host-32,stg:datastore-736@<vm2_name>/<vm2_name>-flat.vmdk" Failed to backup file "nfc://conn:<vcenter_server_name>,nfchost:host-32,stg:datastore-736@<vm2_name>/<vm2_name>-flat.vmdk" to "/vmfs/volumes/4a799858-07483c89-2b15-00145e2b33c0/VeeamBackup/<vm2_name>(vm-733)/replica.vbk" Replicate failed Client error: Connection reset by peer Unable to retrieve next block transmission command. Number of already processed blocks: [33872]. Cannot backup file in the service console mode. File: [nfc://conn:<vcenter_server_name>,nfchost:host-32,stg:datastore-736@<vm2_name>/<vm2_name>-flat.vmdk]. VBK: [veeamfs:*:/<vm2_name>-flat.vmdk@/vmfs/volumes/4a799858-07483c89-2b15-00145e2b33c0/VeeamBackup/<vm2_name>(vm-733)/replica.vbk]. RBK: [veeamfs:9:/<vm2_name>-flat.vmdk@/vmfs/volumes/4a799858-07483c89-2b15-00145e2b33c0/VeeamBackup/<vm2_name>(vm-733)/2009-09-08T234610.vrb]. Server error: An existing connection was forcibly closed by the remote host Backup request cannot be processed. Source file path: [nfc://conn:<vcenter_server_name>,nfchost:host-32,stg:datastore-736@<vm2_name>/<vm2_name>-flat.vmdk]. 

And the iSCSI replication gave me these errors: (this one tried to run for 15 minutes)

Code: Select all

Completing current object backup process RemoveSnapshotfailed, snapshotRefsnapshot-1748, timeout 3600000. Failed to login to "<vcenter_server_name>" by SOAP, port 443, user "domain\administrator", proxy srv: port:0 Unable to connect to the remote server No connection could be made because the target machine actively refused it <vcenter_server_IP>:443

and this error on 3 machines (each of these tried to run for 20 seconds)

Code: Select all

Validating object Failed to login to "<vcenter_server_name>" by SOAP, port 443, user "domain\administrator", proxy srv: port:0 Unable to connect to the remote server No connection could be made because the target machine actively refused it <vcenter_server_IP>:443

Also of possible note (if it matters) the vCenter Server is one of the machines that responded with the 4th listed error.
Another noteworthy item is that the replica jobs go across a VPN to a remote site. Right now I have that configured locally (we have two internet connections) and I have a VPN built between the two for the replica jobs. This is for proof of concept before I actually move the replica jobs to our co-lo site.

[Gostev]

These issues would be best to take to our support, they may have seen them before in other environments and may be able to advice with resolution. By the way, do you have custom web service port configured on your vCenter server by any chance?

[withanh]

Nope, no custom web service port. I'll email the above post to support.

Any thoughts about the daisy-chaining and why the 2nd job didn't retry?

Thanks,

h

[Gostev]

We should be able to see the reason for not retrying from the full log files (from Help | Support Information).

[withanh]

Which log should I look at? There is only 1 from last night at 8:45pm, the next closest is from 2 weeks ago (8/29). Are my logs not configured to generate properly?

[tsightler]

You need to look in the directory for the user the service is running under if this is different than what you are logged in as. This confused me at first as well.

[withanh]

Got it. That is confusing!