On Scheduled Backup create two copy Sessions

[aantonacci@shci.ca]

Hi Guys,

I have a few customer where a backup is configured for 2 VMs (with a copy job) On. the Copy Job, the VM get fully processed and the other fails.

The error that is logged is this one "2022-05-06 5:13:03 PM :: Processing MyVMName Error: Failed to connect to the port [MyCloudServer.com:1234]."

Once the first session finishes, VEEAM appears to retry by creating a second session. In the second session its skips the successful VM and completes successfully second one.
The first session status is failed, the second one is warning.

I am trying to figure out, how the two sessions tie into each other, In my code, i need to merge both to see if the backup ended being good. Since the first session is failed, and the second is a warning it gets a little hard to be sure the backup ended up being successful.

[HannesK]

Hello,
first: I would solve the root cause (with help of Veeam support if needed) instead of creating PowerShell scripts The error sounds like a network issue and might be outside Veeam.

I need to merge both to see if the backup ended being good

the backup copy job "last result" should have the "final state". As the retry for unknown reason ends with "warning" in your case, the result of the backup copy job "last result" is "warning".

Best regards,
Hannes

[david.domask]

Overall strong agree with Hannes on this, the behavior itself should be resolved, not have exceptions written.

I've not dug into this deeply, but CBackupSession objects have an IsRetry property if I remember right and you can use this on the same day to know whether or not you're working with a retry or not. Since you can know the Retry count from the job settings, you can just check all sessions before the next scheduled one and check the IsRetry value.

But the behavior you describe is quite unusual, and should be resolved. I get your goal is just how to handle Veeam's retry logic in scripting, but the "best" way is just to solve the failures in the first place, not write code exceptions for it.

[aantonacci@shci.ca]

Actually, yes the example above was a more exceptional situation, here is one that we have not figured out how to resolve ... (hense why i want to merge multiple sessions)

our Theory is, we setup a backup job to backup 2 VM (big ones), them setup to "immediately" run a copy job. Finally, have the backupjob do a health check on the local copy...

So what we think is happening is the backupjob "calls" the copyjob as soon as the backup part is complete, ignoring the fact that it needs to run a healthcheck.
When the copyjob attemps to upload the VM it fails since its busy doing a healthCheck... (again our theory)

in the logs of the copyjob we get:
2022-04-23 11:09:58 AM :: Job started at 2022-04-23 11:09:52 AM
2022-04-23 11:29:58 AM :: Required backup infrastructure resources have been assigned
2022-04-23 11:30:41 AM :: Source restore point is locked by another job
2022-04-23 11:30:42 AM :: Source restore point is locked by another job


in the Backupjob we clearly see: (i dont see the start of the healthcheck)

2022-04-23 11:29:42 AM :: Backup files health check has been completed

Besides playing M. Calendar and manually setting copyjobs (not going there) - how do I tell Veeam to wait for itself to finish a job before starting the next one?

[david.domask]

hey Anthony,

Does the job actually fail though or just wait? AFAIk it should just wait -- the lines you see are Veeam tracking the lease as appropriate, and it should resume once the file lock is released.

[aantonacci@shci.ca]

No the jobs passes in the end, but the first job session is failed and the second is in warning. I don't like warnings in my stuff.
Also in powershell it gets difficult match up both sessions as they don't appear to be tied together in any way.

VEEAMs lame response is to stop using the automatic "run copy job after jobs ends" because they (VEEAM) appear to be unable to handle the fact that they are running an health check.

This will be the third major bug that we find in the App and they are unable to fix !

How can an product like VEEAM be full of so much issues and reach version 12 without fixing them .... I switch to Veeam hoping the app was robust and reliable

"The job did not fail completely. The way the system work, it will copy all the backup data that is accessible. If the file is locked by another job, Veeam will make multiple attempts to access it before giving up. That job session is then marked as failed and retried with only the missing data being copied since Veeam is aware of what was processed in the previous attempt.

If this is a behavior that has been recurring with this specific job, it might be a good idea to change the copy job schedule to prevent it from running at the same time as Weekly 12 as my colleague recommended."

[HannesK]

Hello,

VEEAMs lame response is to stop using the automatic "run copy job after jobs ends" because they (VEEAM) appear to be unable to handle the fact that they are running an health check.

That recommendation sounds strange. Because with the periodic mode, many customers start it some minutes before the backup job so that it kicks in after the backup job finished. For mirror mode, there is no dedicated schedule. Do you have a support case number with a bug number (6 digits) already?

that they are running an health check.

I got lost on the PowerShell topic, but health check will be schedulable in V12. So it's not tied to the backup job schedule anymore.

Best regards,
Hannes

[david.domask]

@aantonacci@shci.ca what's the case number please?