Backup Copy Job, no longer completing

[ferrus]

For several months, we have used GFS rotation for one out largest VM copy jobs, using the parent backup files as the source.
The size of the job, and the performance of the destination Data Domain, meant the copy job usually took 48-60 hours to complete.

This wasn't a problem previously. The backup copy job would effectively pause (marked incomplete), for the nightly backup job - and then continue once it had completed.
It meant that we lost a couple of days restore points - but only the secondary RPs - and the Weekly, Monthly RPs are more important anyway.
The Weekly/Monthly backup would be marked as Success once it was complete.

For the last couple of months - possibly around the time of the 9.5 Update 2 installation, the behaviour seems to have changed:

In the Disk (Copy) job history, the Weekly/Monthly restore points are left as incomplete, and never change to Success.
I'm wondering if Veeam is still returning to complete these jobs - or just leaving the GFS RPs Incomplete - and moving on to the Daily copies.

Is there any configuration setting that affects this behaviour?

[foggy]

Hi, this looks like an issue we're currently investigating and planning to release a hotfix for. Please contact technical support for verification. Thanks.

[ferrus]

Thanks, I've placed the support call - #02315618

[r2d2]

Here is an issue No 114620
Please refer to it when you start a conversation with support.

[ferrus]

Thanks for returning to this thread, r2d2.
Unfortunately, the support call I placed (#2315618), has now been closed.

Are you able to give a description of issue 114620, or the hotfix foggy referred to?

The reason I ask, I'm still not confident about this backup.
The result of the support call was that everything is working as designed, and to get rid of the incomplete status - backup less frequently.
It's still a regression in my eyes however, and looking at the disparity in backup size - I'm still not sure it's working correctly:



While the Backup Job data sizes are relatively constant, the full Backup Copy Job sizes (above) vary from 10.2TB to 7.4TB.
I'm excluding the most recent full backup - 4.58TB, because that still has incremental points, and the way it was described - those incrementals make up the rest of the incomplete chain.

Previous weekly and monthly GFS restore points don't have incrementals, and should be complete. So it's a leap of faith that the incrementals have been merged into these GFS points before deletion.
I've reduced the number of incomplete restore points, by moving the GFS point to Friday instead of Saturday (utilizing the whole weekend), but doubts still remain. Unfortunately I can't test this because of another support call we have regarding this Backup Job (#02220662 - BCJ restore points appearing as encrypted), which we're awaiting a private patch to fix.
Does this appear normal?

[foggy]

Are you using active full GFS (i.e. the "Read from source..." check box is selected in the backup copy job settings)?

[ferrus]

Yes.
The Backup Copy Job destination, is an EMC DD2500 - so the previous strategy before "Read from source" took several hours each night, sometimes >24 hours to merge.

The weekly Active Full still takes over 24 hours, but this is only once a week - not every night.

[foggy]

Then most likely the mentioned hotfix should help. I recommend re-opening the case and ask the engineer to check its applicability.

[ferrus]

Thanks, foggy.

I've opened another support call, and referenced the other cases and issue number.
The new call is #02367520

[ferrus]

Update on this call ...

Support forwarded the patch to me, in time for this weekend's Monthly/Weekly RP - so perfect timing for testing.
Pleased to report that everything is now back to the old behaviour, ie. keep restarting the Full BCJ until completion.

For the first time in months, I have a clear 'Complete' BCJ, with full data size 18.1TB.
I think this may mean that the other RPs are truly incomplete/invalid, but I'm glad it's sorted now. Would recommend the patch to anyone with the same issue.

Thanks foggy and r2d2 - for returning to this forum post, after the original call was closed.