Virtual full not working after upgrade to 9.5u4

[holzhei]

On Saturdays we schedule a full tape backup and then have incrementals to tape Sunday, Monday, Tuesday, Wednesday, Thursday, Friday. The source jobs are 10x backup jobs, and 2x backup copies. The 10x backup jobs have active fulls run every Saturday, which in turn forces a full backup to tape every Saturday (and the virtual full setting is ignored), and then incremental are backed up to tape every other day. The 2x backup copies are forever forward, and from my understanding should have a virtual full backup (vbm) created on the tape full backup day (Saturday), and incrementals are backed up to tape Sunday, Monday, Tuesday, Wednesday, Thursday, Friday. This was working for us on 9.5u3.

Since upgrading to 9.5u4 a few weeks ago, the virtual fulls are not created reliably to tape for the 2x backup copy jobs. Some times the are created on the Sunday run, other times they are not created at all.

For instance this week one of the backups copy jobs had the virtual full created on Sunday, and the other one has still not created a virtual full, only incrementals on Saturday, Sunday, Monday, Tuesday, Wednesday.

Any hints on where I should be looking?

Case number is 03432873

[HannesK]

Hello,
and welcome to the forums.

As you say that you did not change anything and it worked with U3, the only thing I can recommend is to continue working with support. Also your observation with "sometimes they are created, sometimes not" seems strange to me.

There seems to be a typo in the case number, as the stated case was closed with "no answer from customer" some time ago.

Best regards,
Hannes

[nathano]

That sounds like a problem I have, since U4 getting synthetic fulls to tape for one of my backup copy jobs is hit and miss. I have 2 BCJs and several local full backups. the local backups generate a synthetic full to tape each night but the BCJs are the problem.

my case# is 03442611

I will update my case and reference your case ID

[holzhei]

We logged two cases, but got similar responses each time saying that this is normal as the incremental is locked. Previously it would create a virtual full with the latest available incremental (which may have been the previous one if the latest was locked, and we were okay with that). Looks like u4a was just released with reliability fixes for tape, although it does not go into much detail.

After not being able to get the normal tape backups with incremental working, we switched to a GFS media pool which now allows daily incremental since the u4 upgrade. Although we aren't able to schedule it to run after our last backup job like we could with a normal tape job.

[HannesK]

@holzhei: it would help me to search for details if you could post the case numbers here. Nathano is currently testing U4a but I cannot see a case update yet (is "on hold")

[nathano]

I'm still working with support on the issue.. next thing I am trying is this.. will see if it backs up OK tonight.

"
Hi Please remove the backup copy job from the configuration and re-import the backup.

1. Open the Home view.
2. Disable the backup copy job.
3. In the inventory pane, select backups-> disk(copy).
4. In the working area, right-click the backup copy job ->right click->select "remove from configuration".
5. select Backup infrastructure view-> rescan backup copy repository
6. The backup files will appear under backups-> disk imported.
7.Edit the backup copy job->target->map backup->select the backup copy job.
8. This will move the backup files from disk imported -> disk(copy).
9. Edit the tape job->remove the backup copy job and re-add.
10. Run the backup tape job.
"

[nathano]

the above steps looked like it fixed the problem, but no, last night both BCJs were left off the tape.

This is getting ridiculous.

[Dima P.]

Hello Nathan,

I've asked QA folks to review your case. I'll let you know once I hear back anything. Cheers!

[nathano]

Dima

I hope there's a resolution soon, our backup to tape is a critical part of our backup strategy, you know, 3-2-1 etc..

There should not be a green tick next to a component of the job that failed to backup any data, as you can see in this screen shot, it failed to capture any data for one of the requested backup to tape components and it still ticks it off as green! Crazy stuff, this has to be a warning, can this please be logged as a feature request.

[sweepsd]

I'm having a similar issue - although I am new to Veeam so not sure if it would have worked in the past. I cannot get a backup copy to create a synthetic full for a specific day, all it does is just copy the oldest full to the tape. If I alter the same job using the same media pool and choose a VM with a backup source (rather than a backup copy) it creates the synthetic full as expected.

I've moved from Backup exec to Veeam and beginning to pull my hair out over this product! I have logged a ticket 035142 but would like some help from anyone here if they can. I am running 9.5u4a

Andy

[Dima P.]

Was told that this behavior looks like a known issue and we already working on a private fix. Please keep in touch with your support engineer. Please let me know how it goes!

[sweepsd]

Hi
I was given a hotfix after the techie said there was a bug, but it hasn't resolved the issue. If I run 2 identical jobs but with different sources - backup and backup copy, the backup source works as expected. The backup copy always just copies the oldest full to tape.

[HannesK]

Hello,
finally there is an hotfix available fixing the issue. If you run into it, please contact support.

Best regards,
Hannes

[nathano]

yes I got the hotfix last week and so far so good. it took a lot longer than I would have liked but at least I have confidence in the tape jobs again..

I'll have to test a scenario where there's no new restore points and see if I get a warning on the job as I would expect.

[lyapkost]

Hi Nathan, thanks for the feedback. Keep in mind that we're not planning to include this fix to the upcoming minor update 4b. So in case you're going to install it (for example to support Linux 5.0 kernel in your Agent backups) please contact support to have an updated dll. It should be ready within 1-2 weeks from the release date.

[joespirit88]

We also ran into the same issue but the hotfix looks to have fixed this for us.

There should not be a green tick next to a component of the job that failed to backup any data, as you can see in this screen shot, it failed to capture any data for one of the requested backup to tape components and it still ticks it off as green! Crazy stuff, this has to be a warning, can this please be logged as a feature request.

100% agree, I raised this with my support engineer who said this needs to be raised a feature request. If i schedule a backup job i expect it to back up data.
We have holes in our backup data because this issue went unnoticed for a few weeks due to it not being seen as a problem, it was only because i was poking around some settings i thought something was off.
With backup copies i get warnings if no new safesets were detected during a run, why would this be different for tapes?
Also why would this hotfix not be included in the next release? Surely as a backup product if it is not performing backups and not informing the administrators this is a major issue?

[Gostev]

It's a dangerous fix to include in an update, since it touches many components. It will for sure be included in the next major release though. Thanks!

[joespirit88]

Thanks for the response Gostev however I have a couple more questions:

Will there be a reference for us when we contact support to have the hotfix for the new minor updates or a KB we can refer detailing the bug and letting us know that the hotfix is available for that release?

As we have pointed out Veeam is incorrectly saying that tape backups are successful when they are not, will this also be fixed?

[Dima P.]

Will there be a reference for us when we contact support to have the hotfix for the new minor updates or a KB we can refer detailing the bug and letting us know that the hotfix is available for that release?

You can use this thread or your previous case for the reference, I'll discuss the KB article with the link to the fix with our support team as well. The fix will be rebuild right after U4b goes live, so it would be possible to installed again after applying the update.

As we have pointed out Veeam is incorrectly saying that tape backups are successful when they are not, will this also be fixed?

Yes, that is going to be fixed as well. Our goal was to fix the root cause on the first place and fix the issues with the backup, once the issue is fixed - reporting will work correctly.

[DerOest]

Case # 03630614
I think we hit the same or similar problem, but did not get far with support yet ("lets see how it works next saturday").

Last saturday, only 1 or 2 of 13 Tape jobs picked up the GFS-copies that should go to tape.
The other jobs all were "successful", but had "no new files found" and didn't back up anything - which is dangerously misleading! see above, "holes in our backup data"

Forcing a "Weekly" on those jobs then worked flawless and they picked up the GFS as they should.


Looks like this issue is more widespread than expected and should be covered better than "not going into U4b, request a new private fix (that not every support staff knows about)".
We've also got several other tape issues, it would be nice if there was some kind of a preview-changelog for the tape-fixes in 4b. Would be a waste if i open several other tickets and that's all going to be fixed soon anyway.

[Dima P.]

Hello DerOest,

I am sorry to hear it. I've reached your support engineer and asked to review the case details to make sure that you are affected by this issue. If that's the case support team will share a private fix with you.

it would be nice if there was some kind of a preview-changelog for the tape-fixes in 4b.

We share the list of fixed issues together with the new version, so we will have it publicly available when U4b goes live.

[Dima P.]

Hello folks,

KB2975 is ready and contains the link to the hotfix. The fix is fully tested against Veeam B&R 9.5 Update 4a. Cheers!

EDIT: The attached fix works for Veeam B&R 9.5 Update 4b as well.

[DerOest]

Just a heads up, the hotfix for 9.5 Update 4b is now available in that KB, too:
https://www.veeam.com/kb2975

[nathano]

Hello folks,

KB2975 is ready and contains the link to the hotfix. The fix is fully tested against Veeam B&R 9.5 Update 4a. Cheers!

EDIT: The attached fix works for Veeam B&R 9.5 Update 4b as well.

I logged case to check and was told there was no hotfix for 4b yet (case 03442611).

Can you please clarify as I would like to update to 4b.

Thanks
Nathan

[DerOest]

nathano, there is a download link for the 4b fix:

Just a heads up, the hotfix for 9.5 Update 4b is now available in that KB, too:
https://www.veeam.com/kb2975

[Dima P.]

Hello Nathan,

Indeed the hot fix for U4b is available at the mentioned KB article. I am also trying to find out why the misleading information was provided by support team but the case you mentioned was related to Update 3 and not Update 4. Is there any other case ID I should review?

Thank you in advance!

[holzhei]

We are still having issues with virtual synthetic fulls not getting backuped to tape intermittently when the source is a backup copy job. Case number 03651953. We have been recommended to recreate the backup copy jobs, which we have done in the past with out success.

Is there any chance we could get this escalated.

The last reply they hinted that the following could be the reason.

"If the source backup chain contains a full backup that was created on the virtual full day (for example, an active full backup created manually), the tape job will copy it and will not create a virtual full."
https://helpcenter.veeam.com/docs/backu ... l?ver=95u4

I don't believe this should be the case as the backup copy jobs are always forever forward incrementals, and the full backup that they talk about is just the result of a merge operation, not an active or synthetic full operation. Edit: Also this would happen every day with forever forward incrementals (oldest via being merged into the vbk).

Note: we don't have kb2975 installed after the 4b update. I believe it was for a different issue, and we did have this fix installed for u4a and it still did not correct the issue we are having.

We also have KB2981 installed.

Cheers,
Michael

[Dima P.]

Hello Michael,

I've asked experienced support engineers to review your case details. Stay tuned.

[holzhei]

Awesome, thanks Dima.

[nathano]

I haven't done the 4b update yet, I'd really like to but a little worried about our issue occurring again.