Discussions specific to tape backups
holzhei
Novice
Posts: 4
Liked: 1 time
Joined: Mar 13, 2019 10:50 pm
Full Name: Michael Holzheimer
Contact:

Virtual full not working after upgrade to 9.5u4

Post by holzhei » Mar 13, 2019 11:36 pm 1 person likes this post

On Saturdays we schedule a full tape backup and then have incrementals to tape Sunday, Monday, Tuesday, Wednesday, Thursday, Friday. The source jobs are 10x backup jobs, and 2x backup copies. The 10x backup jobs have active fulls run every Saturday, which in turn forces a full backup to tape every Saturday (and the virtual full setting is ignored), and then incremental are backed up to tape every other day. The 2x backup copies are forever forward, and from my understanding should have a virtual full backup (vbm) created on the tape full backup day (Saturday), and incrementals are backed up to tape Sunday, Monday, Tuesday, Wednesday, Thursday, Friday. This was working for us on 9.5u3.

Since upgrading to 9.5u4 a few weeks ago, the virtual fulls are not created reliably to tape for the 2x backup copy jobs. Some times the are created on the Sunday run, other times they are not created at all.

For instance this week one of the backups copy jobs had the virtual full created on Sunday, and the other one has still not created a virtual full, only incrementals on Saturday, Sunday, Monday, Tuesday, Wednesday.

Any hints on where I should be looking?

Case number is 03432873

HannesK
Veeam Software
Posts: 3698
Liked: 441 times
Joined: Sep 01, 2014 11:46 am
Location: Austria
Contact:

Re: Virtual full not working after upgrade to 9.5u4

Post by HannesK » Mar 14, 2019 7:15 am

Hello,
and welcome to the forums.

As you say that you did not change anything and it worked with U3, the only thing I can recommend is to continue working with support. Also your observation with "sometimes they are created, sometimes not" seems strange to me.

There seems to be a typo in the case number, as the stated case was closed with "no answer from customer" some time ago.

Best regards,
Hannes

nathano
Veeam Vanguard
Posts: 55
Liked: 1 time
Joined: Sep 05, 2016 5:08 am
Full Name: Nathan Oldfield
Contact:

Re: Virtual full not working after upgrade to 9.5u4

Post by nathano » Mar 26, 2019 7:46 am

That sounds like a problem I have, since U4 getting synthetic fulls to tape for one of my backup copy jobs is hit and miss. I have 2 BCJs and several local full backups. the local backups generate a synthetic full to tape each night but the BCJs are the problem.

my case# is 03442611

I will update my case and reference your case ID

holzhei
Novice
Posts: 4
Liked: 1 time
Joined: Mar 13, 2019 10:50 pm
Full Name: Michael Holzheimer
Contact:

Re: Virtual full not working after upgrade to 9.5u4

Post by holzhei » Apr 01, 2019 12:24 am

We logged two cases, but got similar responses each time saying that this is normal as the incremental is locked. Previously it would create a virtual full with the latest available incremental (which may have been the previous one if the latest was locked, and we were okay with that). Looks like u4a was just released with reliability fixes for tape, although it does not go into much detail.

After not being able to get the normal tape backups with incremental working, we switched to a GFS media pool which now allows daily incremental since the u4 upgrade. Although we aren't able to schedule it to run after our last backup job like we could with a normal tape job.

HannesK
Veeam Software
Posts: 3698
Liked: 441 times
Joined: Sep 01, 2014 11:46 am
Location: Austria
Contact:

Re: Virtual full not working after upgrade to 9.5u4

Post by HannesK » Apr 01, 2019 6:46 am

@holzhei: it would help me to search for details if you could post the case numbers here. Nathano is currently testing U4a but I cannot see a case update yet (is "on hold")

nathano
Veeam Vanguard
Posts: 55
Liked: 1 time
Joined: Sep 05, 2016 5:08 am
Full Name: Nathan Oldfield
Contact:

Re: Virtual full not working after upgrade to 9.5u4

Post by nathano » Apr 08, 2019 4:05 am

I'm still working with support on the issue.. next thing I am trying is this.. will see if it backs up OK tonight.

"
Hi Please remove the backup copy job from the configuration and re-import the backup.

1. Open the Home view.
2. Disable the backup copy job.
3. In the inventory pane, select backups-> disk(copy).
4. In the working area, right-click the backup copy job ->right click->select "remove from configuration".
5. select Backup infrastructure view-> rescan backup copy repository
6. The backup files will appear under backups-> disk imported.
7.Edit the backup copy job->target->map backup->select the backup copy job.
8. This will move the backup files from disk imported -> disk(copy).
9. Edit the tape job->remove the backup copy job and re-add.
10. Run the backup tape job.
"

nathano
Veeam Vanguard
Posts: 55
Liked: 1 time
Joined: Sep 05, 2016 5:08 am
Full Name: Nathan Oldfield
Contact:

Re: Virtual full not working after upgrade to 9.5u4

Post by nathano » Apr 11, 2019 10:27 pm

the above steps looked like it fixed the problem, but no, last night both BCJs were left off the tape.

This is getting ridiculous.

Dima P.
Product Manager
Posts: 10328
Liked: 837 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Virtual full not working after upgrade to 9.5u4

Post by Dima P. » Apr 12, 2019 7:34 pm

Hello Nathan,

I've asked QA folks to review your case. I'll let you know once I hear back anything. Cheers!

nathano
Veeam Vanguard
Posts: 55
Liked: 1 time
Joined: Sep 05, 2016 5:08 am
Full Name: Nathan Oldfield
Contact:

Re: Virtual full not working after upgrade to 9.5u4

Post by nathano » Apr 15, 2019 9:34 pm

Dima

I hope there's a resolution soon, our backup to tape is a critical part of our backup strategy, you know, 3-2-1 etc..

There should not be a green tick next to a component of the job that failed to backup any data, as you can see in this screen shot, it failed to capture any data for one of the requested backup to tape components and it still ticks it off as green! Crazy stuff, this has to be a warning, can this please be logged as a feature request.

Image

sweepsd
Influencer
Posts: 16
Liked: never
Joined: Mar 23, 2019 2:41 pm
Contact:

Re: Virtual full not working after upgrade to 9.5u4

Post by sweepsd » Apr 16, 2019 9:08 am

I'm having a similar issue - although I am new to Veeam so not sure if it would have worked in the past. I cannot get a backup copy to create a synthetic full for a specific day, all it does is just copy the oldest full to the tape. If I alter the same job using the same media pool and choose a VM with a backup source (rather than a backup copy) it creates the synthetic full as expected.

I've moved from Backup exec to Veeam and beginning to pull my hair out over this product! I have logged a ticket 035142 but would like some help from anyone here if they can. I am running 9.5u4a

Andy

Dima P.
Product Manager
Posts: 10328
Liked: 837 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Virtual full not working after upgrade to 9.5u4

Post by Dima P. » Apr 16, 2019 5:53 pm 1 person likes this post

Was told that this behavior looks like a known issue and we already working on a private fix. Please keep in touch with your support engineer. Please let me know how it goes!

sweepsd
Influencer
Posts: 16
Liked: never
Joined: Mar 23, 2019 2:41 pm
Contact:

Re: Virtual full not working after upgrade to 9.5u4

Post by sweepsd » Apr 26, 2019 10:59 am

Hi
I was given a hotfix after the techie said there was a bug, but it hasn't resolved the issue. If I run 2 identical jobs but with different sources - backup and backup copy, the backup source works as expected. The backup copy always just copies the oldest full to tape.

HannesK
Veeam Software
Posts: 3698
Liked: 441 times
Joined: Sep 01, 2014 11:46 am
Location: Austria
Contact:

Re: Virtual full not working after upgrade to 9.5u4

Post by HannesK » Jun 05, 2019 11:00 am

Hello,
finally there is an hotfix available fixing the issue. If you run into it, please contact support.

Best regards,
Hannes

nathano
Veeam Vanguard
Posts: 55
Liked: 1 time
Joined: Sep 05, 2016 5:08 am
Full Name: Nathan Oldfield
Contact:

Re: Virtual full not working after upgrade to 9.5u4

Post by nathano » Jun 05, 2019 11:19 am

yes I got the hotfix last week and so far so good. it took a lot longer than I would have liked but at least I have confidence in the tape jobs again..

I'll have to test a scenario where there's no new restore points and see if I get a warning on the job as I would expect.

lyapkost
Veeam Software
Posts: 170
Liked: 27 times
Joined: Nov 27, 2015 2:26 pm
Full Name: Konstantin
Location: Saint Petersburg
Contact:

Re: Virtual full not working after upgrade to 9.5u4

Post by lyapkost » Jun 06, 2019 10:04 am

Hi Nathan, thanks for the feedback. Keep in mind that we're not planning to include this fix to the upcoming minor update 4b. So in case you're going to install it (for example to support Linux 5.0 kernel in your Agent backups) please contact support to have an updated dll. It should be ready within 1-2 weeks from the release date.

joespirit88
Enthusiast
Posts: 36
Liked: 5 times
Joined: Jul 04, 2017 12:53 pm
Full Name: Joe Spirit
Contact:

Re: Virtual full not working after upgrade to 9.5u4

Post by joespirit88 » Jun 14, 2019 8:37 am 2 people like this post

We also ran into the same issue but the hotfix looks to have fixed this for us.
nathano wrote:
Apr 15, 2019 9:34 pm
There should not be a green tick next to a component of the job that failed to backup any data, as you can see in this screen shot, it failed to capture any data for one of the requested backup to tape components and it still ticks it off as green! Crazy stuff, this has to be a warning, can this please be logged as a feature request.
100% agree, I raised this with my support engineer who said this needs to be raised a feature request. If i schedule a backup job i expect it to back up data.
We have holes in our backup data because this issue went unnoticed for a few weeks due to it not being seen as a problem, it was only because i was poking around some settings i thought something was off.
With backup copies i get warnings if no new safesets were detected during a run, why would this be different for tapes?
Also why would this hotfix not be included in the next release? Surely as a backup product if it is not performing backups and not informing the administrators this is a major issue?

Gostev
SVP, Product Management
Posts: 24473
Liked: 3413 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Virtual full not working after upgrade to 9.5u4

Post by Gostev » Jun 25, 2019 4:18 pm 1 person likes this post

It's a dangerous fix to include in an update, since it touches many components. It will for sure be included in the next major release though. Thanks!

joespirit88
Enthusiast
Posts: 36
Liked: 5 times
Joined: Jul 04, 2017 12:53 pm
Full Name: Joe Spirit
Contact:

Re: Virtual full not working after upgrade to 9.5u4

Post by joespirit88 » Jun 26, 2019 8:16 am

Thanks for the response Gostev however I have a couple more questions:

Will there be a reference for us when we contact support to have the hotfix for the new minor updates or a KB we can refer detailing the bug and letting us know that the hotfix is available for that release?

As we have pointed out Veeam is incorrectly saying that tape backups are successful when they are not, will this also be fixed?

Dima P.
Product Manager
Posts: 10328
Liked: 837 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Virtual full not working after upgrade to 9.5u4

Post by Dima P. » Jun 26, 2019 8:47 am

Will there be a reference for us when we contact support to have the hotfix for the new minor updates or a KB we can refer detailing the bug and letting us know that the hotfix is available for that release?
You can use this thread or your previous case for the reference, I'll discuss the KB article with the link to the fix with our support team as well. The fix will be rebuild right after U4b goes live, so it would be possible to installed again after applying the update.
As we have pointed out Veeam is incorrectly saying that tape backups are successful when they are not, will this also be fixed?
Yes, that is going to be fixed as well. Our goal was to fix the root cause on the first place and fix the issues with the backup, once the issue is fixed - reporting will work correctly.

DerOest
Enthusiast
Posts: 40
Liked: 5 times
Joined: Oct 30, 2015 10:10 am
Contact:

Re: Virtual full not working after upgrade to 9.5u4

Post by DerOest » Jun 26, 2019 11:26 am

Case # 03630614
I think we hit the same or similar problem, but did not get far with support yet ("lets see how it works next saturday").

Last saturday, only 1 or 2 of 13 Tape jobs picked up the GFS-copies that should go to tape.
The other jobs all were "successful", but had "no new files found" and didn't back up anything - which is dangerously misleading! see above, "holes in our backup data"

Forcing a "Weekly" on those jobs then worked flawless and they picked up the GFS as they should.


Looks like this issue is more widespread than expected and should be covered better than "not going into U4b, request a new private fix (that not every support staff knows about)".
We've also got several other tape issues, it would be nice if there was some kind of a preview-changelog for the tape-fixes in 4b. Would be a waste if i open several other tickets and that's all going to be fixed soon anyway.

Dima P.
Product Manager
Posts: 10328
Liked: 837 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Virtual full not working after upgrade to 9.5u4

Post by Dima P. » Jun 26, 2019 4:09 pm

Hello DerOest,

I am sorry to hear it. I've reached your support engineer and asked to review the case details to make sure that you are affected by this issue. If that's the case support team will share a private fix with you.
it would be nice if there was some kind of a preview-changelog for the tape-fixes in 4b.
We share the list of fixed issues together with the new version, so we will have it publicly available when U4b goes live.

Dima P.
Product Manager
Posts: 10328
Liked: 837 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Virtual full not working after upgrade to 9.5u4

Post by Dima P. » Jul 04, 2019 6:04 pm 1 person likes this post

Hello folks,

KB2975 is ready and contains the link to the hotfix. The fix is fully tested against Veeam B&R 9.5 Update 4a. Cheers!

EDIT: The attached fix works for Veeam B&R 9.5 Update 4b as well.

DerOest
Enthusiast
Posts: 40
Liked: 5 times
Joined: Oct 30, 2015 10:10 am
Contact:

Re: Virtual full not working after upgrade to 9.5u4

Post by DerOest » Jul 13, 2019 8:33 am

Just a heads up, the hotfix for 9.5 Update 4b is now available in that KB, too:
https://www.veeam.com/kb2975

nathano
Veeam Vanguard
Posts: 55
Liked: 1 time
Joined: Sep 05, 2016 5:08 am
Full Name: Nathan Oldfield
Contact:

Re: Virtual full not working after upgrade to 9.5u4

Post by nathano » Jul 25, 2019 11:34 pm

Dima P. wrote:
Jul 04, 2019 6:04 pm
Hello folks,

KB2975 is ready and contains the link to the hotfix. The fix is fully tested against Veeam B&R 9.5 Update 4a. Cheers!

EDIT: The attached fix works for Veeam B&R 9.5 Update 4b as well.
I logged case to check and was told there was no hotfix for 4b yet (case 03442611).

Can you please clarify as I would like to update to 4b.

Thanks
Nathan

DerOest
Enthusiast
Posts: 40
Liked: 5 times
Joined: Oct 30, 2015 10:10 am
Contact:

Re: Virtual full not working after upgrade to 9.5u4

Post by DerOest » Jul 26, 2019 6:50 am

nathano, there is a download link for the 4b fix:
DerOest wrote:
Jul 13, 2019 8:33 am
Just a heads up, the hotfix for 9.5 Update 4b is now available in that KB, too:
https://www.veeam.com/kb2975

Dima P.
Product Manager
Posts: 10328
Liked: 837 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Virtual full not working after upgrade to 9.5u4

Post by Dima P. » Jul 26, 2019 10:19 am

Hello Nathan,

Indeed the hot fix for U4b is available at the mentioned KB article. I am also trying to find out why the misleading information was provided by support team but the case you mentioned was related to Update 3 and not Update 4. Is there any other case ID I should review?

Thank you in advance!

holzhei
Novice
Posts: 4
Liked: 1 time
Joined: Mar 13, 2019 10:50 pm
Full Name: Michael Holzheimer
Contact:

Re: Virtual full not working after upgrade to 9.5u4

Post by holzhei » Jul 31, 2019 11:00 pm

We are still having issues with virtual synthetic fulls not getting backuped to tape intermittently when the source is a backup copy job. Case number 03651953. We have been recommended to recreate the backup copy jobs, which we have done in the past with out success.

Is there any chance we could get this escalated.

The last reply they hinted that the following could be the reason.

"If the source backup chain contains a full backup that was created on the virtual full day (for example, an active full backup created manually), the tape job will copy it and will not create a virtual full."
https://helpcenter.veeam.com/docs/backu ... l?ver=95u4
I don't believe this should be the case as the backup copy jobs are always forever forward incrementals, and the full backup that they talk about is just the result of a merge operation, not an active or synthetic full operation. Edit: Also this would happen every day with forever forward incrementals (oldest via being merged into the vbk).

Note: we don't have kb2975 installed after the 4b update. I believe it was for a different issue, and we did have this fix installed for u4a and it still did not correct the issue we are having.

We also have KB2981 installed.

Cheers,
Michael

Dima P.
Product Manager
Posts: 10328
Liked: 837 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Virtual full not working after upgrade to 9.5u4

Post by Dima P. » Aug 01, 2019 12:59 pm

Hello Michael,

I've asked experienced support engineers to review your case details. Stay tuned.

holzhei
Novice
Posts: 4
Liked: 1 time
Joined: Mar 13, 2019 10:50 pm
Full Name: Michael Holzheimer
Contact:

Re: Virtual full not working after upgrade to 9.5u4

Post by holzhei » Aug 01, 2019 10:11 pm

Awesome, thanks Dima.

nathano
Veeam Vanguard
Posts: 55
Liked: 1 time
Joined: Sep 05, 2016 5:08 am
Full Name: Nathan Oldfield
Contact:

Re: Virtual full not working after upgrade to 9.5u4

Post by nathano » Aug 08, 2019 6:04 am

I haven't done the 4b update yet, I'd really like to but a little worried about our issue occurring again.

Post Reply

Who is online

Users browsing this forum: No registered users and 3 guests