Comprehensive data protection for all workloads
Post Reply
Lewpy
Enthusiast
Posts: 66
Liked: 15 times
Joined: Nov 27, 2012 1:00 pm
Full Name: Lewis Berrie
Location: Southern England
Contact:

Does B&R 9.5 U2 contain kb1985 fix?

Post by Lewpy »

Subject says it all :)

Does the recently released B&R 9.5 Update 2 contain the workaround/fix from kb1985? https://www.veeam.com/kb1985

I couldn't find any mention in the release notes, but I am sure there are many smaller fixes not documented as fixed https://www.veeam.com/kb2283

Thanks :)
Lewpy
Enthusiast
Posts: 66
Liked: 15 times
Joined: Nov 27, 2012 1:00 pm
Full Name: Lewis Berrie
Location: Southern England
Contact:

Re: Does B&R 9.5 U2 contain kb1985 fix?

Post by Lewpy »

Although DDOS v6.0.1.10 corrects the issue reported in "ETA 498457: Data Domain DD OS 6.0: File segments may be unexpectedly deleted during the DD OS cleaning cycle, which may result in data loss", which I assume is the issue kb1985 is referring to?
I have already upgraded the customer (for whom I installed kb1985) to DDOS v6.0.1.10, so hopefully that covers them :)
ndum
Influencer
Posts: 14
Liked: 4 times
Joined: Jan 30, 2017 11:17 am
Location: Bern, Switzerland
Contact:

Re: Does B&R 9.5 U2 contain kb1985 fix?

Post by ndum »

Hi,

Is there already a news about it?
Updating the Data Domains is not for everybody possible and this is still a huge issue.

Thanks :D
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Does B&R 9.5 U2 contain kb1985 fix?

Post by foggy »

The actual issue here is on Dell/EMC side and should be fixed in DD OS firmware. Even with kb1985 hotfix applied, you'd better schedule the jobs so that they do not overlap with the DD OS cleaning process. I cannot comment on whether the mentioned fix in DDOS v6.0.1.10 addresses this very issue, better contact Dell/EMC for confirmation. Thanks.
dustinn3
Influencer
Posts: 22
Liked: 6 times
Joined: Oct 14, 2013 1:53 pm
Full Name: Dustin Newby
Contact:

Re: Does B&R 9.5 U2 contain kb1985 fix?

Post by dustinn3 »

I'm concerned about this as well. I've upgraded to DDOS V6.0.1.10, but you recommend scheduling my jobs around the cleaning cycle. So if it takes around 12 hours to do a clean on Tuesday at 6am and I go into my copy jobs and disable the transfer schedule from 6am - 7pm on Tuesday I should be good then?

Thanks,
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Does B&R 9.5 U2 contain kb1985 fix?

Post by foggy »

Yes. However, I recommend to first check whether the DD OS version you have addresses the issue - this would allow you to avoid the reconfiguration.
ndum
Influencer
Posts: 14
Liked: 4 times
Joined: Jan 30, 2017 11:17 am
Location: Bern, Switzerland
Contact:

Re: Does B&R 9.5 U2 contain kb1985 fix?

Post by ndum »

Okay it seems that we also need to update the data domains.

This EMC Note is also very bad....

Code: Select all

EMC Note 498457 recently announced – Segments ingested while the unit is in a cleanup cycle may be deleted, as the unit incorrectly marks them as invalid.  You would not know you lost these segments until you attempted to restore!

Affected versions:  DD OS 6.0, DD OS 6.0 6.0.0.1, DD OS 6.0 6.0.0.2, DD OS 6.0 6.0.0.20, DD OS 6.0 6.0.0.21, DD OS 6.0 6.0.0.30, DD OS 6.0 6.0.0.5, DD OS 6.0 6.0.0.9, DD Virtual Edition 3 & 3 SP1 All Editions
Is DD OS 6.0.1.10 fully supported by Veeam?

Thanks for the information.
ndum
Influencer
Posts: 14
Liked: 4 times
Joined: Jan 30, 2017 11:17 am
Location: Bern, Switzerland
Contact:

Re: Does B&R 9.5 U2 contain kb1985 fix?

Post by ndum »

Ahh the Note to ETA 498457 is already posted... Sorry i'm confused ;)
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Does B&R 9.5 U2 contain kb1985 fix?

Post by foggy »

ndum wrote:Is DD OS 6.0.1.10 fully supported by Veeam?
Yes, minor updates are typically supported by default.
layte
Lurker
Posts: 2
Liked: never
Joined: Mar 24, 2010 10:08 am
Contact:

Re: Does B&R 9.5 U2 contain kb1985 fix?

Post by layte »

I'm running 6.0.1.0-556307 so looking at the note I take it I do not need to rush into updating to 6.0.1.10

Heavens forbid EMC make finding patch and fix notes not an exercise in annoyance and failure.
Lewpy
Enthusiast
Posts: 66
Liked: 15 times
Joined: Nov 27, 2012 1:00 pm
Full Name: Lewis Berrie
Location: Southern England
Contact:

Re: Does B&R 9.5 U2 contain kb1985 fix?

Post by Lewpy »

Dell/EMC have updated the release notes for v6.0.1.10 to Rev 02, and have now included the following under "Fixed Issues"
Issue 183969 - This release resolves a race condition between ingest and Garbage Collection (GC) which could cause newly ingested segments to be mistakenly deleted, resulting in data loss and a file system panic.
Foggy - while I appreciate your stance that it is up to Dell/EMC to say whether the issue is fixed or not, contacting Dell/EMC to ask whether they have resolved an issue highlighted by a 3rd party in a 3rd party KB article won't really get far :cry:
Now, I assume that either Veeam discovered the issue and discussed it with Dell/EMC, or where informed of the issue by Dell/EMC and instigated the workaround. If so, was Veeam issued with an "Issue Number" by Dell/EMC to define/document the issue? And can that be shared? If so, it might match the above Issue number, or I can approach Dell/EMC and ask directly if a particular Issue is resolved (which is more likely to get an answer).

Thanks :D
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Does B&R 9.5 U2 contain kb1985 fix?

Post by foggy » 1 person likes this post

Hi Lewis, I was given the 180010 number by our support techs.
Lewpy
Enthusiast
Posts: 66
Liked: 15 times
Joined: Nov 27, 2012 1:00 pm
Full Name: Lewis Berrie
Location: Southern England
Contact:

Re: Does B&R 9.5 U2 contain kb1985 fix?

Post by Lewpy » 1 person likes this post

Thanks foggy :)
I've raised an SR with Dell/EMC, and I'll post anything pertinent from their reply.
Lewpy
Enthusiast
Posts: 66
Liked: 15 times
Joined: Nov 27, 2012 1:00 pm
Full Name: Lewis Berrie
Location: Southern England
Contact:

Re: Does B&R 9.5 U2 contain kb1985 fix?

Post by Lewpy » 1 person likes this post

I have had a reply back from Dell/EMC
I just found bug 180010 and from the notes is a duplicate of 183969 which have been addressed on the latest DDOS code.
Updating to v6.0.1.10 resolves the issue that kb1985 addressed, so upgrading to B&R v9.5U2 should be okay :)
This doesn't help those who can't update to latest DDOS: you either need the hotfix (on B&R v9.5U1), or carefully manage your cleaning cycle timings :?
But I can say that having upgraded to DDOS 6 with Veeam 9.5 Update 1, I have seen much improved performance around merges, etc. 8)
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Does B&R 9.5 U2 contain kb1985 fix?

Post by foggy » 1 person likes this post

Thanks for checking, Lewis. Much appreciated.
BGA-Robert
Service Provider
Posts: 60
Liked: 8 times
Joined: Feb 03, 2016 5:06 pm
Full Name: Robert Wakefield
Contact:

Re: Does B&R 9.5 U2 contain kb1985 fix?

Post by BGA-Robert »

Glad I stumbled across this thread. I may be having this issue.

I didn't see an answer to the original question. Is KB1985 included in v9.5U2?

What errors would I be seeing if I am having this issue?

How about:
Failed to compact full backup file Error: Agent: Failed to process method {Transform.CompileFIB}: An unexpected network error occurred. Failed to read data from the file
Failed to generate points Error: Agent: Failed to process method {Transform.CompileFIB}: An unexpected network error occurred. Failed to read data from the file
Failed to compact full backup file Error: Agent: Failed to process method {Transform.CompileFIB}: bad allocation
Failed to generate points Error: Agent: Failed to process method {Transform.CompileFIB}: bad allocation

All where generated during a clean cycle...

This appears to be also breaking compacts similar to this thread:
cloud-connect-backup-f43/back-copy-jobs ... 40420.html
I've had a few .VBK files stop compacting. They seem to be related to similar errors, meaning I'll see some of the above errors, then the backup copy jobs will stop attempting to compacting. I have the work-around fix, but it's a real nuisance.

Thanks,

Robert
Lewpy
Enthusiast
Posts: 66
Liked: 15 times
Joined: Nov 27, 2012 1:00 pm
Full Name: Lewis Berrie
Location: Southern England
Contact:

Re: Does B&R 9.5 U2 contain kb1985 fix?

Post by Lewpy »

I can't answer for Veeam, but what I have picked up on is that Veeam don't want to issue workaround/fixes for an issue in a 3rd party product that has now been fixed by the 3rd party.
So, when kb1985 came out, there was no fix from Dell/EMC for the issue.
We now know that DDOS v6.0.1.10 resolves the issue that Veeam issued the fix for, so Veeam can [quite rightly] say "install the Dell/EMC firmware update to resolve the issue".

The issue only affects users running DDOS 6 (lower than v6.0.1.10), according the Dell/EMC ETA 498457 (top Dell/EMC Support Searching: use the Product "Data Domain OS" or "DDOS" rather than your actual Data Domain appliance when searching, and you will find the actual notes and firmware you need).

DDOS 6 is still not ratified by EMC as an "EMC Data Domain Target Code" version yet, so you are getting ahead of the support curve installing it (I went for it based on the level of integration that Veeam has with DDOS 6, and the enhanced performance). So if you are running DDOS 6, then upgrading to the latest release isn't altering your support standing with Dell/EMC.

In fact, all previous releases of DDOS 6 have been deprecated on the Dell/EMC support site, if you try to download them: they link you directly to v6.0.1.10 now.

So, if you are a Data Domain user running DDOS 6, then you really should upgrade to v6.0.1.10 at your earliest opportunity.
This also makes the need for a Veeam workaround fix redundant.

Note: I've read some other Dell/EMC support articles that are referencing v6.1 code, so expect another minor upgrade release for DDOS 6 sometime soon[ish] :)
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Does B&R 9.5 U2 contain kb1985 fix?

Post by foggy » 1 person likes this post

Hi Robert, Lewis is correct, if you're seeing the same issue, the only way to avoid it is to install the mentioned DD OS update, since the hotfix described in Veeam KB1985 doesn't address it. We supposed that adding some additional checks/file closings would work around the issue in DD OS, however it didn't.

To verify whether you're seeing this very issue, please contact either Veeam or Dell/EMC support.
Gostev
Chief Product Officer
Posts: 31457
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Does B&R 9.5 U2 contain kb1985 fix?

Post by Gostev » 1 person likes this post

But to answer the original question, KB1985 (bug ID 95715) is in fact a part of 9.5U2, because EMC suggested to use the approach implemented in the original fix as the best practice. Hopefully this will prevent future issues as they update or refactor DD OS code.
Post Reply

Who is online

Users browsing this forum: ybarrap2003 and 242 guests