Does B&R 9.5 U2 contain kb1985 fix?

[Lewpy]

Subject says it all

Does the recently released B&R 9.5 Update 2 contain the workaround/fix from kb1985? https://www.veeam.com/kb1985

I couldn't find any mention in the release notes, but I am sure there are many smaller fixes not documented as fixed https://www.veeam.com/kb2283

Thanks

[Lewpy]

Although DDOS v6.0.1.10 corrects the issue reported in "ETA 498457: Data Domain DD OS 6.0: File segments may be unexpectedly deleted during the DD OS cleaning cycle, which may result in data loss", which I assume is the issue kb1985 is referring to?
I have already upgraded the customer (for whom I installed kb1985) to DDOS v6.0.1.10, so hopefully that covers them

[ndum]

Hi,

Is there already a news about it?
Updating the Data Domains is not for everybody possible and this is still a huge issue.

Thanks

[foggy]

The actual issue here is on Dell/EMC side and should be fixed in DD OS firmware. Even with kb1985 hotfix applied, you'd better schedule the jobs so that they do not overlap with the DD OS cleaning process. I cannot comment on whether the mentioned fix in DDOS v6.0.1.10 addresses this very issue, better contact Dell/EMC for confirmation. Thanks.

[dustinn3]

I'm concerned about this as well. I've upgraded to DDOS V6.0.1.10, but you recommend scheduling my jobs around the cleaning cycle. So if it takes around 12 hours to do a clean on Tuesday at 6am and I go into my copy jobs and disable the transfer schedule from 6am - 7pm on Tuesday I should be good then?

Thanks,

[foggy]

Yes. However, I recommend to first check whether the DD OS version you have addresses the issue - this would allow you to avoid the reconfiguration.

[ndum]

Okay it seems that we also need to update the data domains.

This EMC Note is also very bad....

Code: Select all

EMC Note 498457 recently announced – Segments ingested while the unit is in a cleanup cycle may be deleted, as the unit incorrectly marks them as invalid.  You would not know you lost these segments until you attempted to restore!

Affected versions:  DD OS 6.0, DD OS 6.0 6.0.0.1, DD OS 6.0 6.0.0.2, DD OS 6.0 6.0.0.20, DD OS 6.0 6.0.0.21, DD OS 6.0 6.0.0.30, DD OS 6.0 6.0.0.5, DD OS 6.0 6.0.0.9, DD Virtual Edition 3 & 3 SP1 All Editions

Is DD OS 6.0.1.10 fully supported by Veeam?

Thanks for the information.

[ndum]

Ahh the Note to ETA 498457 is already posted... Sorry i'm confused

[foggy]

Is DD OS 6.0.1.10 fully supported by Veeam?

Yes, minor updates are typically supported by default.

[layte]

I'm running 6.0.1.0-556307 so looking at the note I take it I do not need to rush into updating to 6.0.1.10

Heavens forbid EMC make finding patch and fix notes not an exercise in annoyance and failure.

[Lewpy]

Dell/EMC have updated the release notes for v6.0.1.10 to Rev 02, and have now included the following under "Fixed Issues"

Issue 183969 - This release resolves a race condition between ingest and Garbage Collection (GC) which could cause newly ingested segments to be mistakenly deleted, resulting in data loss and a file system panic.

Foggy - while I appreciate your stance that it is up to Dell/EMC to say whether the issue is fixed or not, contacting Dell/EMC to ask whether they have resolved an issue highlighted by a 3rd party in a 3rd party KB article won't really get far
Now, I assume that either Veeam discovered the issue and discussed it with Dell/EMC, or where informed of the issue by Dell/EMC and instigated the workaround. If so, was Veeam issued with an "Issue Number" by Dell/EMC to define/document the issue? And can that be shared? If so, it might match the above Issue number, or I can approach Dell/EMC and ask directly if a particular Issue is resolved (which is more likely to get an answer).

Thanks

[foggy]

Hi Lewis, I was given the 180010 number by our support techs.

[Lewpy]

Thanks foggy
I've raised an SR with Dell/EMC, and I'll post anything pertinent from their reply.

[Lewpy]

I have had a reply back from Dell/EMC

I just found bug 180010 and from the notes is a duplicate of 183969 which have been addressed on the latest DDOS code.

Updating to v6.0.1.10 resolves the issue that kb1985 addressed, so upgrading to B&R v9.5U2 should be okay
This doesn't help those who can't update to latest DDOS: you either need the hotfix (on B&R v9.5U1), or carefully manage your cleaning cycle timings
But I can say that having upgraded to DDOS 6 with Veeam 9.5 Update 1, I have seen much improved performance around merges, etc.

[foggy]

Thanks for checking, Lewis. Much appreciated.

[BGA-Robert]

Glad I stumbled across this thread. I may be having this issue.

I didn't see an answer to the original question. Is KB1985 included in v9.5U2?

What errors would I be seeing if I am having this issue?

How about:
Failed to compact full backup file Error: Agent: Failed to process method {Transform.CompileFIB}: An unexpected network error occurred. Failed to read data from the file
Failed to generate points Error: Agent: Failed to process method {Transform.CompileFIB}: An unexpected network error occurred. Failed to read data from the file
Failed to compact full backup file Error: Agent: Failed to process method {Transform.CompileFIB}: bad allocation
Failed to generate points Error: Agent: Failed to process method {Transform.CompileFIB}: bad allocation

All where generated during a clean cycle...

This appears to be also breaking compacts similar to this thread:
cloud-connect-backup-f43/back-copy-jobs ... 40420.html
I've had a few .VBK files stop compacting. They seem to be related to similar errors, meaning I'll see some of the above errors, then the backup copy jobs will stop attempting to compacting. I have the work-around fix, but it's a real nuisance.

Thanks,

Robert

[Lewpy]

I can't answer for Veeam, but what I have picked up on is that Veeam don't want to issue workaround/fixes for an issue in a 3rd party product that has now been fixed by the 3rd party.
So, when kb1985 came out, there was no fix from Dell/EMC for the issue.
We now know that DDOS v6.0.1.10 resolves the issue that Veeam issued the fix for, so Veeam can [quite rightly] say "install the Dell/EMC firmware update to resolve the issue".

The issue only affects users running DDOS 6 (lower than v6.0.1.10), according the Dell/EMC ETA 498457 (top Dell/EMC Support Searching: use the Product "Data Domain OS" or "DDOS" rather than your actual Data Domain appliance when searching, and you will find the actual notes and firmware you need).

DDOS 6 is still not ratified by EMC as an "EMC Data Domain Target Code" version yet, so you are getting ahead of the support curve installing it (I went for it based on the level of integration that Veeam has with DDOS 6, and the enhanced performance). So if you are running DDOS 6, then upgrading to the latest release isn't altering your support standing with Dell/EMC.

In fact, all previous releases of DDOS 6 have been deprecated on the Dell/EMC support site, if you try to download them: they link you directly to v6.0.1.10 now.

So, if you are a Data Domain user running DDOS 6, then you really should upgrade to v6.0.1.10 at your earliest opportunity.
This also makes the need for a Veeam workaround fix redundant.

Note: I've read some other Dell/EMC support articles that are referencing v6.1 code, so expect another minor upgrade release for DDOS 6 sometime soon[ish]

[foggy]

Hi Robert, Lewis is correct, if you're seeing the same issue, the only way to avoid it is to install the mentioned DD OS update, since the hotfix described in Veeam KB1985 doesn't address it. We supposed that adding some additional checks/file closings would work around the issue in DD OS, however it didn't.

To verify whether you're seeing this very issue, please contact either Veeam or Dell/EMC support.

[Gostev]

But to answer the original question, KB1985 (bug ID 95715) is in fact a part of 9.5U2, because EMC suggested to use the approach implemented in the original fix as the best practice. Hopefully this will prevent future issues as they update or refactor DD OS code.