DD Boost Issues and 9.5 U3 Update

[adb98]

Can anyone clear this up for me.

Since U3 upgrade we have been having random backup and copy back job issues where they would just stop in mid stride and get stuck. I went thru support and was told to just reboot to clear it. I had to hold off on a reboot until a change window as the jobs that were stuck were test servers and it could wait. Well yesterday I had a prod copy back job that locked up and even went as far as stopping without anyone trying to stop it. It was dead stuck. We then started seeing other jobs not processing when they got started. They would just sit at 0%

Error log shows the last line as
[14.02.2018 14:22:33] <01> Info [DDBoost] Setting DDBoost credentials. DDServer: server IP removed, Storage unit: , User: boostuser

I decided to take support up on their recommendation and rebooted and then all hell broke loose. None of the jobs would take off. They would start but that was all she wrote. I put a ticket in with support 02614039. I was then told to completely uninstall the transport and installer services and sc them to ensure they were gone and then reinstall them on the main VBR server. Before installing them I was told to remove the transport folder in C:\Program Files(x86)\Veeam

All seems to be working after the reinstall by my question is this............

In the old transport folder (one support told me to delete), my DD Boost plugin (LibDDBoost.dll) had a modified date of 11/12/2016. Now the modified date shows 12/14/2017. This leaves me to believe that the plugin never got updated (or the transport) when we upgraded to U3.

Do we know if there is a bug in the U3 upgrade path were this is not getting updated like it should?

C:\Program Files (x86)\Veeam\Backup Transport\x64\ddboost

[Zew]

Interesting, I haven't even heard of this before so I did some searching.

Is this DDBoost a special API that utulized by a particular storage Vendor (Dell/EMC) in this case that utilizes storage snap shots?

[adb98]

DD Boost is utilized on EMC Backup Storage Arrays (Data Domain). Data Domain Boost distributes parts of the deduplication process to a backup server or application client, resulting in 50-percent faster backups and requiring up to 99 percent less local area network (LAN), wide area network (WAN), or storage area network (SAN) bandwidth. This is all we use and it had been working great. I can see us having issues if we have a new version Veeam that now supports DDOS (Data Domain OS) 6.1 but the LibDDBoost never got up upgraded so Veeam and what it is using for DD Boost are not on the right page.

Example would Veeam knowing it now has a way or a new way to set the DDBoost Creds but the LibDDBoost (driver) has no clue what to do with that command so we just sit.

[foggy]

Hi Aaron, DDBoost client didn't change in U3. To understand the actual reasons of this behavior, I recommend to continue looking into this with the help of support (escalate the case, if required), since this might be caused by some environmental issue.

[bpayne]

+1 here, same issue. We have a couple Data Domains. DD4500 (2), DD2500, all on DDOS 5.7.5.0 and I have had non-stop issues with Backup Copy Jobs since updating to 9.5 Update 3 (I updated about 3-4 weeks ago).

Here are my current issues with Backup Copy Jobs only being sent to Data Domain:
-Copy Jobs stuck in 'stopping' status. Trying to Disable will not work. I have to go in to Task manager and kill all the processes associated with the copy job. It will automatically start a merge after that.
-Copy Jobs with VM's stuck with status "restore point is locked by job" when clearly the Backup Job is NOT running. Disabling the job works. Then re-enable seems to get things back in order.
-Copy jobs with VM's stuck at 0% progress, the last action item "VM size: xxx" and nothing more comes after that. I in fact have had this issue twice this week, including this morning. Disabling the job works. Then re-enable seems to get things back in order.
-Sometimes the above solutions don't work. I will CLONE the backup copy job, keeping ALL settings the same and starting the new clone job seems to get things running again, at most for a few days.
-As adb98 mentioned, I also on a few occasions needed to reboot the Veeam server itself, fixing itself temporarily.

Never had any issues related to these Copy Jobs before Update 3. Everything with the Copy Jobs ran very smoothly. No changes to our Data Domain's. Sounds like we have very similar issues since updating to Update 3.

I did open a support ticket but I made the mistake of having 2 completely separate issues and I tried to "kill two birds with one stone" on the same call. This case ended up getting forgotten, partially my fault. I just opened up a new case #02615386

[Zew]

Thanks Aaron for explaining that to me. Sort of sounds like VIIA but for backup storage units instead of hypervisor directly. Sounds really cool, sorry to hear you are having issues with the technology since upgrading.

I'm simply now following along to maybe learn something new along the way.

[adb98]

bpayne glad I am not alone. We have a DD4200 and are on OS: 5.7.4.10-559104.

Interested to know what modified date your modified dates are C:\Program Files (x86)\Veeam\Backup Transport\x64\ddboost.

After reinstalling the transport mine are from December of 2017 and so far no issues. I have requested my ticket be escalated last night. We will see what we get.

[bpayne]

The date modified on my libDDboost.dll file is 11/12/2016.

I did hear back from support but they suggested recreating all my Backup Copy jobs from scratch. I cannot accept that as a solution right now, being we have thousands of VM's I'd have to start over with, plus I'd need double the amount of storage to do something like that since I can't just delete my current backups at my DR site. I asked them to look further in the logs I provided and will see what comes of that. I'm hoping a Veeam representative can further comment on this thread since in the past I have gotten good traction in the forums for issues like this.

[adb98]

I agree that is BS. That solution would really suck. If you have little change then you should be ok with space as the DD would dedup the hell out of it. Issue is going thru and creating the clones of the jobs and removing the originals from the catalog and then having to manage the retention on the originals. That would be a nightmare and very messy. Especially with 1000s of VMs

I am still having the same issue after reinstalling the transport. I had a copy back job that started at 5pm and was stuck with several VMs at 0%. Only a reboot fixes it. I would escalate your ticket to Lv2 if its not there already. Here is the latest update on my ticket

Hi ADB,

This case has been escalated and I will be assisting you. I understand the question you are asking, but I'm in the process of confirming what the expected behavior should be. I have another case with similar behavior, and I am currently trying to get this question confirmed with our development team. I'll let you know as soon as I' have more information.


Thank you,
Spencer West
Veeam Support

[foggy]

Hi Brandon, while I agree that re-creating the jobs from scratch is not a solution but a workaround, however, I cannot further comment on this issue until investigation of your and Aaron's cases is done by support engineers. They are looking into this, so let's see what they can come up with as the actual reason of the observed behavior.

[pbohlken]

Hi to all,

After doing a upgrade on a test server also proxy server. I notice that my libDDboost.dll file is still 11/12/2016. Like in the other post above.
I upgrade my DD 4200 Datadomain from 6.0.1 to 6.0.2.0-575017 a few weeks ago, because of issues in replication to other DD4200. With the Veeam 9.5 update3 i can upgrade to 6.1 i read. I am going to do this soon.

This test server does not have backup jobs on the DD4200 yet. I will test this as soon as possible to see if i have issues with the lastest Veeam 9.5 Update3.
I can do another backup test job to DD4200 with a 2nd test server before upgrading to 9.5 update3. Then upgrate to Veeam 9.5 update 3 and test if that test job still works (writing to DD4200).


Still my DD OS version is higher. I am not sure if this is a good comparison.

[pbohlken]

Quick update -> Test backup to DD4200 with the old libDDboost.dll worked fine.

Test with other test server (with Veeam 9.5 U2) also before upgrade to the DD4200 did also work fine...will do an upgrade of Veeam 9.5 to update 3 after the weekend and test again.

[tkeith]

-Copy Jobs stuck in 'stopping' status. Trying to Disable will not work. I have to go in to Task manager and kill all the processes associated with the copy job. It will automatically start a merge after that.
-Copy Jobs with VM's stuck with status "restore point is locked by job" when clearly the Backup Job is NOT running. Disabling the job works. Then re-enable seems to get things back in order.
-Copy jobs with VM's stuck at 0% progress, the last action item "VM size: xxx" and nothing more comes after that. I in fact have had this issue twice this week, including this morning. Disabling the job works. Then re-enable seems to get things back in order.
-Sometimes the above solutions don't work. I will CLONE the backup copy job, keeping ALL settings the same and starting the new clone job seems to get things running again, at most for a few days.
-As adb98 mentioned, I also on a few occasions needed to reboot the Veeam server itself, fixing itself temporarily.

Never had any issues related to these Copy Jobs before Update 3. Everything with the Copy Jobs ran very smoothly. No changes to our Data Domain's. Sounds like we have very similar issues since updating to Update 3.

Add us to this list for post Update 3 issues with DD and the RPC issues... we also have an active ongoing case to try and figure this out.... #02464083

[bdaeumling]

Can you please post your DD-OS-Version? Im testing with DDOS6.1 and everything seems fine.

[bpayne]

DDOS 5.7.5.0 for all my Data Domains.

[datadrop]

Are the DD attached to a backup server's proxy or to standalone proxies? After update to 9.5 Update 3 we had problems with standalone proxies in remote locations sending backup data via ddboost to DDs. The jobs failed with the following error: " Failed to call RPC function 'DDBoostFcIsExist': in use. Err: 5012. Failed to initialize DDBoost Library."

Rebooting these proxies solved our issues. Must have been the DLL you mention above. Obviously we were lucky not having to reconfigure everything.

[tkeith]

I was given a hotfix on friday unfortunately it was 64bit and did not work. I was provided an updated 32bit yesterday which I will test today.

[bpayne]

Thanks tkeith for that information. I just called support earlier for an update to the case and haven't heard anything yet. I am waiting anxiously for your testing!

[adb98]

I am in the same boat. Waiting for a reply from support. My latest update from support was that this may be caused by older versions of SQL Express may not be able to handle a stored procedure and upgrading to SQL Express 2017 might fix it. Only issue with that is that we are not using SQL Express and are on full SQL 2014 sp1. I updated support and like Bpayne am waiting for an update.

The issue datadrop is referring to if I am looking at the error is DDBoost over Fibre Channel. We are using DD Boost over Ethernet. Seems like there are more than one or two bugs with DDBoost and this latest version.

[BGA-Robert]

Hmmm... I posted this in January
veeam-cloud-service-providers-forum-f34 ... 47855.html
Backup Copy Jobs Hang After Update 3 with Data Domain

Veeam support # 02455058

After a few reboots, things seemed to settle down and I closed the ticket without resolution.
But I feel like since there's no RCA or resolution, it's likely that I'll see this issue again.

Support could have been more helpful.

FYI, I did escalate to L2

-Robert

[tkeith]

Ok further testing of 32bit hotfix today has been positive. My Backup Copy jobs are running through and I haven't seen anything stick or error out. I'm able to enable/disable without any current issues. Our true test will be watching over tonights run when our local copies process and roll into the new copy interval on the copy jobs and if everything performs normally again. I still feel some logic must have changed with regards to VM processing as I feel its limiting the concurrent processes but i'm more looking forward to copy jobs that are working properly and not giving me freezing or RPC type issues that end up crahsing up the installer service before i venture into the little things.

Our environment is quite large and Backup Copy jobs have always been the headache so this was a real kick in the pants after we upgraded to Update 3. Let's hope we are good going forward.

I'm sure you can possibly reference our case number if you need to push support.

[adb98]

Glad everything is pushing in the right direction. Support did a webex with me today and installed a hotfix though they didn't explain to me what it was. We had a stuck job when they joined and it was still stuck after the hotfix so its back to the drawing board for my case. I will mention your case to my support rep.

[tkeith]

I did a post check remote session with our support. He mentioned there was cases with others where it wasn't clear if the fix worked for them or didn't seem to work. Good news is it seems they are working actively to try and address the issue. We are making some changes to address some performance issues but i'm happy to report I have not seen the installer service issue or the copy jobs having any problems yet. (fingers crossed)

[BGA-Robert]

Hmmm... I posted this in January
veeam-cloud-service-providers-forum-f34 ... 47855.html
Backup Copy Jobs Hang After Update 3 with Data Domain

Veeam support # 02455058

After a few reboots, things seemed to settle down and I closed the ticket without resolution.
But I feel like since there's no RCA or resolution, it's likely that I'll see this issue again.

Support could have been more helpful.

FYI, I did escalate to L2

-Robert

Yup, I jinxed myself... Checked today and one of the jobs was hung on a stopping status. I'm going to restart it and see if it runs.

[tkeith]

You jinxed me too... ok my update this morning is good and bad...
The hotfix for me seemed to have corrected the issue (so far) with the backup copy jobs causing the installer service to crash and kill all my jobs including locals...so i wake up in the morning to 200 failures... This morning happy to report i can browse everything and primary local jobs ran fine.
However...I noticed there are 4 copy jobs stuck in the "stopping" state this morning so i've concluded that the hotfix does not fix everything we are experiencing with regards to copy jobs.

[adb98]

That is not good. I was told last night from support that Dev can't reproduce the errors in their labs.

I am thinking they are using a virtual data domain and not the real thing. Maybe that could explain it. I am just about to the point that if I was told to upgrade the DDs to fix it I would put the change in to get it done.

[tkeith]

I was told another race condition was identified by development and I was provided an updated hotfix which im working on now... keep you posted.

[Gostev]

Right, it looks like devs have finally nailed the real root cause... we're very sorry about this mess race conditions are also the worst type of bugs, because they take so much luck to run into during the internal testing.

[adb98]

It looks like the latest patch has resolved our issue. I will know for sure if everything is good tomorrow morning. I greatly appreciate support jumping on this so quick and helping us with a quick resolution.

[bpayne]

I applied the newest private fix late yesterday and after applying it, my backup copy jobs previously stuck in a stopping state, automatically started up and worked. I also checked this morning and things are looking good. The real test is to see if it continuously works over the next 5 days so I will continue to monitor and will post back. Thanks to those involved in providing a quick fix.