Moving Backups To DataDomain Boost

[b.vanhaastrecht]

Running version 5.5.0.8-456696 here. Thruput is ok, but the processing of jobs is very slow. Every step in Veeam takes very long to process.

[NightBird]

With Backup copy job (with GFS set), each time a merge full backup run (for retention policy), the merge task stuck to 99% a while.

Normally the first created .vib merge in the .vbk to respect retention rules, this is fine, but at 99% Veeam create also a compact .vbk (compact should not run as GFS is set)

I see this after mounting the DDboost mtree with CIFS while the merge task stuck

[vStorage]

It seems that the only reliable way to get backups from Veeam 8 to DD boost enabled DataDomains at this point involves active fulls (or switching to CIFS for incremental runs)

Incremental and synthetic full are extremely inconsistent in performance, often stalling completely for hours at a time and taking 4-5 times longer than an active full.

The inconsistency is that sometimes incremental works fine performance wise with no change in job configuration, number of VMs or data change rate/amount of data transferred

[carlingtech]

R&D provided an updated .dll file that fixed the Veeam -> DataDomain issue for me, at least for incremental backups. I'm still having an issue with Synthetic fulls running for hours with no visible work being done. I have passed those logs into my case, so hopefully they'll find something else that needs attention.

You should open a case as I know they're working on testing the hotfix to resolve the incremental backup issue.

[vStorage]

Thanks - already onto a case, will see what happens

[vStorage]

Did you try the private fix from Veeam ? (39801) ?

Just got access to this patch

Does not work. Veeam Backup now fails to connect to DD boost repositories at all, just hangs the management GUI hard

[carlingtech]

I applied the 39801 hotfix earlier today. My DD repos still work, backups still function. Have not yet tested a synthetic full.

Remember this hotfix has to be applied to all proxies.

[vStorage]

It did get applied to all

I had to revert patch on Backup/mgmt server restart services and then do it again. Then it could see the repositories. Strange

[b.vanhaastrecht]

Are the slow processing problems solved after installing 39801 hotfix?

[b.vanhaastrecht]

Are the slow processing problems solved after installing 39801 hotfix?

To anwser my own question: We've applied the hotfix, but it did solve the problem in our situation. Ticket 00692321.



Edit: new image.

[CHRISATTWOOD]

Im still waiting for EMC to update me on this... its so frustrating.. as this was the update I was most looking forward to using!!

[NightBird]

I insist but someone at Veeam could explain why with ddboost the full backup merge create each time a compact file at the end of the merging process



You can see that at 19:37 the merging of the .vib in the .Vbk was finished and Veeam start to create a compact full at 99% from the merging process
At the end of the compact file creation Veeam replace the .vbk from 19:37 by the compact.vbk, strange behavior !


Veeam bug ? by ddboost design ?

Thx for prompt reply

[Gostev]

@Nightbird by design. This is how DDBoost-accelerated transforms are performed (new VBK is synthesized by referencing existing data already on the storage).

To anwser my own question: We've applied the hotfix, but it did solve the problem in our situation. Ticket 00692321.

This issue is documented in the Known Issues sticky forum post. It is not specific to Data Domain storage and requires 38623 hot fix. Thanks!

[NightBird]

@Nightbird by design. This is how DDBoost-accelerated transforms are performed (new VBK is synthesized by referencing existing data already on the storage).

Why do a transform as the main .vbk file was already touch (for retention policy ?) as the can see with the file timestamp ?

For transform full weekly, monthly and so on I can understand, but here ??? create a compact.vbk to replace the main .vbk that was touch some minutes ago... time consuming and quite silly, don't you think so ?

This imply that a simple merge full for rentention policy (first .vib merge in the .vbk) is slower to make with ddboost that with cifs... missing something !

[Gostev]

I don't understand what you are saying here, I'm sorry. Also, if whatever you are talking about is unrelated to the main topic of this discussion, then let's not derail it and instead create the new one.

[myrdin]

Hi,

there is a ticket, but so far we didnt figure out the issue (00698997)

I have a bunch of dd2200 with boost and vsphere 5.1 running with V8.

I wasnt particoularly impressed with backup performances, especially for syntethics (that actually takes more than active fulls...).

Having the ESX host connected on a 4Gb aggregated link directly to a autotiering storage with mixed SAS/SSD all connected to an aggregated 4Gb/s link to DD, and getting if i am lucky 80MB/s well...not very promising. Processing speed for full are like 20MB/s with no data pulled from the prod NAS, so purely in the DD.

I am doing NDB but hotadd give me the same results.

In restore is even worst, it takes MORE THAN 30 MINUTEs (and counting, it never finished so i actually dont know) for a 300GB backup file. The funny thing is that looking at the DD while Veeam is trying to mount the image, it looks like the DD is doing nothing or very very little, so it looks like Veeam is not pulling the info from DD the right way? The restore tests made with no other jobs running.

Load on proxies (3x proxis total 10CPU available) very low.

I had a guy from EMC looking into the autosupport logs for the DD, and he said Veem is pushing/pulling only one stream to the DD and the DD is heavily underutilized.

Ideas?

[KevinK]

Be aware that DDBoost is not supported over WAN by EMC.
But, let's not derail this topic with an unrelated discussion.

From the 5.5.0.4 release notes

WAN support—backup to Data Domain over WAN, enabling Avamar DT/LT (DeskTop/
LapTop) and ROBO (Remote Office Back Office) use cases to Data Domain and
DDBoost

[KevinK]

Here is a DD fastcopy command that allows you to update the destination rather than overwrite.

filesys fastcopy update source [source] destination [destination]

[chrisdearden]

Thanks for the update Kevin - have you had a chance to test it ?

[KevinK]

I've just received a DDBoost key for testing - will let you know.

[Gostev]

From the 5.5.0.4 release notes

You are quoting my statement about DDBoost, and then DD OS release notes. But DDBoost and DD OS are two absolutely different things

I can confirm that DDBoost 2.6.x used in B&R v8 is not supported over WAN by EMC regardless of DD OS version. I had a discussion about this specifically directly with EMC R&D a few months ago. This is a hard limitation.

On top of DD OS 5.5.0.4, WAN support also requires newer DDBoost 3.x (which was not ready in time for our v8 release).

[KevinK]

My initial testing with DDBoost - A single SQL VM on EMC 15k SAS.

The backup server and Data Domain are both on 10Gbps.



Traffic to the DD without boost averaged around 130MB/s - Traffic with boost average around 10MB/s - Incredible

Is it correct that it is highlighting network as the bottleneck in both instances? Perhaps as this component is working the hardest and not necessarily bottlenecking performance?

As we already have a high through-put network I'm not seeing big benefits yet, however once I run our regular backup jobs which are based on datastores I believe the CPU load will increase. I hope this doesn't affect the amount of jobs/vDisks the server can backup at once.

Two potential perks are being able to still backup quickly even if the DD is running a clean job and less bloating on the data DD after each backup.

All credit to Veeam on this integration - if it was available two years ago we could have saved on the networking hardware

[haslund]

To anwser my own question: We've applied the hotfix, but it did solve the problem in our situation. Ticket 00692321.



Edit: new image.

Is this image before or after applying the hotfix?
I am seeing exactly the same behavior.

[zuerom]

Hello,

We added a new DataDomain to our environment (Veeam8 u1).
What is the best / fastest / easiest way to move the Backups from BackupCopyjobs (Windows Repository) to the new DataDomain with DDBoost (attached as an appliance)?

regards
Roman

[friko]

It's been a few releases since this thread was opened. Do you guys know if there's now a way to do this?

[foggy]

Jonathan, could you please elaborate on what specifically are you trying to do?

[friko]

Hi,

I want to move my existing backups (on a CIFS) to a DD Boost Storage Unit on a new Data Domain.

[foggy]

a new Data Domain.

Is this a separate storage device from the one you're currently have backups on?

[rnt-guy]

I too am trying to move my data from a usb hard drive (veeam 9 backed up to a usb device, now connected to a server \\server1\usb\ that can see the DD. I've already been backing up to the DD boost LSU for our other jobs but we want to import a large backup (1.5TB) from the usb drive (via a \\ share).

This thread went a couple different directions. Any advice for a Windows guy?

[foggy]

The easiest way would be to export the DDBoost repository via CIFS and copy data between shares then. Another approach is using the Files tab in Veeam B&R.