Comprehensive data protection for all workloads
b.vanhaastrecht
Service Provider
Posts: 833
Liked: 154 times
Joined: Aug 26, 2013 7:46 am
Full Name: Bastiaan van Haastrecht
Location: The Netherlands
Contact:

Re: Moving Backups To DataDomain Boost

Post by b.vanhaastrecht »

Running version 5.5.0.8-456696 here. Thruput is ok, but the processing of jobs is very slow. Every step in Veeam takes very long to process.
======================================================
Veeam ProPartner, Service Provider and a proud Veeam Legend
NightBird
Expert
Posts: 242
Liked: 57 times
Joined: Apr 28, 2009 8:33 am
Location: Strasbourg, FRANCE
Contact:

Re: Moving Backups To DataDomain Boost

Post by NightBird »

With Backup copy job (with GFS set), each time a merge full backup run (for retention policy), the merge task stuck to 99% a while.

Normally the first created .vib merge in the .vbk to respect retention rules, this is fine, but at 99% Veeam create also a compact .vbk (compact should not run as GFS is set)

I see this after mounting the DDboost mtree with CIFS while the merge task stuck

Image
vStorage
Influencer
Posts: 11
Liked: 3 times
Joined: Nov 06, 2014 9:32 pm
Contact:

Re: Moving Backups To DataDomain Boost

Post by vStorage »

It seems that the only reliable way to get backups from Veeam 8 to DD boost enabled DataDomains at this point involves active fulls (or switching to CIFS for incremental runs)

Incremental and synthetic full are extremely inconsistent in performance, often stalling completely for hours at a time and taking 4-5 times longer than an active full.

The inconsistency is that sometimes incremental works fine performance wise with no change in job configuration, number of VMs or data change rate/amount of data transferred
Barrie
https://twitter.com/vStorage
carlingtech
Novice
Posts: 6
Liked: never
Joined: Apr 16, 2014 10:25 pm
Contact:

Re: Moving Backups To DataDomain Boost

Post by carlingtech »

R&D provided an updated .dll file that fixed the Veeam -> DataDomain issue for me, at least for incremental backups. I'm still having an issue with Synthetic fulls running for hours with no visible work being done. I have passed those logs into my case, so hopefully they'll find something else that needs attention.

You should open a case as I know they're working on testing the hotfix to resolve the incremental backup issue.
vStorage
Influencer
Posts: 11
Liked: 3 times
Joined: Nov 06, 2014 9:32 pm
Contact:

Re: Moving Backups To DataDomain Boost

Post by vStorage »

Thanks - already onto a case, will see what happens
Barrie
https://twitter.com/vStorage
vStorage
Influencer
Posts: 11
Liked: 3 times
Joined: Nov 06, 2014 9:32 pm
Contact:

Re: Moving Backups To DataDomain Boost

Post by vStorage »

NightBird wrote:Did you try the private fix from Veeam ? (39801) ?
Just got access to this patch

Does not work. Veeam Backup now fails to connect to DD boost repositories at all, just hangs the management GUI hard
Barrie
https://twitter.com/vStorage
carlingtech
Novice
Posts: 6
Liked: never
Joined: Apr 16, 2014 10:25 pm
Contact:

Re: Moving Backups To DataDomain Boost

Post by carlingtech »

I applied the 39801 hotfix earlier today. My DD repos still work, backups still function. Have not yet tested a synthetic full.

Remember this hotfix has to be applied to all proxies.
vStorage
Influencer
Posts: 11
Liked: 3 times
Joined: Nov 06, 2014 9:32 pm
Contact:

Re: Moving Backups To DataDomain Boost

Post by vStorage »

It did get applied to all

I had to revert patch on Backup/mgmt server restart services and then do it again. Then it could see the repositories. Strange
Barrie
https://twitter.com/vStorage
b.vanhaastrecht
Service Provider
Posts: 833
Liked: 154 times
Joined: Aug 26, 2013 7:46 am
Full Name: Bastiaan van Haastrecht
Location: The Netherlands
Contact:

Re: Moving Backups To DataDomain Boost

Post by b.vanhaastrecht »

Are the slow processing problems solved after installing 39801 hotfix?
======================================================
Veeam ProPartner, Service Provider and a proud Veeam Legend
b.vanhaastrecht
Service Provider
Posts: 833
Liked: 154 times
Joined: Aug 26, 2013 7:46 am
Full Name: Bastiaan van Haastrecht
Location: The Netherlands
Contact:

Re: Moving Backups To DataDomain Boost

Post by b.vanhaastrecht »

b.vanhaastrecht wrote:Are the slow processing problems solved after installing 39801 hotfix?
To anwser my own question: We've applied the hotfix, but it did solve the problem in our situation. Ticket 00692321.

Image

Edit: new image.
======================================================
Veeam ProPartner, Service Provider and a proud Veeam Legend
CHRISATTWOOD
Novice
Posts: 4
Liked: never
Joined: Feb 11, 2014 10:52 am
Full Name: Chris Attwood
Contact:

Re: Moving Backups To DataDomain Boost

Post by CHRISATTWOOD »

Im still waiting for EMC to update me on this... its so frustrating.. as this was the update I was most looking forward to using!!

:(
NightBird
Expert
Posts: 242
Liked: 57 times
Joined: Apr 28, 2009 8:33 am
Location: Strasbourg, FRANCE
Contact:

Re: Moving Backups To DataDomain Boost

Post by NightBird »

I insist but someone at Veeam could explain why with ddboost the full backup merge create each time a compact file at the end of the merging process

Image

You can see that at 19:37 the merging of the .vib in the .Vbk was finished and Veeam start to create a compact full at 99% from the merging process
At the end of the compact file creation Veeam replace the .vbk from 19:37 by the compact.vbk, strange behavior !


Veeam bug ? by ddboost design ?

Thx for prompt reply ;)
Gostev
Chief Product Officer
Posts: 31457
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Moving Backups To DataDomain Boost

Post by Gostev »

@Nightbird by design. This is how DDBoost-accelerated transforms are performed (new VBK is synthesized by referencing existing data already on the storage).
b.vanhaastrecht wrote: To anwser my own question: We've applied the hotfix, but it did solve the problem in our situation. Ticket 00692321.
Image
This issue is documented in the Known Issues sticky forum post. It is not specific to Data Domain storage and requires 38623 hot fix. Thanks!
NightBird
Expert
Posts: 242
Liked: 57 times
Joined: Apr 28, 2009 8:33 am
Location: Strasbourg, FRANCE
Contact:

Re: Moving Backups To DataDomain Boost

Post by NightBird »

Gostev wrote:@Nightbird by design. This is how DDBoost-accelerated transforms are performed (new VBK is synthesized by referencing existing data already on the storage).
Why do a transform as the main .vbk file was already touch (for retention policy ?) as the can see with the file timestamp ?

For transform full weekly, monthly and so on I can understand, but here ??? create a compact.vbk to replace the main .vbk that was touch some minutes ago... time consuming and quite silly, don't you think so ?

This imply that a simple merge full for rentention policy (first .vib merge in the .vbk) is slower to make with ddboost that with cifs... missing something !
Gostev
Chief Product Officer
Posts: 31457
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Moving Backups To DataDomain Boost

Post by Gostev »

I don't understand what you are saying here, I'm sorry. Also, if whatever you are talking about is unrelated to the main topic of this discussion, then let's not derail it and instead create the new one.
myrdin
Enthusiast
Posts: 57
Liked: 3 times
Joined: Jul 02, 2013 4:17 am
Full Name: NIck
Contact:

[MERGED] Veeam 8 and DataDomain : Performance (so far NOT go

Post by myrdin »

Hi,

there is a ticket, but so far we didnt figure out the issue (00698997)

I have a bunch of dd2200 with boost and vsphere 5.1 running with V8.

I wasnt particoularly impressed with backup performances, especially for syntethics (that actually takes more than active fulls...).

Having the ESX host connected on a 4Gb aggregated link directly to a autotiering storage with mixed SAS/SSD all connected to an aggregated 4Gb/s link to DD, and getting if i am lucky 80MB/s well...not very promising. Processing speed for full are like 20MB/s with no data pulled from the prod NAS, so purely in the DD.

I am doing NDB but hotadd give me the same results.

In restore is even worst, it takes MORE THAN 30 MINUTEs (and counting, it never finished so i actually dont know) for a 300GB backup file. The funny thing is that looking at the DD while Veeam is trying to mount the image, it looks like the DD is doing nothing or very very little, so it looks like Veeam is not pulling the info from DD the right way? The restore tests made with no other jobs running.

Load on proxies (3x proxis total 10CPU available) very low.

I had a guy from EMC looking into the autosupport logs for the DD, and he said Veem is pushing/pulling only one stream to the DD and the DD is heavily underutilized.

Ideas?
KevinK
Enthusiast
Posts: 28
Liked: 10 times
Joined: Apr 24, 2013 9:18 am
Full Name: Kevin Kissack
Contact:

Re: Moving Backups To DataDomain Boost

Post by KevinK »

Gostev wrote:Be aware that DDBoost is not supported over WAN by EMC.
But, let's not derail this topic with an unrelated discussion.
From the 5.5.0.4 release notes
WAN support—backup to Data Domain over WAN, enabling Avamar DT/LT (DeskTop/
LapTop) and ROBO (Remote Office Back Office) use cases to Data Domain and
DDBoost
KevinK
Enthusiast
Posts: 28
Liked: 10 times
Joined: Apr 24, 2013 9:18 am
Full Name: Kevin Kissack
Contact:

Re: Moving Backups To DataDomain Boost

Post by KevinK »

Here is a DD fastcopy command that allows you to update the destination rather than overwrite.
filesys fastcopy update source [source] destination [destination]
chrisdearden
Veteran
Posts: 1531
Liked: 226 times
Joined: Jul 21, 2010 9:47 am
Full Name: Chris Dearden
Contact:

Re: Moving Backups To DataDomain Boost

Post by chrisdearden »

Thanks for the update Kevin - have you had a chance to test it ?
KevinK
Enthusiast
Posts: 28
Liked: 10 times
Joined: Apr 24, 2013 9:18 am
Full Name: Kevin Kissack
Contact:

Re: Moving Backups To DataDomain Boost

Post by KevinK »

I've just received a DDBoost key for testing - will let you know.
Gostev
Chief Product Officer
Posts: 31457
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Moving Backups To DataDomain Boost

Post by Gostev »

KevinK wrote: From the 5.5.0.4 release notes
You are quoting my statement about DDBoost, and then DD OS release notes. But DDBoost and DD OS are two absolutely different things :D

I can confirm that DDBoost 2.6.x used in B&R v8 is not supported over WAN by EMC regardless of DD OS version. I had a discussion about this specifically directly with EMC R&D a few months ago. This is a hard limitation.

On top of DD OS 5.5.0.4, WAN support also requires newer DDBoost 3.x (which was not ready in time for our v8 release).
KevinK
Enthusiast
Posts: 28
Liked: 10 times
Joined: Apr 24, 2013 9:18 am
Full Name: Kevin Kissack
Contact:

Re: Moving Backups To DataDomain Boost

Post by KevinK » 1 person likes this post

My initial testing with DDBoost - A single SQL VM on EMC 15k SAS.

The backup server and Data Domain are both on 10Gbps.

ImageImage

Traffic to the DD without boost averaged around 130MB/s - Traffic with boost average around 10MB/s - Incredible

Is it correct that it is highlighting network as the bottleneck in both instances? Perhaps as this component is working the hardest and not necessarily bottlenecking performance?

As we already have a high through-put network I'm not seeing big benefits yet, however once I run our regular backup jobs which are based on datastores I believe the CPU load will increase. I hope this doesn't affect the amount of jobs/vDisks the server can backup at once.

Two potential perks are being able to still backup quickly even if the DD is running a clean job and less bloating on the data DD after each backup.

All credit to Veeam on this integration - if it was available two years ago we could have saved on the networking hardware :)
haslund
VeeaMVP
Posts: 839
Liked: 149 times
Joined: Feb 16, 2012 7:35 am
Full Name: Rasmus Haslund
Location: Denmark
Contact:

Re: Moving Backups To DataDomain Boost

Post by haslund »

b.vanhaastrecht wrote: To anwser my own question: We've applied the hotfix, but it did solve the problem in our situation. Ticket 00692321.

Image

Edit: new image.
Is this image before or after applying the hotfix?
I am seeing exactly the same behavior.
Rasmus Haslund | Twitter: @haslund | Blog: https://rasmushaslund.com
zuerom
Novice
Posts: 8
Liked: 2 times
Joined: Feb 28, 2014 11:14 am
Full Name: Roman Zürcher
Contact:

[MERGED] Move Longterm Backup (WinRepo) to DataDomain

Post by zuerom »

Hello,

We added a new DataDomain to our environment (Veeam8 u1).
What is the best / fastest / easiest way to move the Backups from BackupCopyjobs (Windows Repository) to the new DataDomain with DDBoost (attached as an appliance)?

regards
Roman
friko
Novice
Posts: 3
Liked: never
Joined: Sep 27, 2016 3:01 pm
Full Name: Jonathan P
Contact:

Re: Moving Backups To DataDomain Boost

Post by friko »

It's been a few releases since this thread was opened. Do you guys know if there's now a way to do this?
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Moving Backups To DataDomain Boost

Post by foggy »

Jonathan, could you please elaborate on what specifically are you trying to do?
friko
Novice
Posts: 3
Liked: never
Joined: Sep 27, 2016 3:01 pm
Full Name: Jonathan P
Contact:

Re: Moving Backups To DataDomain Boost

Post by friko »

Hi,

I want to move my existing backups (on a CIFS) to a DD Boost Storage Unit on a new Data Domain.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Moving Backups To DataDomain Boost

Post by foggy »

friko wrote:a new Data Domain.
Is this a separate storage device from the one you're currently have backups on?
rnt-guy
Enthusiast
Posts: 61
Liked: 1 time
Joined: Feb 04, 2016 12:58 pm
Full Name: RNT-Guy
Contact:

Re: Moving Backups To DataDomain Boost

Post by rnt-guy »

I too am trying to move my data from a usb hard drive (veeam 9 backed up to a usb device, now connected to a server \\server1\usb\ that can see the DD. I've already been backing up to the DD boost LSU for our other jobs but we want to import a large backup (1.5TB) from the usb drive (via a \\ share).

This thread went a couple different directions. Any advice for a Windows guy? ;)
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Moving Backups To DataDomain Boost

Post by foggy »

The easiest way would be to export the DDBoost repository via CIFS and copy data between shares then. Another approach is using the Files tab in Veeam B&R.
Post Reply

Who is online

Users browsing this forum: dnaxy, Google [Bot] and 165 guests