Moving Backups To DataDomain Boost

[meilicke]

Hello,

What is the best way to move existing backups to a newly created DataDomain Boost enabled repository?

Thanks.

[meilicke]

I opened a sev. 4 case and will post back when I hear more - # 00673705.

[Gostev]

Hi, Scott. I am not hands on with DataDomain, but can you not enable access to the DDBoost storage unit via SMB temporarily, and move your backups there using regular file copy? Thanks!

[jj281]

I haven't done this from mtree to DDBoost enabled Mtree, but you should be able to do a Data Domain fastcopy (from DD, filesys fastcopy) to quickly copy your data over to the DDBoost SU/Mtree.

SMB copy could be a PITA especially if you're talking about a lot of data.

PS Feature request, Backup Job Repository Move leveraging DD Boost... In case you don't want to switch an entire repository, and all its jobs, over to a DDBoost SU/Mtree.

[Gostev]

Hi, Scott. I am not hands on with DataDomain, but can you not enable access to the DDBoost storage unit via SMB temporarily, and move your backups there using regular file copy? Thanks!

So I've ran this scenario by some EMC DD folks on our periodic call, and they have noted one potential gotcha. Be careful not to accidentally set CIFS ACL on the copied files, or DDBoost will have issues with file access. Other than that, this should be doable!

[EdwardPoll]

Gostev,

Interesting point about the permissions - this is something we are experiencing, however we aren't setting the permissions ourselves!

We currently target the backups to CIFS shares on our DD 4200(s), we've enable DDBoost etc and all is working fine for *new* jobs. However when we use FASTCOPY to get the 'old' backups from the CIFS share into the DDBoost MTree they come across with the same permissions they had previously i.e. those of the CIFS share/directory. To that end when the DDBoost user tries to access them to 'continue' the backup chain it fails with 'no permission' errors when trying to open/read the files. We've tried to flush permissions down from the root level of the DDBoost MTree but this effectively wipes the CIFS permissions and does not correctly apply the necessary permissions for DDBoost.

As a work around, we can remove the old backups (from the job, not from the disk!) and it will start the job from fresh as a new backup chain (obviously if we need to go back to the old backups we just reimport from the disk), but this as you can imagine is a real PITA.

Any ideas on how to move them and have the permissions correct? Or are we simply missing something blindly obvious!

Thanks

Ed

[veremin]

Isn't that possible to make FastCopy copy backups without preserving original permissions? Or it won't work in this case? Thanks.

[EdwardPoll]

Not that we can see....

filesys fastcopy source <src> destination <dest>

Not much configuration in that....

Any other ideas anyone?

[veremin]

What about using different copy utility, such as RoboCopy or similar? If I'm not mistaken, the said tool allows you to copy files without preserving permissions. Thanks.

[jj281]

What about using different copy utility, such as RoboCopy or similar? If I'm not mistaken, the said tool allows you to copy files without preserving permissions. Thanks.

When using fastcopy, perhaps NFS mount the (DDBoost) mtree then "push" permissions back down from the root.

I suppose you can do that via CIFS, I just haven't had much luck in those cases.

[tomb80]

I have had no success with copying old backups into the new DDBoost storage device. I created all new jobs and deleted the old jobs. After that, I just imported the backup files from the old jobs until I manually delete them for the retention period. That has seemed to work best with me.

Has anyone seen any problems with V8 and DDBoost? My backups are painfully slow when they should be blazing fast. We are backing up from a Pure Flash Array and a CX480. I'm wondering if the Pure dedupe and compression could be causing issues with V8.

[kgillett14]

Have you had your DD Boost device removed from the list of devices in the DD OS after doing the fastcopy? I did and EMC support had to make a file named ".ddboost" in the directory to make it appear again.

[jj281]

I have had no success with copying old backups into the new DDBoost storage device. I created all new jobs and deleted the old jobs. After that, I just imported the backup files from the old jobs until I manually delete them for the retention period. That has seemed to work best with me.

Has anyone seen any problems with V8 and DDBoost? My backups are painfully slow when they should be blazing fast. We are backing up from a Pure Flash Array and a CX480. I'm wondering if the Pure dedupe and compression could be causing issues with V8.

Have you confirmed DDBoost is "working". ddboost show stats interval # does it seem to match up?

I'm not sure why the source storage dedupe/compression would matter unless it's slower...

Is your Proxies (or Gateway) capable of handling the DDBoost load?

I did run the beta and the synthetics were noticeably faster (4x). I'm running the release right now but haven't done a synthetic on a Backup Job... However I am doing a backup copy job right now. What's.... odd is the Backup Copy seems to be reading faster from the source DataDomain with Boost vs without Boost (like double), I thought generally there weren't improvements on read? Chalk it up to protocol efficiency over CIFS?

Active Fulls/Incrementals are about the same in the release vs non-DDBoost... I would hope for some improvement there by just the fact the DataDomain doesn't have to ingest as much data, but I have to test from a flash array or additional proxies before making any conclusions...

[carlingtech]

I have had no success with copying old backups into the new DDBoost storage device. I created all new jobs and deleted the old jobs. After that, I just imported the backup files from the old jobs until I manually delete them for the retention period. That has seemed to work best with me.

Has anyone seen any problems with V8 and DDBoost? My backups are painfully slow when they should be blazing fast. We are backing up from a Pure Flash Array and a CX480. I'm wondering if the Pure dedupe and compression could be causing issues with V8.

I have also had performance issues with v8 and Boost. Jobs were taking quite a bit longer (3 hours for what was a 30 minute incremental in v7) with long periods where it didn't appear Veeam was doing anything. Jobs were run for 30 minutes or more with no data backed up. I have reverted back to a CIFS share and things are working as well as they did with v7. I have a case open with support (00673245), but have not gotten any solutions on it yet.

[Gostev]

I thought generally there weren't improvements on read? Chalk it up to protocol efficiency over CIFS?

Correct. DDBoost is used for both writes and reads (so, it should help with restores too). And, it is definitely more efficient than CIFS. Thanks!

[Gostev]

I have a case open with support (00673245), but have not gotten any solutions on it yet.

Source bottleneck indicates that your issue has nothing to deal with Data Domain.

[carlingtech]

Yeah, the tech support engineer wasn't listening (or didn't care to listen)- that source bottleneck only happens with the DDBoost jobs. When running the jobs with a CIFS share to the Datadomain device, the bottleneck is the target and the jobs run as expected.

I'll be escalating the ticket on Monday. I'm not pleased with the response to date.

[vStorage]

Make sure you are using the pre release 5.5 OS firmware for the Data Domains.

I had exactly the same issue with Veeam 8 and incremental backups sitting there for hours with proxy servers having hot-add VM snapshots open and 0% progress

The issue appears to be 5.4.x firmware on the Data Domains. 5.5.0.7 solved it completely for me

Open a case with EMC and get access to the pre release DD OS package

[carlingtech]

Thanks! I was wondering if it had something to do with the release. Currently running 5.4.4.2-453843. I will contact EMC.

[Gostev]

We are running 5.4.0.8-404909 and 5.5.0.2-421637 on our two QC boxes.

[vStorage]

Just for clarification, the DD OS was 5.4.2.1 (which is the current shipping GA firmware I believe)

EMC provided 5.5.0.7 (which is the only version they are giving to customers at this stage it seems)

[Bazillus]

The latest GA I could download from EMC is: 5.4.4.2
All 5.5 are DA (like a Beta Version) at the Moment.
Downgrade to 5.4.0.8 or upgrade to a DA is in our Environment not an Option.
So we have to wait for a offical GA, correct?

[ale]

I have the same issue with v8 and EMC DDBoost. Incremental Backups are very slow on DD2500 with DDOS 5.5.0.4. Update to 5.5.0.7 didn't resolve the issue. Please help!
Veeam Case # 00680905

[Gostev]

Alexander, please continue to work with our support, as we are unable to troubleshoot issue through the forum posts. Thanks!

[CHRISATTWOOD]

Hi everyone

Apologies if this has been covered already, I have trawled through the last 4 or 5 pages and I cant see a topic on it.

I have recently upgraded to Veeam V8 on the premise of using DDboost on our DD2500. I have set everything up on the DD and I have ran several jobs to the new storage unit I created.

The jobs complete, but the throughput im getting is around 4MBS.. My backups to my NFS targets are getting speeds of 40MBS minimum up to 150MBS... the channel it traverse has 10GB throughput

I have upgraded my DD2500 to 5.4.2.1 but the throughput is the same, currently logged a call with emc to get 5.5 on DA, just waiting for it at the moment..

But has anyone else experienced this or could share any thoughts?

Thanks

Chris

[meilicke]

Following up, I worked with both Veeam and EMC tech support, and we were not successful getting the old backups from the CIFS only mtree to the Boost mtree. It came down to permissions that were copied over with the fastcopy. We dropped down to bash within the DD, and tried dd_xcacls and good old chmod/chown, to no avail. So now I am removing backup files from Veeam (not disk), re-importing, and then modifying the job to point to the new Boost repository. At least all of the traffic for the new full backup is staying within the proxy servers. From the proxy that is servicing the DD, traffic is minimal, ~300MB/s between proxies and 4-5MB/s between the Boost proxy and the DD.

Now I need to find out which is more efficient, using the Veeam WAN Accelerator for backup copies, or creating a boost repository on the remote DD, but assigning a local boost proxy to it. In theory the boost proxy should do a similar job as the Veeam WAN Accel, maybe better?

[Gostev]

Thanks for the update re: moving backups.

Be aware that DDBoost is not supported over WAN by EMC.
But, let's not derail this topic with an unrelated discussion.

[tomb80]

I have brand new HP BL460c blades with more than enough resources to handle this. I have tried it on older Hp servers as well. Any backup that I do with DDBoost is painfully slow. Right now I am running test jobs to the following:

DDBoost
DDBoost over FC
CIFS
NFS through Linux

Also different transports

SAN
Hot Add
Network

DDBoost is getting killed and it is working. It is just slow. We just upgraded our DD to 5.5.0.8 hoping that would help some issues. The job I have running right now has been running for an hour and a half and it usually finishes in 45 minutes or less without DDBoost. I am running on 8GB fiber from a Pure storage array through Dell R720s and HP blades running ESXi 5.5. Veeam 7 kills Veeam 8 on performance so far in CIFS/NFS with no comparison for DDBoost.

[carlingtech]

Working with support currently on same issue with DataDomain backups slow or never completing. They are tracking a number of issues backing up to Data Domain devices and are now working with EMC to track down the issue.

What model DataDomain do you have? I have the same problem with a DD620 running the same codeset.

[NightBird]

Did you try the private fix from Veeam ? (39801) ?