Veeam, DataDomain and Linux NFS share

[hartza]

Hi,

I try to use DataDomain (DD610) NFS share to see if it gets better performance with Veeam Backup software than CIFS. I have added linux server to Veeam and mounted NFS share to that linux server. I am able to create backup job using the NFS share with Veeam and actually job writes .vbk file to DD NFS but then backup job just hangs. Size of .vbk is 0. I need to reboot Veeam backup server to stop that backup job.

Anyone can help me?

[Gostev]

Hi,

I believe this happens because DataDomain doesn't support flock file locking with their NFS implementation (which is very uncommon), so we hit an impasse each time we write for backup. However, it's an easy fix to disable flock on the mount with nolock parameter.

Command to mount NFS share on your Linux server should be something like:

Code: Select all

mount -t nfs -o nolock,hard,intr,nfsvers=3,tcp,bg 172.16.4.67:/data/col1 /dd/rstr01/backup

Hope this helps.

[hartza]

Solved the issue. Can now start testing out performance of DD NFS share. Thank you very much

[ckbrou]

I just started using Data Domain with Veeam and am using CIFS to start out. It works fine and makes for a simple solution but am curious if you found NFS to perform much better? The only complaint I have with CIFS is the performance of UAIR but that may be more of limitation of deduplication technology more than the storage protocol.

[tsightler]

Data Domain has a whitepaper in which they discuss integration with Veeam. They're document (the last I saw it anyway), provides setup for both CIFS and NFS, but recommends the Linux NFS setup for performance. Veeam support has had many cases of resolving performance issues with DD by getting customers to use Linux NFS instead of CIFS. The whitepaper has very specific mount options to provide the best performance.

Also, the Linux NFS setup provides some benefits for UAIR setup, specifically the Linux NFS client cache seems more efficient than when using CIFS, thus reducing the number of request that most go to the storage system by servicing these requests from the cache. I've even helped some users configure the Linux FS-Cache for even better NFS client performance for U-AIR and Surebackup.

http://www.linuxtopia.org/online_books/ ... emain.html

[ckbrou]

I was able to get the Linux NFS setup working and also confirgured the Linux FX-Cache. U-Air works sort of. In about 2 hours it brings up my DC and my Exchange Server but many of my Exchange services will not start. The deduped storage is just too slow and the services keep timing out. When I open up the Exchange server console from the SureBackup Session it is painfully slow.

It's too bad that there isnt a way to just restore the application group to production storage but still keep it in its own segregated network. You could still use the virtual lab / UAIR functionality but it would actually be fast. Unless something like this is possible and I just don't see how to do it?

Now if I want to restore Exchange data I may need to first copy the backup file to faster disk and then run the SureBackup job from it there... but that will of course be a big pain.

[tsightler]

Wow, 2 hours to bring up a DC is worse than any I have ever seen even with DataDomain. Have you tested your Surebackup performance when writing the backups to a different storage device? I'm wondering if there is something else going on in your case that's making the performance even worse? I've certainly seen fairly large DC environments that take 40-45 minutes or so to boot from DataDomain.

[ckbrou]

The whole Job takes about two hours. That includes the DC booting (about 30 minutes), DC application tests (about another 15), and then the Exchange server booting (about 45 minutes), Exchange application tests (about another 15). The problem though is that even though the Exchange server is up and running the exchange services just will not start. They time out it seems because things are just so slow...

[cghall11]

The DD610 is a very low spec DD, Im going to do some testing on a larger one and I will update you with the results. Im EMC's premier partner BRS senior technical architect and in the middle of finding out what configurations work best for Veeam to DD backups. Update you soon!!

[ckbrou]

Thanks for looking into this! I will be curious as to what you find related to NFS vs CIFS performance. Originally I thought that NFS provided better performance but after using it for a couple weeks now I am not so sure (it seems like a negligible difference). It obviously makes the setup more complex and adds an extra hop to the network path with the NFS Linux gateway.

[novakk_AH]

[merged]

Hello,

I have a problem with surebackup job if the backup VM machine is placed on datadomain DD620.

When is size VM at about 40GB surebackup is success but bigger VM has problem with start in time.
Results: Cannot detect VM starting because of timeout

I increased the timeout value for the start but did not help. When I let the same job from another repository passes OK.

I see through the console of the machine running, but slowly and can not start up correctly service windows. ( vmware tools , MS SQL 2008 etc... )

Does anyone have experience?


Configuration network datadomain eth0a and eth0b is veeam backup

port enabled DHCP IP address netmask type additional setting
----- ------- ---- ------------- ------------- ---- -----------------------
eth0a yes n/a n/a n/a n/a bonded to veth0
eth0b yes n/a n/a n/a n/a bonded to veth0
eth3a yes no 192.168.62.21 255.255.255.0 n/a
eth3b no n/a n/a n/a n/a
veth0 yes no 192.168.51.20 255.255.255.0 n/a roundrobin: eth0a,eth0b
----- ------- ---- ------------- ------------- ---- -----------------------

[Arildm]

[merged]

Hey guys.
First time poster, and a couple of months old reader. Thx for the help i have gotten so far from this forum.

we have now started to try out DataDomain and bought a DD620 (firmware 5.0). I have read on this forum that a lot of people have managed to connect this to their veeam software, but my problem already starts here. As far as i have understood it, DataDomain is a downstripped version of linux with some extra things from EMC. But since i cant do a vi on it / check log-files or anything similar, i really dont understand how you guys have managed to connect Veeam to a datadomain.

20.03.2012 19:17:50 Fail Failed to save Backup Repository: Error: , answer: That doesn't look like a valid command, displaying help...

Help is available on the following topics:

adminaccess enclosure ntp alerts filesys replication
alias ipmi route archive license snapshot
authentication log snmp autosupport migration storage
cifs mtree support cluster ndmpd system
config net user ddboost nfs vtl
disk

Type "help <topic>" to view help for the given topic.
Type "help <keyword>" to search the commands for a specific keyword.

I manage to get this to talk to my qnap, and take backup to this one for the moment but dont have a "out of house" Disaster backup.
Hope somebody have a idea that really rocks my socks and show a IT-junior how things should be done .
i use Veeam 6.0.0 and takes backup of a Hyper-V enviroment.



-----------------
From the cold North
Arildm

------------------

[tomas.olsen]

[merged]

Hello,

I am experiencing rather bad performance when testing lets say my domain controller with veeam surebackup.
When backing up to a de-dupe storage I usually backup up with ordinary incremental backup and weekly full backups.
But a surebackup job takes almost up to an hour to complete, if it even does complete.
the domain controller is part of a larger backup job, containing from 15 to 40 vm's at different customers.
If I give the virtual backup server a second disk, does a backup of just the domain controller to a local disk, I can perform a surebackup of that domain controller in just 15 minutes.
I know that datadomain is slower on read, especially if you have both write and read operations done at the same time. So I have made sure that the surebackup runs when no other job runs.
If I try to surebackup exchange or SQL servers, they boot to slow so that about all the services is left unstarted, and the job fails. Even if I increase maximum allowed boot time.
If I open a consol to the surebackup vm, and hits CTRL+ALT+INSERT, the vm hardly responds, and each task takes almost minutes to complete.

Is there a general understanding that surebackup just doesn't work with datadomain, or could there be some bad configuration from my side?
Any thoughts or ideas??

Best regards, Tomas

[cghall11]

Hi all sorry for the slow response on this, I wanted to test all ways of setting up Datadomain to Veeam and have just had two

weeks of for a new baby. There are many ways that work totally fine but the easiest is the following but its critical to

configure all settings, If you miss any steps it drasticly reduces the performance of the system.
Overview: Veeam backing up to cifs share on DD and with the ability to power the server on from the backup software. Steps:
Repositry: Go in to Veeam right click backup repository and add, name it(next), then shared folder(next), put the share path (\\datadomain\backup) and username/password also direct, then click advanced and do not align blocks but do tick decompress backup blocks before storing(very important).(next) Enable vpower NFS(next). Repositry finished.

Backup Job:Right click backups and select backup, name it(next)select the server(next) then click advanced on the backup tab click incrementals select this and enable synthetic fulls. Click the storage tab untick enable inline data deduplication and set the compression level to None(very important), leave other settings. next till the end.
If you use instant recover send the change blocks to local storage on the VM host. my email:chris.hall@synapsecn.com if you get stuck

[murdocmk]

I'd definitely be interested in hearing the experiences others have with data domain using the settings pointed out by Chris. I sort of assumed most would be choosing to decompress backup blocks before storing them on a deduplication device, but perhaps not.

But even with compression enabled and poor deduplication ratios for Veeam backup data, should that really affect the random read performance from the data domain? It seems like there will always be a significant penalty there, but maybe I'm wrong. The root of the problem appears to be the random IO pattern for SureBackup/IR jobs and the poor performance of a deduplication appliance in that situation.. ?