Veeam, DataDomain and Linux NFS share

[hartza]

Hi,

I try to use DataDomain (DD610) NFS share to see if it gets better performance with Veeam Backup software than CIFS. I have added linux server to Veeam and mounted NFS share to that linux server. I am able to create backup job using the NFS share with Veeam and actually job writes .vbk file to DD NFS but then backup job just hangs. Size of .vbk is 0. I need to reboot Veeam backup server to stop that backup job.

Anyone can help me?

[Gostev]

Hi,

I believe this happens because DataDomain doesn't support flock file locking with their NFS implementation (which is very uncommon), so we hit an impasse each time we write for backup. However, it's an easy fix to disable flock on the mount with nolock parameter.

Command to mount NFS share on your Linux server should be something like:

Code: Select all

mount -t nfs -o nolock,hard,intr,nfsvers=3,tcp,bg 172.16.4.67:/data/col1 /dd/rstr01/backup 

Hope this helps.

[hartza]

Solved the issue. Can now start testing out performance of DD NFS share. Thank you very much

[ckbrou]

I just started using Data Domain with Veeam and am using CIFS to start out. It works fine and makes for a simple solution but am curious if you found NFS to perform much better? The only complaint I have with CIFS is the performance of UAIR but that may be more of limitation of deduplication technology more than the storage protocol.

[tsightler]

Data Domain has a whitepaper in which they discuss integration with Veeam. They're document (the last I saw it anyway), provides setup for both CIFS and NFS, but recommends the Linux NFS setup for performance. Veeam support has had many cases of resolving performance issues with DD by getting customers to use Linux NFS instead of CIFS. The whitepaper has very specific mount options to provide the best performance.

Also, the Linux NFS setup provides some benefits for UAIR setup, specifically the Linux NFS client cache seems more efficient than when using CIFS, thus reducing the number of request that most go to the storage system by servicing these requests from the cache. I've even helped some users configure the Linux FS-Cache for even better NFS client performance for U-AIR and Surebackup.

http://www.linuxtopia.org/online_books/ ... emain.html

[ckbrou]

I was able to get the Linux NFS setup working and also confirgured the Linux FX-Cache. U-Air works sort of. In about 2 hours it brings up my DC and my Exchange Server but many of my Exchange services will not start. The deduped storage is just too slow and the services keep timing out. When I open up the Exchange server console from the SureBackup Session it is painfully slow.

It's too bad that there isnt a way to just restore the application group to production storage but still keep it in its own segregated network. You could still use the virtual lab / UAIR functionality but it would actually be fast. Unless something like this is possible and I just don't see how to do it?

Now if I want to restore Exchange data I may need to first copy the backup file to faster disk and then run the SureBackup job from it there... but that will of course be a big pain.

[tsightler]

Wow, 2 hours to bring up a DC is worse than any I have ever seen even with DataDomain. Have you tested your Surebackup performance when writing the backups to a different storage device? I'm wondering if there is something else going on in your case that's making the performance even worse? I've certainly seen fairly large DC environments that take 40-45 minutes or so to boot from DataDomain.

[ckbrou]

The whole Job takes about two hours. That includes the DC booting (about 30 minutes), DC application tests (about another 15), and then the Exchange server booting (about 45 minutes), Exchange application tests (about another 15). The problem though is that even though the Exchange server is up and running the exchange services just will not start. They time out it seems because things are just so slow...

[cghall11]

The DD610 is a very low spec DD, Im going to do some testing on a larger one and I will update you with the results. Im EMC's premier partner BRS senior technical architect and in the middle of finding out what configurations work best for Veeam to DD backups. Update you soon!!

[ckbrou]

Thanks for looking into this! I will be curious as to what you find related to NFS vs CIFS performance. Originally I thought that NFS provided better performance but after using it for a couple weeks now I am not so sure (it seems like a negligible difference). It obviously makes the setup more complex and adds an extra hop to the network path with the NFS Linux gateway.

[novakk_AH]

[merged]

Hello,

I have a problem with surebackup job if the backup VM machine is placed on datadomain DD620.

When is size VM at about 40GB surebackup is success but bigger VM has problem with start in time.
Results: Cannot detect VM starting because of timeout

I increased the timeout value for the start but did not help. When I let the same job from another repository passes OK.

I see through the console of the machine running, but slowly and can not start up correctly service windows. ( vmware tools , MS SQL 2008 etc... )

Does anyone have experience?


Configuration network datadomain eth0a and eth0b is veeam backup

port enabled DHCP IP address netmask type additional setting
----- ------- ---- ------------- ------------- ---- -----------------------
eth0a yes n/a n/a n/a n/a bonded to veth0
eth0b yes n/a n/a n/a n/a bonded to veth0
eth3a yes no 192.168.62.21 255.255.255.0 n/a
eth3b no n/a n/a n/a n/a
veth0 yes no 192.168.51.20 255.255.255.0 n/a roundrobin: eth0a,eth0b
----- ------- ---- ------------- ------------- ---- -----------------------

[Arildm]

[merged]

Hey guys.
First time poster, and a couple of months old reader. Thx for the help i have gotten so far from this forum.

we have now started to try out DataDomain and bought a DD620 (firmware 5.0). I have read on this forum that a lot of people have managed to connect this to their veeam software, but my problem already starts here. As far as i have understood it, DataDomain is a downstripped version of linux with some extra things from EMC. But since i cant do a vi on it / check log-files or anything similar, i really dont understand how you guys have managed to connect Veeam to a datadomain.

20.03.2012 19:17:50 Fail Failed to save Backup Repository: Error: , answer: That doesn't look like a valid command, displaying help...

Help is available on the following topics:

adminaccess enclosure ntp alerts filesys replication
alias ipmi route archive license snapshot
authentication log snmp autosupport migration storage
cifs mtree support cluster ndmpd system
config net user ddboost nfs vtl
disk

Type "help <topic>" to view help for the given topic.
Type "help <keyword>" to search the commands for a specific keyword.

I manage to get this to talk to my qnap, and take backup to this one for the moment but dont have a "out of house" Disaster backup.
Hope somebody have a idea that really rocks my socks and show a IT-junior how things should be done .
i use Veeam 6.0.0 and takes backup of a Hyper-V enviroment.



-----------------
From the cold North
Arildm

------------------

[tomas.olsen]

[merged]

Hello,

I am experiencing rather bad performance when testing lets say my domain controller with veeam surebackup.
When backing up to a de-dupe storage I usually backup up with ordinary incremental backup and weekly full backups.
But a surebackup job takes almost up to an hour to complete, if it even does complete.
the domain controller is part of a larger backup job, containing from 15 to 40 vm's at different customers.
If I give the virtual backup server a second disk, does a backup of just the domain controller to a local disk, I can perform a surebackup of that domain controller in just 15 minutes.
I know that datadomain is slower on read, especially if you have both write and read operations done at the same time. So I have made sure that the surebackup runs when no other job runs.
If I try to surebackup exchange or SQL servers, they boot to slow so that about all the services is left unstarted, and the job fails. Even if I increase maximum allowed boot time.
If I open a consol to the surebackup vm, and hits CTRL+ALT+INSERT, the vm hardly responds, and each task takes almost minutes to complete.

Is there a general understanding that surebackup just doesn't work with datadomain, or could there be some bad configuration from my side?
Any thoughts or ideas??

Best regards, Tomas

[cghall11]

Hi all sorry for the slow response on this, I wanted to test all ways of setting up Datadomain to Veeam and have just had two

weeks of for a new baby. There are many ways that work totally fine but the easiest is the following but its critical to

configure all settings, If you miss any steps it drasticly reduces the performance of the system.
Overview: Veeam backing up to cifs share on DD and with the ability to power the server on from the backup software. Steps:
Repositry: Go in to Veeam right click backup repository and add, name it(next), then shared folder(next), put the share path (\\datadomain\backup) and username/password also direct, then click advanced and do not align blocks but do tick decompress backup blocks before storing(very important).(next) Enable vpower NFS(next). Repositry finished.

Backup Job:Right click backups and select backup, name it(next)select the server(next) then click advanced on the backup tab click incrementals select this and enable synthetic fulls. Click the storage tab untick enable inline data deduplication and set the compression level to None(very important), leave other settings. next till the end.
If you use instant recover send the change blocks to local storage on the VM host. my email:chris.hall@synapsecn.com if you get stuck

[murdocmk]

I'd definitely be interested in hearing the experiences others have with data domain using the settings pointed out by Chris. I sort of assumed most would be choosing to decompress backup blocks before storing them on a deduplication device, but perhaps not.

But even with compression enabled and poor deduplication ratios for Veeam backup data, should that really affect the random read performance from the data domain? It seems like there will always be a significant penalty there, but maybe I'm wrong. The root of the problem appears to be the random IO pattern for SureBackup/IR jobs and the poor performance of a deduplication appliance in that situation.. ?

[patrickds]

Also, the Linux NFS setup provides some benefits for UAIR setup, specifically the Linux NFS client cache seems more efficient than when using CIFS, thus reducing the number of request that most go to the storage system by servicing these requests from the cache. I've even helped some users configure the Linux FS-Cache for even better NFS client performance for U-AIR and Surebackup.

Am i missing something?
U-AIR uses the vPower NFS service, which is only available with a Windows target, doesn't it?

[tsightler]

A Linux repository can still be vPower NFS enabled, you simply have to select a Windows server (typically the Veeam server or a Windows proxy) to act as the vPower NFS frontend for the repository. I should point out that, while I've seen this Linux setup help in some cases, in others it's been only a very small difference. I haven't been able to fully correlate which combinations seem to make a difference, but I think it's typically lower end DD systems combined with clients that have relatively small backups (low 100's of GB per VBK). This allows the Linux server to cache the majority of required data. Unfortunately, for larger systems, with larger backups, the difference seems negligible as the Linux cache hit rate is too low.

[rbrambley]

Am i missing something?
U-AIR uses the vPower NFS service, which is only available with a Windows target, doesn't it?

Patrick,

vPower NFS still needs I/O from the backup storage repository (the dd in this scenario). We publish the requested VM restore point from within the Veeam files and actually present the vmdks through the vPower NFS, but the I/O for this process occurs on the storage device. We do not move any data off the backup target disk. The .vmx, logs, vram, and other smaller files that VMware needs to run a VM are created on the Veeam server. Hence the Linux server cache to potentially help with performance.

[patrickds]

So vPower nfs service presents data from from the linux repo, which has the NFS mount from the DD (or another nfs store).
I thought the vPower service had to run on the repository server itself.

And this is still faster than using CIFS on the DD, without the additional linux server in between?

sorry for hijacking this thread with my questions, but i was doing the research on using DD with Veeam because one of our presales guys asked me about it, and this isn't very clear in the documentation.

[rbrambley]

And this is still faster than using CIFS on the DD, without the additional linux server in between?

For using DD as a backup target - not usually. It's different in each environment, however. The extra hop could be minimal impact.

For using DD for Veeam vPower restores - usually. We are trying to make a device / disk target which was not designed for running VMs do just that from the backup files with the help of some extra cache. Veeam is about the restore jobs!

Clear as mud?

[rbrambley]

I strongly suggest DD customers check with EMC for the latest Veeam Integration Guide. My copy says internal only otherwise I would post a link.

Steps to get yours:

Log into the support portal at: https://my.datadomain.com/documentation.

3. To view integration-related documents, perform the following steps:
a. Click Integration Documentation.
b. Select the vendor from the Vendor menu.
c. Select the desired title from the list and click View.

My copy is tiled "Veeam Backup and Replication Integration Guide, Version 5 (CIFS and NFS)" from AUG 2011. I am not sure if there is an updated version. Another reason for you to get your own!

I did a quick Google search to see if there was a public copy but did not see one. Double check me and post a link here if you find one.

[tomas.olsen]

Should it be necessary to decompress before storing, if you choose not to compress in the backup job???
and why do you choose to make synthetic full that creates heavily random io pattern, instead of activly full backup every week which would just create mostly writes to the datadomain?

I will try your suggestion though, and get back to you with my experience.

[tsightler]

Should it be necessary to decompress before storing, if you choose not to compress in the backup job???

Actually no, if you disable compression on the job then the other option really doesn't do anything. However, in many cases have compression enabled on the job can still be an advantage to reduce the traffic that traverses the VM network. A common config is to have many Veeam virtual proxies, with a single server acting as a repository. This repository connects the dedupe appliance using a high-speed interconnect or perhaps even a direct network connection. The proxies send the data across the network compressed, which reduces the stress on the network, while the repository uncompresses the data for sending over the fast link.

...and why do you choose to make synthetic full that creates heavily random io pattern, instead of activly full backup every week which would just create mostly writes to the datadomain?

I didn't see that suggested anywhere and certainly you wouldn't want to do that. Incrementals with periodic fulls (weekly or monthly) is the way to go.

[cdickerson]

My two cents. Was having poor write and horrible read performance going to a DD using a CIFS share. Recreated job from scratch and disabled compression and deduplication and doing a incremental type job (not reverse incremental). Performance went from ~150Mb/s to >~500Mb/s. And the DD is doing a much better job of deduplication (more free space). Running the latest DD firmware.

-Craig

[tomas.olsen]

My two cents. Was having poor write and horrible read performance going to a DD using a CIFS share. Recreated job from scratch and disabled compression and deduplication and doing a incremental type job (not reverse incremental). Performance went from ~150Mb/s to >~500Mb/s. And the DD is doing a much better job of deduplication (more free space). Running the latest DD firmware.

-Craig

With Veeam the only number thats actually usefull to monitor is the first time run number which shows the REAL throughput. The second backup is usually done with CBT enabled and will not show the real picture, only a calculated value based on the target size and the time it takes to backup the vm. If you click each vm in the performance monitor you will however see the real CBT read/write values for each vm.

I see that you use Mb/s which is Megabit pr second or do you really mean MB Which is MegaByte /second??

When using the same settings on my backup jobs at several customers I often see performance on the first time run (which is full backup) at about 75 to 130 MegaByte pr second. while on the second run (incremental) and with the use of CBT I often see numbers at 250 - 500 MegaByte pr second.

[cdickerson]

I'm reporting speeds from my switch port. Not what Veeam is reporting.

[miguelangel]

[merged]

Hello All, what will be the recommended setup to connect my Veeam repositories to a Data Domain.
-Do I need to stop Veeams Compression and use Data Domain compressions. What would be the best compression and de dup method for this set up.

My current SAN (Equalogic PS6000) are where my repositories currently lay is basically out of space and We have 2 data domains with a lot more space and HA.

I will also like to know the best way to move the repositories to the Data Domain.


Thx to all.

-Mig

[paulhardy]

I spent many months last year working with DataDomain on behalf of a customer, backup was never an issue but trying to do anything that means booting VM's from the backup store on the DD was a complete nightmare. It all worked but it just took to long like 4-5 hours to be in a position to restore a mailbox using the wizard. Data domain and Veeam support were extremely helpful but in the end the conclusion was the 610 and DD in general are not designed to play well with random IO like running a VM from the backups held on DD. In the end I created a subset of the VM's into a second backup onto traditional storage and as you would expect the restore process was painless and ready to go in 30 minutes.

The HP D2D also suffer with the same issue using Veeam enterprise features of booting multiple VM to allow application level restores. We have spent months working with HP all the way to top guys at Bristol labs but basically all of these types of devices struggle with random IO. To be fair most of issues we have seen involve Exchange item recovery.

[dellock6]

Do not want to be a "sales-like" person, but the only deduplicated storage that works well with Instant Recovery is right now ExaGrid, since the landing zone of the appliance keeps the last version of Veeam backup in a non-deduplicated state, so when you run an Instant Recovery (99% of the times we all use the last backup...) it runs from a clean and plain CIFS storage, and run at full speed.
Problems with Instant Recovery from deduplication appliances are created by the fact DD and others have to deduplicate on-the-fly and run VM from there, and this is something really challenging when it comes to random IO...

Luca.

[zak2011]

[merged]

Hi,
Regarding storing Veeam backups on Data Domain appliances..i had some discussion with EMC. I needed to know how much storage space i am going to need on the DD for the Veeam backups.
So EMC says "If Veeams dedup data and compress data you will not be able to compress the and dedup them again with Data domain. It is better to send data natively from Veeams (without compression and dedup) and then you will get the maximum deduplication ratio on the data domain box"
" Veeam data arriving on the DD box already compressed and dedup, will have a very little effect on dedup"

I am a bit confused here. I always thought having Veeams dedup and compression plus having additional dedup of DD would give me additional disk savings.

So should i disable the dedup and compression of Veeam inorder to get the full benefit of DD?
Thanks!