EMC Data Domain

[SGalbincea]

At first I tried doing backups to primary storage, then copy to DataDomain, but since the copy jobs run the VMs sequentially rather than in parallel (I understand why, but I don't like it) this did not allow us to meet our backup windows. For now we are pointing directly at a DD2500 and getting terrific DDBoost backup speeds. Replication is handled by the DataDomains themselves and I have a scheduled task to rescan the replica repository.

Restores, however, are extremely painful. It took almost 10 hours earlier this week to restore a single mailbox from our Exchange backup (not a big one either, maybe 5GB). Hoping that we can come up with a better solution in the future, but for now this seems to be how it's going to be.

[mmaragno]

Hi,
i have a big problem with FLR using Data domain Boost.
FileServer 1.8TB.
Vmware 5.5.
Veeam 8.0 Patch1.
Datadomain OS 5.5.0.8
when the backup is mounted level veeam server before I am given the opportunity to browser files passes about 1 hour, but if i test the same job from CIFS share the mount look 10 Minutes.
where is the problem?

Thank
Mattia

[SGalbincea]

For this I would enable indexing, if you have not already, then use Enterprise Manager to browse the index and kick off the restore process. Beyond that I don't know what to tell you other than to contact support and have them validate your config.

[Gostev]

the same job from CIFS share the mount look 10 Minutes.

Just to clarify - CIFS share on the same Data Domain, or on some other raw storage?

[ingvargo]

Hi,
i have a big problem with FLR using Data domain Boost.
FileServer 1.8TB.
Vmware 5.5.
Veeam 8.0 Patch1.
Datadomain OS 5.5.0.8
when the backup is mounted level veeam server before I am given the opportunity to browser files passes about 1 hour, but if i test the same job from CIFS share the mount look 10 Minutes.
where is the problem?

Thank
Mattia

i have the same problem, both with FLR and Instant VM recovery, when doing Instant Recovery from CIFS share on DD i can login and use the VM after about 8 minutes, doing the same using DDBOOST is taking nearly 2 hours and the VM is unusable.

FLR on the same machine with CIFS on DataDomain does inventory in about 15 minutes, if i do the same using DDBOOST protocol i gave up after 70 minutes.

it seems that there is some big problem with the DDBOOST on Veeam, i have a case open for this and they are still analyzing logs.

the case number is 00816341

[SGalbincea]

While we discuss this, an FYI - DON'T create a CIFS share on a DDBoost repository that you have been backing up to in the hopes of increased performance. We tried this in testing before we were in production and it pretty much broke everything.

Hopefully though there is some improvement to be had in this integration with DDBoost - remember, we are only at 1.0 of this feature!

[Gostev]

i have the same problem, both with FLR and Instant VM recovery, when doing Instant Recovery from CIFS share on DD i can login and use the VM after about 8 minutes, doing the same using DDBOOST is taking nearly 2 hours and the VM is unusable.

FLR on the same machine with CIFS on DataDomain does inventory in about 15 minutes, if i do the same using DDBOOST protocol i gave up after 70 minutes.

it seems that there is some big problem with the DDBOOST on Veeam, i have a case open for this and they are still analyzing logs.

the case number is 00816341

This case was investigated, and appeared to be apples to oranges comparison. The job going to CIFS share is using the 1MB block size (further 2x compressed), and the DDBoost one is using 8MB block size (uncompressed). As the result, due to a random I/O nature of OS boot process, instant VM recovery from DDBoost repository had to read a few times more data from storage, which defines the time difference.

On the same block size, DDBoost is slightly faster with random reads in our testing.

In any case, I would like to remind that we do not recommend using Data Domain (or any other deduplicating storage for that matter) as a primary backup repository. They do make absolutely fantastic secondary backup targets for long term retention and archival purposes, on the other hand. For more information and recommendations, please refer to the following blog post about our reference architecture.

[stevenme]

Hello, I am experiencing some similar challenges to what has been discussed here.

We are setting up a new backup system for a new active/active datacenter environment. We have 2 sites roughly 20 miles apart with a stretched layer 2 network with dual 10GB site links and under 1ms latency. We have a single vCenter server with a stretched metro cluster with 10 ESXi hosts at each site. Our VMware datastores live on EMC XtremIO flash arrays which are stretched between the sites and presented through EMC VPLEX.

Veeam 8 is installed on a management VM which lives on the stretched storage/network and we have a physical proxy server at each site connected to the SAN fabric. We have an EMC DataDomain 4500 at each site, once fully live the idea is that each proxy server will backup the 10 hosts at the same site using DDBoost on the SAN fabric then the backups taken at each site will be copied to the other site in a sort of bi-directional replication setup.

At first I tried to use backup copy jobs for the replication but it has been nowhere near fast enough and the vast majority of backups taken never get replicated in time before the next nights backups start. I'm almost certain the native DataDomain replication will be able to cope with the amount of transfer, however as mentioned near the start of this thread the job names will be the same so you would never be sure what repository you are restoring from - has anyone found a workaround to this? Is it possible to have the name of the repository displayed in the restore wizard as opposed to just the job name?

Just another thought, if anyone has experience of using Veeam to backup VPLEX volumes, is there a better way of doing this? Given the VMware datastores are available at both sites is there a way to just backup everything at each site hence removing the need to replicate anything? I'm not sure how this would work with a single vCenter server, eg both proxy servers trying to snapshot a VM at the same time?

Many thanks in advance,

Steve

[foggy]

I'm almost certain the native DataDomain replication will be able to cope with the amount of transfer, however as mentioned near the start of this thread the job names will be the same so you would never be sure what repository you are restoring from - has anyone found a workaround to this?

Steven, if I am getting your concern right, then please note that the imported backups are displayed under the Imported node in Veeam B&R console tree, so even having the same backup job/folder name, you will be able to distinguish between the backup sets. Also, there's actually no need to import remote repository until you really need to perform restore.

Just another thought, if anyone has experience of using Veeam to backup VPLEX volumes, is there a better way of doing this? Given the VMware datastores are available at both sites is there a way to just backup everything at each site hence removing the need to replicate anything? I'm not sure how this would work with a single vCenter server, eg both proxy servers trying to snapshot a VM at the same time?

With a single Veeam B&R, the second job will simply wait for the VM to be released.

[Gostev]

Steve, regarding Backup Copy performance - you may want to obtain some performance optimization hot fixes from support that were specifically built for Data Domain, and these will improve Backup Copy performance significantly. Or, you can just wait for Update 2 later this month, which will include those. Thanks!

[stevenme]

Many thanks for the replies.

Alexander - Now I see that - I can initiate the restore from the non-imported backup to make sure it comes from the Data Domain at the local site. Ideally we wouldn't bother importing the replicated backups unless we really needed them, however it's essential in our case as the replicated repository took 14 hours to import and we wouldn't want to be waiting that long in a situation where we need to use backups! If backup jobs will wait for the VM to be released I suppose it might be possible to back up everything at each site and replicate nothing, I'll look to test that thanks.

Anton - If the backup copy jobs were able to keep up that would definitely be preferable to using native DD replication , how would I go about obtaining these hot fixes?

Regards,

Steve

[foggy]

If backup jobs will wait for the VM to be released I suppose it might be possible to back up everything at each site and replicate nothing, I'll look to test that thanks.

Note that in this case source VMs will be snapshotted by both local and remote jobs.

Anton - If the backup copy jobs were able to keep up that would definitely be preferable to using native DD replication , how would I go about obtaining these hot fixes?

Just contact technical support directly to request them.

[stevenme]

Thanks Alexander, I have raised a support call requesting the hotfix.

I was just thinking that if the entire datastore cluster is available at both sites then is there any point replicating data that is already there. From testing I've seen that if the proxy server at site 1 backs up all of the VMs on a host at site 2, VPLEX is clever enough to route the reads to the storage arrays at site 1 hence using SAN transport on the local fabric rather than NBD across the site link.

So I'm thinking I can just create 2 backup jobs in Veeam, both of them backing up the entire metro ESXi cluster, but one will use the proxy server and Data Domain at site 1, the other uses the proxy server and Data Domain at site 2. They could start at the same time but perhaps change the order of the hosts so the site 1 proxy job starts with the site 1 hosts followed by the site 2 hosts, and the site 2 proxy job does the opposite. I suppose in theory this should work but will need to be tested!

[myrdin]

I am pretty much experiencing many of the issues i have seen here. Veeam 8 patch 1 on DDBoost repo. DD2200 on 5.5.0.8. BAckup performance are acceptable, dedup on DD is fantastic but:

- FLR takes forever (2+ hours just to browse the file)
- Exchange (3+ hours for a 300GB store)

Edit: just as you know i have tried with both indexing and not indexing, and it doesnt make any difference. Also I am using the recommended settings after patch1. I did same tests before patch1 and the only difference is that patch1 the synthetic are working faster but that's it, restores are as bed. And again i am talking about to browse the files, when you get to the point to see the folder tree and you copy the file somewhere, it is pretty fast. So the problem is get to browse the files no the actual restore.

I have opened a number of case that has been opened for weeks, in which Veeam engineers kept asking me for logs (same every time), but so far pretty bad.

I dont believe DD is the issue, as if you look into statistics (with all jobs stopped only the restore running) of the DD while the restore is in process, the IOPS reported are close to 0 as well as the bandwidth, it looks like Veeam is waiting for something and DD simply sits there doing nothing. Eventually Veeam finds it and i can slowly see the folder tree. Once i copy the file back, it goes pretty fast (over 120MB/s) and i can see high IOPS and bandwidth then. Veeam seems to have problem reading off the VBK files from the DD, but the DD in turns does nothing, maybe Veeam is pulling the info the wrong way i dont know.

the other frustrating thing is that now Veeam support is started responding to customer "sorry, the problem is the DataDomain bye bye" which is very frustrating as EMC DD has been one of the main selling point of Veeam 8. If you werent sure EMC DD would have been good as primary repo, this should have been stated clearly as NOT RECOMMENDED for primary REPO.

[mmaragno]

after support webex, implement the "Decompress backup data blocks before storing" the FLR and backup now run with excellent time.
my customer ha 24TB of VM, and now the incremental of File Server (1.8TB) during 25 minute

[dumono]

Hello everyone!
I have two sites each have a Data Domain.
You must configure VEEAM for disaster recovery.
Can I use replication DataDomayn under control VEEAM? How can this be implemented? Do official documentation?

[SGalbincea]

Hello everyone!
I have two sites each have a Data Domain.
You must configure VEEAM for disaster recovery.
Can I use replication DataDomayn under control VEEAM? How can this be implemented? Do official documentation?

Veeam cannot do managed DDBoost replication right now. We have solved this by backing up to the HQ DD and letting the DDBoost replication handle the replica to DR. We can't see this in Veeam other then rescanning the destination repository, but it works for now until Veeam (hopefully) adds the manged replication piece.

[jsprinkleisg]

I have some confusion regarding backup job storage settings, and would welcome some feedback.

1. Enable inline deduplication or no? KB 1745 advises to uncheck the option.
2. KB 1745 advises changing the Storage optimization to "LAN target." But both the v8 user guide and the UI recommend the 8MB block size when using a Data Domain repository. However, the larger block size apparently adversely affects FLR and Instant VM recovery performance. So, which block size should I use?
3. Assume local disk or SAN repository for primary backup jobs, and Backup Copy to an offsite Data Domain for DR. Backup Copy jobs inherit their block size from the source backups. So, to comply with recommendation from the user guide & GUI, I should set the primary backup jobs to the 8MB block size. This would likely reduce dedupe savings on the primary repo. And again, won't this adversely affect FLR performance, even if restoring from the primary repo?

[foggy]

James, please review this thread discussing your 1. and 2. questions in detail.

Regarding the third one, yes, your understanding is right, however primary storage is typically much faster and deduplication is not so important there since just a short chain of recent backups for operational restores is stored there.

[jian17]

I am pretty much experiencing many of the issues i have seen here. Veeam 8 patch 1 on DDBoost repo. DD2200 on 5.5.0.8. BAckup performance are acceptable, dedup on DD is fantastic but:

- FLR takes forever (2+ hours just to browse the file)
- Exchange (3+ hours for a 300GB store)

We are having the exact same issues with FLR and the absolute latest version of Veeam. Backing up to the DD is fast and we've implemented every setting suggested from the forums and the white papers from Veeam.

Full VM recoveries only get 15-30MB/s restore and FLR only sees about 1-2MB/s. We tested a Netbackup restore which also stores to the DD and the same FLR went at 90MB/s.

I've called Veeam twice and each time they tell me those speeds are good and not to expect much more. That can't be a valid answer.

[foggy]

I've called Veeam twice and each time they tell me those speeds are good and not to expect much more. That can't be a valid answer.

Those speeds are not good, however keep in mind random reads during FLR and overhead required to rehydrate deduplicated data, so restore performance would never be close to what you're seeing during backups. You can always ask for a closer investigation though.

[jian17]

Thanks Foggy,

Yes, we have a ticket open and our rep is escalating it. I'm totally OK with the overhead of rehydration but it's been so slow on FLR that it's almost unusable. Most restores in our latest tests drop down to 100-200K/s on a DD2500.

[ian0x0r]

I too have this issue trying to recover using Other OS FLR from a DD Boost repository. Restoring using Windows FLR from the same backup set on DD Boost storage takes 12 minutes. Other OS FLR times out after an hour but the logs don't seem to show the job is actually doing anything. The VM is a 2008 R2 File server which is 4.2TB in size. Everything else works fine, its just the other OS FLR times out for some unknown reason. Other OS FLR fires up straight away if mounting the same data from JBOD before the data is processed by a backup copy job.

I have a case open with support 01016664 but this is not the first time I have logged this issue and was also told previously that it is the DD at fault, which I don't believe. To me it seems there is a five minute timeout trying to mount the data to cache which you can see from the logs below:

04.09.2015 13:23:06] <05> Info [LeaseKeeper] Created, ttl 120sec
[04.09.2015 13:23:06] <05> Info Mounting restore point. Vm: [FileServerName], BackupDate: [23/08/2015 22:11:44], Oib: [08c8a858-8d03-475c-82ee-8c2b20a8ad76]
[04.09.2015 13:28:22] <28> Info Removing from cache. Reason: Expired
[04.09.2015 13:28:22] <28> Info [Soap] Logout from "https://CustomerVC"
[04.09.2015 13:28:22] <28> Info Server thumbprint: CAE6EF444DCC9C8852755364A59807A6F8C41939
[04.09.2015 14:23:38] <05> Error Failed to mount oib "08c8a858-8d03-475c-82ee-8c2b20a8ad76". Lease: "79b81a72-646e-4bf2-bf7e-3acdec2888c4"

Where as on a successful job it looks like this:

[03.09.2015 08:56:09] <05> Info Mounting restore point. Vm: [FileServerName], BackupDate: [23/08/2015 22:11:44], Oib: [cb3c0f42-a84e-4e77-86dd-c4ee16dab635]
[03.09.2015 08:59:52] <05> Info Restore point was mounted. Result: [MountFolder: ;ApplianceHostId: faa07a45-3ff3-4f3c-9307-386cb7cc3f22;]
[03.09.2015 08:59:52] <05> Info [Creds] Creating credentials 'root'
[03.09.2015 08:59:52] <05> Info [Ssh] Creating new connection [host: 'xxx.xxx.xxx.xxx', port: 22, user: 'root', elevation to root: 'no', autoSudo: no, auth type: 'Password'.].

Did anyone else get a fix to this or has noticed the 5 minute timeout trying to mount restore point using Other OS FLR?

Thanks,

Ian

[foggy]

Ian, any particular reason for using Other OS wizard for Windows VM?

[ian0x0r]

Hi Alexander,

Yes I wanted to use the FTP recovery option as I have 10's of thousands of files to recover. FTP also provides the option to recover if data has changed or the file size is different. I was told in the past by a Veeam engineer that FTP is better for lots of small files to be recovered, and on the whole I have to agree.

Thanks,

Ian

[foggy]

Got it. I must admit that Linux FLR from DD Boost does indeed have some lack of performance in the current version. However, some optimizations are coming.

[ian0x0r]

Any hints at what the optimisations are going to be? Or even an interim hot fix?

If there is some kind of workaround to extend all FLR timeouts to their maximum I'm happy enough with that for now. I have already set remotingTimeout to 7200 and FlrVmToolsWaitingTIme to 3600.

I suppose for now as a workaround I could write a script to extract the files and only overwrite if date or size modified using Windows FLR.

[foggy]

Stay tuned for future v9 announcements. Regarding timeout, I'd consult with your support engineer.

[chris.childerhose]

Anyone tested v9 with DD? How is it? Just wondering as we are getting ready to implement with a DD2500 and DD670.

[dustinn3]

I loaded it last week and it seems to be working well with my DD2500. I'm not sure about any increase in speed or dedupe yet. I did have an issue with Veeam saying the files were locked on my DD because my proxy rebooted during a backup job. However, I rebooted the Proxy and my Veeam Server this morning and it ran fine.