Backing up to VTL, DataDomain

[bvdutchie]

We need to backup to disk and are looking at purchasing appliances that dedup however most use VTL (virtual tape library). Is Veeam developing a solution to write to tape so this would work? I don't have the space nor the time window to write to disk then backup to the VTL.

[Gostev]

Barry - yes, we have plans to add support for VTL in future versions. However, as far as I know most widely used dedup appliance do not rely on VTL. For instance, if we take a look at dedupe storage market leader (DataDomain), they do not require VTL. Veeam is actually technology partner for DataDomain, and Veeam Backup is officially DataDomain-compatible and included into their support matrix.

[bvdutchie]

We actually use DataDomain at the moment and are looking for an alternative....they sized us for the wrong unit and aren't taking responsibility for it (plus they are way over priced especially with many other players entering the market). I know that you are tech partners however I have to open a case with DataDomain because all my Veeam backups don't dedup very well (4x).

[Gostev]

Barry, do you have compression disabled in Veeam Backup job's settings? This is required in case you are storing your backups on DataDomain device.

[bvdutchie]

Yes, compression is disabled. About 25 OS drive only backups of mostly W2K3. I should be getting better compression then 4x.

[Gostev]

Barry, and what compression levels do you get based on source data size? Veeam Backup does basic first level deduplication as well (as part of backup process), so this should be taken into account too. Or is it 4x based on source data size?

[bvdutchie]

4x is based on source data size. 422GB original bytes and the local size is 105GB.

[Gostev]

Yes, this is not too good. I wonder what is the reason. I've been seeing much better results in my testing with DataDomain (although my test VMs were made from the same template).

[bvdutchie]

Yes, we use the same template for most of our servers. I guess DataDomain doesn't want to address it; this is your response:

In regards to your issue you describe below, I checked in with our escalation team on this and unfortunately, Veeam SW does not "chunk" data as most backup applications do. It throws multiple small files at the restorer, therefore compression rates will be somewhat lower than traditional compression rates are with other backup software suites. Our apologies if this is not what your understanding was in this regards. With other backup software suites, you can indeed see those higher compression rates.

[Gostev]

This response does not make sense to me given the full compatibility testing that has been performed by DataDomain during certification. I was the one working with the compat testing lead and he was actually happy with compression he was getting. The only issue mentioned was slow performance in case of CIFS integration (no such issue in case NFS integration). This issue still being researched (could be due to CIFS protocol implementation on DD devices side, or due to the way Veeam Backup does writes - accorging to compat team lead). I've just contacted them for update a few days ago, because we've taken a look at how we do writes from our side and we could not confirm this.

Neither we "throw small files", our VBK is actually large monolithic file. I also don't understand what "chunking" are they talking about, that other backup software does provide and we don't.

Anyway, I will ask them about this as soon as I get response back. Thanks for letting me know these details.

[burakuysal]

Hi

I am a Veeam fun, I am doing some testing with Veeam and Data Domain, I am surprised to see quite a lot of issues.

Testing is done with Veeam 3.01, DDR120 and DDR510 firmware 4.6.1.0-88314, ESX 3.5, VC 2.5, Proxy X86 Win 2003, VCB SAN connection.

Compression: Backing up AD VM 15GB Hard Disk
Disable compression in Veeam, 1.8GB .vbk file created on DD share. On the actual DD box filsystem space shows pre-comp 1.8GB and post-comp 2.7GB

Actual disk allocation in DD increased giving a - dedupe ratio. (I also tested with Vision Core and Standard VCB scripts post-comp was 1.3GB)

My Veeam backup job finished with success, I can restore Full, Individual .vmdk files but not File Level Restores. Mount operation always fail which I tested with different VMs and result is same. On VCB mode error is about decompression error expected bytes xxx and actual xxx with a few KB offset.

I am still doing my testing but it is not going very well. In DD OS Comp Guide Version4.6.1.0, only supported protocol is NFS. Surely you use CIFS on VCB Proxy running Windows.

"Table 43: Veeam Backup and Fast SCP 3.0 Compatibility OS Version CPU Min DD OS Protocol VTL Tape Driver/Changer Driver
Replication Notes
Windows 2K3 x86 4.5.3.0 NFS N 1
1. Set compression to none for each network backup. By default, the Veeam 3.0 Windows
backup server sets this function to on."

I appreciate some more information about Veeam and Data Domain. There is no integration document for Veeam from Data Domain(There is one for Vranger). I also do not have any documentation from Veeam side with DD integration.

Did you get any feedback from Data Domain about your enquiries?

Regards,

Burak Uysal
Storage Consultant

[burakuysal]

The reason I have seen so many issues is because CIFS not supported with Data Domain and Veeam. That is why it is only certified with NFS in DD oS Compatibility list. I used Data Domain NFS shares to ESX server further on with my testing.

Aim:
- Fast and easy DR test
- Consistent backups in DR site
- Reduced bandwith
- Meeting RTO for DR

All above achieved by the design below;

Design elements : 2 Data Domain appliances 1 at main site(ddr-main), 1 at DR site(ddr-dr). Veeam installed in a VM in main site.ESX servers on each site.

To be able to get best of both worlds, I use Data Domain dedupe and replication to DR, Veeam replica jobs backing up VMs to NFS datastore presented by Data Domain.

If I use Veeam backup instead of replica then I end up having .vbk files despite turning compression off. This means I have to use Veeam to import and restore the backup in DR site, increasing the steps for DR.

If I use Veeam Replication jobs and turn compression off then I end up having Virtual Machines files as .vmdk, .vmx, .log. Do not worry you get extremely good dedupe ration with Data Domain. These Virtual Machines are intact hence you can boot straight off a Data Domain datastore in DR.

I explain the setup step by step;

1- Assign a share from Data Domain to a ESX Stdby server. Create a datastore on this share for ex NFS datastore /backup/NFS. This server will be the target for Veeam Replica jobs. You can also use production servers if you can put an extra 2 Ghz CPU load on existing ESX servers. If answer is no you can use Replication with VCB.

2- Create a Veeam Replication job with Compression set to "None". In Destination tab choose Stdby ESX server and NFS datastore we just created in Step 1. Configure VSS, versions to keep etc.

3- When you run the job all replicas are kept in /ddr-main/backup/NFS/VeeamBackup. Setup replication from this share to DR DD.
replication add source dir://ddr-main/backup/NFS/VeeamBackup destination dir://ddr-dr/backup/NFS/VeeamBackup

4- Now we have all our replica VMs at DR site. Create a datastore similar to the one step1 this time from ddr-dr to ESX DR /backup/NFS, name the datastore as NFS.When you browse this datastore you will see VeeamBackup main folder and all replica VMs in subfolders. Do not forget this is the replication target and it is Read Only. So this is not any good to us.

5- Copy all the contents of VeeamBackup to another folder to make it R/W. There is a superb functionality called fastcopy which copies 1TB pre-comp, 50GB post-comp in 0.02 seconds! Actually creates logical pointers to this folder, does not physically copy anything.
command : filesys fastcopy source /backup/NFS/VeeamBackup destination /backup/NFS/DR

6- Browse to Datastore again you will see a new folder called DR has all your replica VMs. Simply add .vmx files to inventory and power them on. Your DR test is complete. If you are in a real DR situation then consider getting critical systems booting from Data Domain and start Veeam FastSCP copies for others. You can also use SVmotion to move critical ones back to primary storage while they are running.

In summary this is the fastest DR you can achieve, very efficient solution. I had 20:1 dedupe ration on my first full replica job (20 VMs 1TB). You should expect at least 40:1 over a months retention.

You can still have File Level Restores with Veeam Replication jobs, you can roll back your VM to a previous version. More importantly because of file structure you do not need to run another restore in DR situation, you do not even need a Veeam DR backup server simplifying steps in DR and DR tests.

You should consider performance of each Data Domain model when you decide how many simultaneous jobs to run, 1 or more Veeam backup servers needed or not.

Throughput info for models running 4.6 Firmware

DD120 - 83 MB/s
DD510 - 120 MB/s
DD530 - 150 MB/s
DD565 - 277 MB/s
DD660 - 555 MB/s
DD690 - 750 MB/s
DDX Array - 12000 MB/s

Contact me if you have any questions about my setup and testing.

Gostev : Can you give us an update about CIFS and DD compatibility? I believe you are working on it.

Burak Uysal
Storage Consultant

[Gostev]

Burak, thank you for such a thorough description, this is very useful.

Here's my CIFS and DD compatibility update. I've brought this issue up with DataDomain again. They have confirmed that during their testing, they do not see the same "slowness" when Veeam Backup was writing to regular Windows share, so this slowness is specific to DataDomain device design. However, they said they are not aware if anything is being done to address this issue from their side.

We have also analyzed some DD device side network traces from their testing. From the traces, it appears that DD devices do not "like" when data is manipulated in very small blocks in CIFS mode, and this results in very slow performance. So, we decided that we could probably address this faster from our side, instead of waiting for DataDomain to fix CIFS implementation. We will be bringing this enhancements as a part of some other backup storage related improvement that will come in Veeam Backup 4.0.

[ckd5150]

what was done in the 4.x release to address this?

[Vitaliy S.]

Hello Charles, please check out the following links for the information requested: V4.0 and Data Domain and Data Domain and Boost software utilization