VEEAM with Data Domain as PRIMARY TARGET = NOT GOOD!

[Gasp100]

Am I screwed??? VEEAM is pretty cool, even in POC we have run into a number of strange issues but I was able to basically build out the infrastructure by myself and things are working. But at this point, EXTREMELY SLOW and abysmal RESTORE performance has reared it's ugly head... I have a NexSAN E48 (LOVE NEXSAN!) which I can carve up as a staging or landing zone for VEEAM, but that was not in the cards and it's VERY disappointing to have done all of the work with VEEAM pointing to a DD640 with expansion shelf in PROD and a DD2500 in DR only to realize restore times SUCK TERRIBLY... Sorry, no other way to describe... VEEAM's Exchange (and SQL, and SharePoint) explorer look EXTREMELY ATTRACTIVE, SureBackup (if it works) looks really neat and I must get replication tested and working as well. But testing an Exchange restore of a 30GB database (from a 1.5TB total sized server) is ludicrous... so many hours to mount I fell asleep.
I have EMC Networker AND Microsoft DPM 2012R2 in house and have used for years and I'm yearning to get that crap OUT for VEEAM but I'm very disappointed in the fact I can backup like a charm but can't restore for sh*t LOL. Full VM's = OK, file explorer = eh, anything "application" where it wants to mount the backup to be able to restore info (Exchange, SQL, SharePoint) = laughably slow.
As it stands now I have to push VEEAM components to ANOTHER physical server, connect it to my Nexsan E48, carve up storage and present, TEST regular backups/restores, repoint jobs currently backing up directly to Data Domain (not sure if I want to try and copy existing and re-import or just let that fall off the Data Domain due to attrition), redo copy backup jobs.... oh yea, and if I ever expect to use VEEAM in DR looks like I MUST use replication for everything or have another staging area in DR before using the DD2500 out there for longer term retention. And who is to say someone will be willing to sit for 2 days for a 3 month old backup to be restored anyway???
I'm in talks with Commvault and Avamar / EMC and while I'm trying to stick up for my $55K investment in VEEAM it's getting harder and harder to justify while Avamar integrates incredibly well with Data Domain and Commvault just kicks ass (but is expensive as hell).

If someone can tell me VEEAM deduplication to "normal storage" NexSAN array will allow an acceptable backup/restore/replicaton/copybackup scheme and give me a DECENT amount of deduplication and savings on storage it will help.
So, what do you have for me?

[foggy]

The subject tells it all, we do not recommend using any deduplicating storage as a primary backup repository. That said, the soon-to-be-released v9 will have some optimizations in FLR, especially for DD Boost enabled repositories.

[tjgrie]

Why were you unable to use the Nexsan as a backup repository?

We have two DD2500 and while restore performance isn't amazing, it's tolerable. I believe that some OS versions of the Data Domain's work better with Veeam, or at least I recall seeing some posts on the forum indicating this to be the case. We also replicate the backups from the production DD to the DR DD using the native DD replication. We figure remounting them in Veeam won't take terribly long on the off chance they would be required.

We also use Avamar for physical machines, and if you mean "integrates with DD" by using it as a backup target, that's about it. EMC seems to be steering people away from the Avamar hardware and backending everything with Data Domain. We tried out Avamar's VM backup capabilities about three years ago and weren't terribly impressed. At the time it required appliance VMs and the Avamar client to be installed on the VM (if you wanted to do granular/application aware backups, which still took another scheduled backup to accomplish.) This may not be the case today. Combine that with the archaic GUI and JRE requirements, it's not a terribly fun system to administer.

[emachabert]

First, using CAPS won't change anything except looking rude
I think the main concern here is about the lack of understanding of what is an inline deduplicating appliance and how it works. The issue is neither with Veeam nor DataDomain but in the infrastructure design and the components placement.

DD is made for high rate deduplication, on very small blocks, on sequential worload.This mean very good at writing and quite good at reading big deduplicated files in a sequential I/o pattern. These appliances are designed for processing multiple parallel datastream to get the best of it.

SureBackup or Explorers require random read I/O over multiple files (unless you restore from a full).

Therefore, that is why you shouldn't use DD as primary backup storage. You should stick to veeam reference architecture and use a landing zone that you will design based on your RTO/RPO goals.

[Gasp100]

So why didn't my salesrep NOR their senior SME even FLINCH when I mentioned multiple times that I was going to use a DD640 (up to the latest DD OS possible) and replicating to a DD2500 in DR. BTW, those contacts who were very eager to attend calls and answer technical emails have been MIA ever since... Backup is not the only thing I do, far from it; in fact I spend a good amount of my time on Exchange infrastructure and I can tell you from direct experience that the VEEAM team is severely lacking in REAL WORLD Exchange infrastructure experience from a DAG backup/restore/testing/DR standpoint. But I digress...
I'm now blowing away an HPDL360 with dual HBA connected to my pristine NexSAN E48 and I'm expecting to be able to:
Add this new machine + fibre channel attached storage as my new repository (primary in PROD)
Reconfigure existing jobs (which are in production use BTW) to use this new repository
Try to mount my PROD DD640 as a CIFS share (something I don't normally ever do) and MOVE multiple TB of deduplicated VEEAM data that now needs to be rehydrated to the Nexsan... then learn how to "import" backups in VEEAM so everything is glued back together?
THEN... reconfigure my copy backup jobs to write to DD640 for onsite PROD backup with longer retention
THEN... create new copy backup jobs that write to PROD (Nexsan) to DR DD2500 directly?
-----
Or, should I skip the whole "importing all old backup data" and just reconfigure current backups on Nexsan and copy backups on DD by tweaking retention policies to get what I want faster?
The long and short of it is our retention period is ludicrous (3 months of DAILIES and disk, so ~90 days). 6 months of WEEKLY's on tape (we do tape out, but my dream is to use Data Domain with replication (from the backup solution level or DD itself) to someday rid ourselves of tape altogether.
-----
Look, I really like VEEAM but there are obviously some quirks and it's not good that it doesn't work well with Data Domain - that's just fact. I haven't seen the information " we do not recommend using any deduplicating storage as a primary backup repository" ANYWHERE in my research before or during POC. And if this is the case school your sales reps and SME's so they are on the same page.
Who cares how fast you can backup if it's next to impossible to restore?
-----
Anyway, I have this afternoon to get this accomplished... maybe you can provide some insights into questions above and the forum will be a huge benefit for me? I'm over the Data Domain for now and need VEEAM to work and accomplish our goals for the next 2-3 years at least.
BTW, my VEEAM server is not physical because I was told it needs to be close to the production storage (along with all of the proxies). I'm hoping that since my paradigm is shifting and I'll be pushing backups to a windows 2012r2 server with NexSan attached that I don't incur major networking issues.
DDBoost kicks ass over network, hoping this new configuration can keep up?
-----
BTW, I assume I should enable VEEAM deduplication on these new backups going to the NexSAN? How and what is my best option of configuration? This is plain old 7200RPM SATA II disk.

[tjgrie]

I don't think you'll be able to share a DDBoost repository as a CIFS share on the DD. I know when Veeam implemented DDBoost, we went from using CIFS shares on the DD to the DDBoost backup repository and were unable to move the backups around. I think there was a way to accomplish it via the command line on the DD but I decided it would be easier to redirect the jobs one by one and manually remove the data on the CIFS shares when they expired. There might be a better way to tackle this.

The Veeam Management server doesn't need to be in the production environment or close to the production storage if you have proxy servers in place. In our setup the management server sits in DR, and we have two physical servers to act as proxy servers, one in production and one in DR, connected to the SAN network. The proxies handle the movement of the data to each other for replication and the production proxy chats with the DD. It really depends on how you want the environment setup, whether you can do direct SAN access, or the virtual proxies.

[Gasp100]

Ok, that's good to know about boost (LOL)? I guess it makes my job easier and not even attempting to import data from the DD's back into the NexSAN.
My question is that I'll be setup similar to if I had a NAS, right (or DD)? The B&R server just directs backup "clients" where to back to (in this case my NexSAN). But for the backups to be pulled from PRODUCTION SAN storage (EMC VNX) and make it's way -- VEEAM proxies or not -- to this new VEEAM target (windows server with fiber connection to NexSAN) it will all have to traverse the network, right? Should I be concerned? This is basically how it worked with the DD but I don't know if BOOST assisted with this network traversal (to or from the DD).

[emachabert]

If I understand well your setup :
Production storage : EMC VNX
Landing zone: Nexsan
Long term : DD

Best would be to use the DL360 as a proxy/repository, using only FC SAN connection. So your job would be using directSAN from the VNX to the Nexsan, the DL360 acting as the computing gateway. In v9 you will be able to leverage backup from VNX snapshot if you upgrade to E+ licence.
Then make a backup copy job from the Nexsan to the DD using DDBoost integration to allow fast syntetic transform.
You'll then have the most efficient kind of setup. You would then buy a second DD to have an asynchronous repoication to an offsite location.

[Gasp100]

I only have one port open in the brocade fabric (which connects to the production EMC VNX) but I might just try that... so, connect the HP DL360 in actually to BOTH production SAN and landing zone SAN (nexsan E48).
Right now I'm doing a test of the current setup with a one off (large VM, Exchange mailbox server 1.9TB but only 300GB worth of mailbox store). This way I can gauge a few things but most importantly how long a RESTORE of mail data will take using the Nexsan as landing zone for immediate data.
Thanks for the input!

[tjgrie]

That's definitely the way to go. It should make for some pretty quick backups and recoveries. Just make sure the Windows box doesn't initialize the VM datastores.

http://www.veeam.com/kb1446

[Gasp100]

OK, I'm seeing ~150MB/s in VEEAM for a singular backup job to the Windows Server + NexSAN and on the NIC in the Windows server I'm averaging 958Mbps
I'm going to let this run, attempt an Exchange restore (using the Exchange explorer and report back). If I see vastly improved results I may move even further and get this server connected to the production EMC VNX as well.
We did also perform a FLR test (simple windows file recovery) where the data was living on our production Data Domain and it wasn't terrible as per my coworker... So, we may just move the transaction application workload to the Windows server + Nexsan and keep any basic flat file stuff directly on DD... have to test more.

[emachabert]

That's definitely the way to go. It should make for some pretty quick backups and recoveries. Just make sure the Windows box doesn't initialize the VM datastores.

http://www.veeam.com/kb1446

Since version 7, veeam proxy setup process automatically disable the automount feature on Windows to avoid such dramatic events !!

[james.gale]

That said, the soon-to-be-released v9 will have some optimizations in FLR, especially for DD Boost enabled repositories.

Hi foggy, please tell me more about the proposed optimizations. What's coming?

I use a DD2500 extensively.

As a general rule I find the backup ingest very good, but the restore performance is abysmal. Doing a few tricks like defragmenting any heavily fragmented filesystems at the source helped, but it's still not particularly great. (One particular 1.2TB fileserver used to take 7-8 hours to populate the FLR file browser, now it's down to ~25m).

OP: Watch out for other DD tasks like replication and cleans. They can take a significant chunk of resources.

[kjstech]

Oh no. I have priced out two data domain units with replication for submission to next years budget. Goal is to replicate to a remote unit over the WAN. Would my money be better spent on building a custom FreeNAS target or their production paid system (or something else)?

We currently use FreeNAS but only have enough space for a weeks worth of backups. DD would give us a years worth of backups, plus allow us to replicate over the wan to protect the data in case of a physical event.

I've been on a few EMC calls and presentations on DD and everyone knows my intentions of using it as a veeam backup repository and everyone is also 100% confident in using this appliance for this purpose.

[tsmith_co]

I have many customers using DDs as their primary targets. While any dedupe appliance is naturally slower than other targets, their are things to do to get the best speed possible.

For DataDomains:
Use DDBoost for fastest transfers - only use synthetic fulls if using boost- otherwise active fulls.
If not licensed for DDBoost, use an NFS proxy. Use CIFs (smb) as a last resort. This will be the slowest. Switching to NFS will just about double the speed. Boost even faster.

To setup and NFS proxy, see here:

http://tsmith.co/2014/veeam-and-datadomain/

Dedupe appliances work best with multiple streams. So, even though 1 stream will max at about 125MB/s, using 2 streams can get more. To use multiple streams, you simply setup multiple shares and tie them to multiple repositories in veeam. However, this is more mgmt overhead.


V9 will increase this even more - but obviously doesn't help for right now.

[tgillispie]

Not all dedup appliances are created equal. ExaGrid's family of scale-out appliances fixes the slow InstantVM Recovery and restore problem by having a high-speed disk cache in each appliance that holds the most recent backups. NO need for primary storage in front of your dedup solution - it's baked in.

I am ExaGrid's product manager, so I won't consume forum bandwidth with all the ways Veeam + ExaGrid work better together at an industry-leading price point. Contact ExaGrid or me for more details.

[james.gale]

Oh no. I have priced out two data domain units with replication for submission to next years budget. Goal is to replicate to a remote unit over the WAN. Would my money be better spent on building a custom FreeNAS target or their production paid system (or something else)?

We currently use FreeNAS but only have enough space for a weeks worth of backups. DD would give us a years worth of backups, plus allow us to replicate over the wan to protect the data in case of a physical event.

I've been on a few EMC calls and presentations on DD and everyone knows my intentions of using it as a veeam backup repository and everyone is also 100% confident in using this appliance for this purpose.

The solution can work fine and there's a lot to like about the DDs, especially the async replication - but there are also a number of caveats you should be aware of. How much front-end data are you looking to back up each day? What model DDs are you looking at?

I'm interested to hear exactly what's being proposed in v9 as I don't believe it's possible to do a perfect DD/Veeam install at the moment due to a combination of the way Backup Copies currently works, and the relative slowness of DD restores.

[emachabert]

The problem with dedupe appliances is not for full restore or file item restore, for this basic task it works most of the time (not super fast, but very acceptable).
Problem is for application level item recovery (SQL, SharePoint, Exchange, AD,ORacle), Instant Recovery and SureBackup.
In those cases, appliances are not built for heavy random I/O workload and therefore do not perform well (or do not perform at all)

[foggy]

Hi foggy, please tell me more about the proposed optimizations. What's coming?

Cannot comment at the moment, details will be available along with the release.

[foggy]

Dedupe appliances work best with multiple streams. So, even though 1 stream will max at about 125MB/s, using 2 streams can get more. To use multiple streams, you simply setup multiple shares and tie them to multiple repositories in veeam. However, this is more mgmt overhead.

You can get multiple I/O streams even with a single repository, if running multiple jobs against it.

[danielbastos]

Hi Foggy,

I restoring from DD610 and getting a stream about 10MB/s in a Gigabit network. How can I get multiple streams with full restore of a VM to increase the restore speed?

[foggy]

Daniel, multiple write streams to the backup repository created by different jobs are meant here. Restore of a single VM is always a single read stream.

[kjstech]

Is V9 a Q1 2016 release? That's when we'd like to slot our backup target upgrade. Very interested in Exagrid now after reviewing their website. When I get back to work next week I will be reaching out to them for more info. I think we have 50-60k set aside for this project and if Exagrid can fit within that budget it sounds like a no brainier (so far).

But the replication of DD is a huge selling point and I want to ensure whatever we go with can have the same ability. So that's also why I'm very interested in Veeam V9 release because if that fixes the issue we can stay on our current path.

[rereduck]

Hi,

I'm using Veeam V8 to backup directly on a DD2500 with DDBOOST.

I agree that restore is slow when using veeam guest file but for me it is acceptable. When I have to restore with guest files this is generally not very urgent. For example, I have to wait arround 2/4 hours until Exchange explorer pop up but restoring an entire VM is faster so that's not a problem for me.

[rreed]

Same experience here, directly to some DD620's and a couple of DD2200's. I inherited the backup infrastructure when I came in; Veeam 8.0x (VM, w/ VM proxies) straight to the DD's. Numerous perpetual issues. Bought a couple of Dell DR4100's to begin retiring our DD's, not much better experience. All seem to fall off the network (network name no longer available, connections reset, can't write to file, etc.). Our record for days w/o backup incident is 4. FOUR. In my previous life I backed up to just a large HDD on a Windows server (windows share), then pulled the backup files off to tape. It just, worked. We recently collapsed one of our satellite locations - this one actually just backed up to Windows HDD as well and just worked - and an old beater SAN we hauled out will get pressed back into service as our "staging" area for 1-2 retention points that will then get (attempted to) copied to our dedupe appliances.

They do make a great sale, artificially high dedupe/compression rates, promises of eternal storage, speed of light, etc. Write speeds are indeed pretty good if architected well (hint: setup LACP and as mentioned do multiple shares and make sure you crank up the multiple streams and concurrency). They are advertised to ingest well and can. Restore? Set end-user expectations realistically. Be prepared to wait days depending on how large and how many folders deep, or especially a folder full of files. Get Indexing working - this will save you an enormous amount of time compared to manually browsing the File Explorer.

As a backup to our backup, because direct to dedupes simply doesn't work well, we also have a separate tape backup system. It just works.

[lgomez]

Good luck with your Data Domain. We have a DD670 and we were in the same boat. Backups are ok but restores suuuuuuck!!!!! In my opinion DD is awesome for archiving/writing not reading. We set Veeam not to do compression or dedupe so DD can do it all like is suggested. But when you need to read the data all that data needs to be uncompressed and dedupe back to VMware. The problem with DD comes with adding and deleting. When you have a backup rotation for jobs that purge jobs often like every week that is when the problem starts. DD slowly start to get fragmented and you start running into "bad file locality" or something like that, that is what DD calls it. Then you call them and they will tell you that they can only fix couple of files or locations. Why a couple and not all? For what they told me because it will fix some and break others, go figure. We switch to a JBOD attached to the backup server running 2012 R2 using storage spaces and backing straight out of the SAN, by the way we were using SAN with DD as well. Is working incredible great for us. Veeam is now doing the dedupe and compression. I can't use windows dedupe because the size of the storage space volume is bigger than 64 TB. In case one of you say something about the dedupe and compression that DD offers you are correct, but for 34K I got a JBOD with 242 TB usable. Ask for a DD quote... just don't get a heart attack when you get the quote from EMC it will be probably close or over 100K. Do the math. Go check storage spaces it is a pretty cool and it is free.

[nmdange]

Good luck with your Data Domain. We have a DD670 and we were in the same boat. Backups are ok but restores suuuuuuck!!!!! In my opinion DD is awesome for archiving/writing not reading. We set Veeam not to do compression or dedupe so DD can do it all like is suggested. But when you need to read the data all that data needs to be uncompressed and dedupe back to VMware. The problem with DD comes with adding and deleting. When you have a backup rotation for jobs that purge jobs often like every week that is when the problem starts. DD slowly start to get fragmented and you start running into "bad file locality" or something like that, that is what DD calls it. Then you call them and they will tell you that they can only fix couple of files or locations. Why a couple and not all? For what they told me because it will fix some and break others, go figure. We switch to a JBOD attached to the backup server running 2012 R2 using storage spaces and backing straight out of the SAN, by the way we were using SAN with DD as well. Is working incredible great for us. Veeam is now doing the dedupe and compression. I can't use windows dedupe because the size of the storage space volume is bigger than 64 TB. In case one of you say something about the dedupe and compression that DD offers you are correct, but for 34K I got a JBOD with 242 TB usable. Ask for a DD quote... just don't get a heart attack when you get the quote from EMC it will be probably close or over 100K. Do the math. Go check storage spaces it is a pretty cool and it is free.

Just to chime in, I have a similar setup with Veeam except it's a physical Windows server with a hardware RAID controller instead of Storage Spaces - though I love Storage Spaces as our primary storage for Hyper-V! I also have a second physical server in a remote location and use Backup Copy jobs to copy the data offsite, so no tape changes required. Performance for both backup and restore are excellent, the bottleneck for our backups is always the source system, never the target.

[rreed]

Another issue w/ dedupe devices' abysmal restore rate is Veeam timing out on FLR's. Out of the box the file browser (or just restore progress window? Veeam help me out here please) will time out and just go away on its own after 30 minutes if you don't click the I'm-still-here button - which I think only gives you 30 seconds to do so!! (please fix this). There is a registry hack (or Veeam could just remove this entirely please, thanks!) to crank up the timeout, but even if you effectively disable just the browsing timeout, even once we select our files/folders to restore, it takes SO LONG to get the data back out of the DD that our FLR just times out/craps out in general and fails. Restoring a large amount of data? It takes days anyway, and after days of waiting and it craps out and you have to start all over? People start getting mad.

Case in point, FLR from one of our 620's, I started a FLR (was able to actually select my folder) 11/04/2015 10:00am. On 11/09/2015 05:50am it apparently died w/ "session became orphaned." I'm researching this right now. The backup job also failed w/ some file locks on I believe this particular backup file. So the FLR has failed after working for five days, and the backups also failed. Our record of FOUR days w/o backup issue still stands unbroken.

[emachabert]

Case in point, FLR from one of our 620's, I started a FLR (was able to actually select my folder) 11/04/2015 10:00am. On 11/09/2015 05:50am it apparently died w/ "session became orphaned."

5 days to restore selected items and it failed ? Looks like there is an environmental issue here. Are you sure you don't have any issue with your network ? I've seen a lot of slow restore from dedupe devices but never that slow.

[rreed]

Perceived network issues are the hallmark of our DD experience. Everything else on our network works fine but all our DD's seem to randomly drop off the network during backups. Veeam ONE has been throwing numerous repository not availble for one of them all day/night, jobs randomly/frequently fail w/ network name no longer available, connection forcibly closed by remote host, etc. on all six of ours. I've been working w/ flow control b/t our Cisco switches and most recent/most problematic DD to no avail. My hot backup project right now is to vacate each one of them at a time and wipe/reload w/ the latest firmware, etc. to see if maybe that might help.

Veeam settings are decompress data blocks before storing on the repository, and job settings to our DD's are classic Saturday full, weekday incremental; disable inline data dedupe, no compression, local target 16+TB backup files.