Why is Veeam v8 so terrible with Dedupe Appliances?

[luckyinfil]

OK, that is simply not true. HP StoreOnce is being used by too many Veeam customers, big or small. Together with HP, we've been selling the joint solution of Veeam B&R + HP StoreOnce for probably over 2 years now, and of course it was certified back before we offered this as the joint solution. So I completely don't understand why those HP engineers you are talking to make it look like using StoreOnce with Veeam is something new.

Incidentally, I've just got a note from one very big and happy customer (well known UN organization), particularly stating "with two jobs running to both B6200 nodes I can achieve sustained 400MB/s and perhaps slightly better" (this is over NFS). That's pretty good for an inline dedupe storage target, if you ask me.

I'm not referring to the backup/sequential restore speeds. I'm talking about the FLRs with larger than your average VBK sizes.

[Gostev]

Unfortunately, being tape replacement, all dedupe appliance are specifically designed for sequential reads and writes... Perhaps you can try using 8MB block for backups (>16TB storage optimization settings), to help the storage to operate more in sequential manner. And this will certainly help to reduce the total number of I/Os significantly, potentially improving FLR performance.

[luckyinfil]

I'm actually testing the larger blocks right now. Perhaps it is something with the storeonce and larger VBK files that affect the FLR performance. I have no issues when I am dealing with smaller VBK files.

[tsightler]

And just to be clear, I'm one of the Veeam employees, along with members of our alliances team, that's working with those HP engineers (as well as having been interacting with the customer mentioned above). The only outstanding issues are those specific to CIFS and 2008R2 and newer and performance of backups and restores which I mentioned above. When using NFS, or CIFS on older Windows versions, both full VM backup and full VM restore performance is pretty decent, as well as tape offload, but as you say, these are sequential operations.

However, FLR is completely different as it requires a significant amount of random I/O. But this is not unique to StoreOnce, it's an issue with all dedupe appliances. We see the exact same behavior with other dedupe appliances with the exception of Exagrid because of their use of a landing zone were recent backups can be retrieved without the rehydration penalty. Note that even Exagrid has slower performance once you retrieve backups from older files that are no longer in the landing zone but only in the dedupe pool.

That being said, in many cases Veeam is still quite a bit faster than some of our competitors. In one case I had a client that was using a well known competing product with their Data Domain and it was taking them over 8 hours to restore a directory with a few hundred 5MB files. With Veeam this same restore took only about 15 minutes (from the very same Data Domain). Admittedly, restores from native disk took only 1-2 minutes, so the restore from the DD was still quite a bit slower, but that's still a huge improvement over 8 hours.

Using larger block sizes should help, but I wouldn't expect it to make restores from a dedupe appliance suddenly be the same speed as restores from native disk. One possible method to mitigate this is to enable file indexing and use EM to find and restore the files. It may still take some extra time for the restore to happen, and you may need to increase timeouts, but the actual process of finding and selecting files will be pretty fast, and the restore will probably still be faster than using the file browser.

One question, you mention having slowness on large backup files, but are you restoring from a large VM as well, or from a small VM that's part of a large VBK file? In my field experience the performance issue is related much more to the size of the actual VM that the size of the VBK itself because larger VMs have a tendency to have a larger number of files, and thus more fragmented MFTs that require retrieving more blocks to mount and browse, thus leading to very slow mount times.

[luckyinfil]

1 Large 3TB VM in it's own backup job. Perhaps I don't have a proper understanding of how dedupe works, but based on what I know, when a request is made for a block to be fetched from a certain address, dedupe just adds an extra hop as the data which should reside on the block is replaced by a pointer to where the deduped data is stored. To me, this extra hop results in minimum delays, unless of course Veeam has to write or make some changes to this block, which then results in the block having to rehydrate before it can be written to. Does Veeam do any write in place at the block level for FLRs?

ie:

In a non-dedupe: Request Block from Address A --> Go to address A --> Retrieve Block from Address A
In a dedupe: Request block from Address A --> Go to address A --> Redirected to Address B where deduped data block is --> Retrieve Block from Address B

[tsightler]

Does Veeam do any write in place at the block level for FLRs?

No, only reads.

In a non-dedupe: Request Block from Address A --> Go to address A --> Retrieve Block from Address A
In a dedupe: Request block from Address A --> Go to address A --> Redirected to Address B where deduped data block is --> Retrieve Block from Address B

This might be true if you're thinking of basic fixed block dedupe like Veeam does (and some online storage systems do), but it's not at all what a variable length dedupe applaince does. Let's assume you use Veeam with default settings, say a 512KB block (i.e. LAN optimization). Now when this hits the dedupe appliance the data in the block will be fingerprinted and analyzed for duplicate strings within the data. This single 512KB may very well be broken into tons of smaller chunks, varying in length from just a few bytes, to several KBs. So that single 512KB may easily be broken into many chunks when are all fairly small, we'll assume an average size of 8K just for the sake of simple math, but in the real world it could be more or less. So now that single 512KB block is broken into 64 chunks of dedupe data which is spread throughout the disk storage. Now, with writes, this isn't too bad because if the data is a duplicate I don't actually have to write it again, just update the reference count in the table and move on.

However, think about what happens when Veeam comes along and needs to request that single 512KB block of data. The appliance must then reference it's internal table to determine what subchunks make up this entire data block and then go and retrieve all of these little chunks from all across the drives in the system and reassemble that data. A single 512KB chunk, which might take say 5-10ms to retrieve from a simple SATA disk or RAID stripe, suddenly becomes 64 random length I/Os that must be serviced from a typically small pool of disk. This is why request latency has a tendency to be much higher with a dedupe device when requesting individual blocks. I/O latency at the disk layer may not seem particularly high, but it's quite common for an given 512K block to take 50-100ms or more to "rehydrate" as the appliance breaks the request for that block down into it many sub-components.

When performing sequential reads the dedupe applaince "cheats" by using read-ahead which helps turn the I/O into streaming requests, but for random reads this actually makes things worse since in many cases it's keeping the disks busy reading the next blocks which aren't even the blocks Veeam actually needs.

[luckyinfil]

That makes sense. I think the part I was missing in my logic was overlooking the variable length block dedupe. If your explanation is correct, then a fixed length dedupe appliance would result in significantly faster FLR times. This has been my experience with a deduped Windows 2012 repository. Have you tried any other fixed block length dedupe appliances?

[luckyinfil]

Something like a netapp filer which uses 4K fixed block deduplication?

[tsightler]

Veeam itself uses fixed block dedupe, and yes I've worked with several different fixed block dedupe solutions and they are better, but there's still significant amount of variance as you're still turning one request into many requests so the I/O count increases. Netapp dedupe seems to work OK on filers with decent resources, but if feels like every time I see it it's on a low end box that's been repurposed to store backups (so several years old and already low end), and performance is pretty sub-par in that case. Not so much restore performance, but just performance in general.

Other fixed block dedupe I've worked with is ZFS based dedupe, and OpenDedupe. Both worked reasonably well if you throw enough memory at them and they have fast enough I/O resources on the back side, but they require a LOT of memory when using such small blocks for dedupe.

[b.vanhaastrecht]

That makes sense. I think the part I was missing in my logic was overlooking the variable length block dedupe. If your explanation is correct, then a fixed length dedupe appliance would result in significantly faster FLR times. This has been my experience with a deduped Windows 2012 repository. Have you tried any other fixed block length dedupe appliances?

I think MS dedupe uses variable chunks sizes. Could be you were using FLR against non-deduped data.

http://tsmith.co/2013/veeam-and-server- ... plication/
http://technet.microsoft.com/nl-nl/libr ... 31602.aspx

[luckyinfil]

That's interesting. FLR is being used against deduped data for sure (getting 90%ish dedupe with Microsoft dedupe). Upon testing with larger block sizes, I see no difference at all. However, while checking the StoreOnceVSA performance settings, I see that there are "write" operations during a FLR. Why would there even be write operations when restores are a 100% read process?

[Gostev]

These write operations cannot be related to the FLR process. Could it be some internal maintenance process within StoreOnce VSA? If you believe these writes originate from Veeam components, you should be able to get more information on them by using the Process Monitor to see what specific process is performing them and what files and offsets it writes into. Thanks!

[lp@albersdruck.de]

Just tested the FLR with a 3TB file and the logs show that it took 32 minutes to mount the FLR drives:

Code: Select all

[02.06.2014 11:18:25][/b] <08> Info     WinFLR: Mounting restore point...
[02.06.2014 11:50:20][/b] <08> Info     WinFLR: Restore point was mounted

Has anyone done testing with decent sized files on dedupe appliances? When I try the same thing with a VBK file that's 40GB, all is fine and dandy but as it get's larger things have more issues. What's the largest VBK that veeam has tested their functionality with?

I did test this with our file server VM (3.8 TB in size), FLR did need around 1 minute to start.
Backup storage is a white box with ZFS dedupe (block size 128K, Linux as repository, decent amount of RAM and CPU, SATA-disks for storage).

[luckyinfil]

These write operations cannot be related to the FLR process. Could it be some internal maintenance process within StoreOnce VSA? If you believe these writes originate from Veeam components, you should be able to get more information on them by using the Process Monitor to see what specific process is performing them and what files and offsets it writes into. Thanks!

It's related to the Veeam FLR for sure. It only occurs as soon as I start the FLR. Perhaps it is something with the StoreOnce though.

[tsightler]

I did test this with our file server VM (3.8 TB in size), FLR did need around 1 minute to start.
Backup storage is a white box with ZFS dedupe (block size 128K, Linux as repository, decent amount of RAM and CPU, SATA-disks for storage).

Are you saying that your using a 128K record size for your ZFS dedupe, that's not really very comparable to what a dedupe appliance does. ZFS dedupe is fixed record size, but if you crank it down to 4K record size you'll get a lot closer to what a variable length dedupe appliance does, of course you'll also need significantly more memory on your ZFS server and performance will most likely fall quite a bit as well as the number of I/O required to service any specific request rise with the amount of data in the dedupe pool.

So good performance with your ZFS setup is predicatable since it's only slightly different that what Veeam dedupe already does, just slightly smaller than our smaller blocksize (and of course can dedupe across files, which should provide additional savings especially if your keeping mulitple full backups), but it's just not a very sane comparison to StoreOnce or other variable length dedupe engines, that can break data into chunks as small as a few KB.

[lightsout]

I've had the same issues with a Quantum DXi deduplicator as well.

Its a Linux based appliance, and runs Samba 3.6. I can see it was using SMB v1. Veeam performance to everything was great, apart from the dedup appliance. Like you, performance directly in Windows explorer to the dedup appliance was fine. As it happens, the Quantum also supports NFS. Using the Server 2012 NFS client wasn't much better than CIFS. However, mounting the Quantum via a Linux server and then adding that as a backup repository worked great. We were able to get full wire speed this way. I spent a lot of time with both Veeam & Quantum's support on this.

To be fair, not ideal to have this Linux server inline, but it did resolve it for us. So I'd suggest NFS as an option right now.

So to follow this up, I've actually fixed my CIFS performance with my Quantum DXi. My security settings in Windows required me to turn on "signed CIFS" support on the Quantum.

I got those security settings altered, turned it off on the Quantum, and my backup performance is now 5x faster.

[b.vanhaastrecht]

Thats interesting, we never got the DXi functioning without erros when using CIFS. Got loads of SMB related timeout errors on the DXi. We had a 4 months joined investigation by Veeam and Quantum DXi level 3 support. Eventually Quantum said their DXi product was not suited for the workload Veeam was putting on it. We even seen the same issue's with NFS. We eventually implemented VTL via FC, this went lightning fast and no more drops. Restore's take more time because we need to stage but backing up has no more problems.

I think the customer had a DXi 6700 with three disk cabinets.

[lightsout]

Sadly my 6520 doesn't support FC otherwise I'd have gone the VTL route as well. So I got plenty of time to spend with CIFS in the meantime. I would say, prior to this change, CIFS would be very unreliable but since changing it the results have been amazing (so far, touch wood!).

[adruet]

So to follow this up, I've actually fixed my CIFS performance with my Quantum DXi. My security settings in Windows required me to turn on "signed CIFS" support on the Quantum.

I got those security settings altered, turned it off on the Quantum, and my backup performance is now 5x faster.

Hello lightsout, sorry for getting back so late after your post, but I would be very interested if you could developp what you applied exactly to turn off the CIFS signing on the DXi ?

On the windows side I can imagine that you did something like this: https://support.exinda.com/topic/how-to ... erformance

But on the Quantum side I only see two things, the "Enable SMB Server signing" which is disabled by default, and the other would be disjoining it from the domain, is that what you did?

I am curious as I have two DXi 6702, I have already applied the above windows cifs signing recommendation to disable it, and the "Enable SMB Server signing" is not checked on my DXi. I know disjoining the domain would again improve performance, but I can't right now due to internal specific usage. I can only wish it could perform faster

I am going to configure the VTL part as some jobs are just not manageable any other way. Any recommendation on that part is welcome

[lightsout]

I'm no help to you I'm afraid. I had to have "Enable SMB Server signing" on, otherwise it wouldn't work. With it on, performance was just terrible. I was able to turn it off, and performance was much better. However during backup copy jobs, it still struggles during the transformations.

I think a VTL is the best way forward for you. Sadly my 6500 doesn't support that either so I'm forced to use CIFS. The other option is using NFS via a Linux proxy, which does give better performance than CIFS.

[adruet]

Have you try to disable the checkbox in the DXi for "Enable SMB Server signing" and do that on the windows side (on the veeam server which does the data transport): https://support.exinda.com/topic/how-to ... erformance .
Then reboot and it should work and allow you to disable smb server signing.

[lightsout]

Yes, I had to do that to get it to work, as my security policy enforced it on.

Update 2 has some changes which should help us http://www.veeam.com/kb2024 too.

[Abigail Sugarman]

Interesting to see a couple people in this thread comment on the potential issues stemming from the StoreOnce VSA. A couple recent reviewers to our site, IT Central Station, have commented on StoreOnce in this regard, one reviewer writing that he had "No issues with the systems size from VSA (Virtual appliance) to a four couplet system." Read the full review here: http://www.itcentralstation.com/product ... 2-t018-l01. Another reviewer specifically wrote about the ease of using Veeam with StoreOnce Dedupe - "Highly recommended the product, be sure to use the right backup application (Veeam or similar), so you can get the most out of it." Read the full review here: http://www.itcentralstation.com/product ... 8-t018-l02. Good luck!

[chris.childerhose]

We are using a DD2500 appliance for backups and while the backup process is much improved restores are still an issue.

I had a 1TB SQL server that I had to restore on our old DD670 array (before upgrade) and it took 16 hours to restore. I tested the same VM restore to a segregated network using the VMware VDP application and it cut the restore time down to 2 hours. I don't understand how this could be going to the same Appliance with DDBoost. Does it have to do with the number of threads that Veeam uses for restores versus backups? Even Symantec Backup Exec 2010 R3 restores faster than Veeam from this same appliance.

It doesn't make sense and would be something to see improved upon in a new version of Veeam. We use a physical server as the Veeam server with 5 proxy servers (1 per host) to do SAN based backups which is great but I cry when I need to restore anything.

[mongie]

Gostev: While I understand the thought behind using a "primary backup storage" in front of a dedupe appliance, with the Veeam Backup Copy job you still have to do a transform eventually if you use GFS... so it only helps partially. I think the real fix is to use DDBoost or Catalyst with the active full backup copy job in V9.

The problem for me is - I have dell dedupe appliances

Is global dedupe on the roadmap for the new "converged" storage feature announced recently in V9? (like AppAssure?)

[lowlander]

Is this topic still applicable with Veeam 9, or are there improvements ?

What we notices is a quite long waiting time for doing FLR with v9. For abour 300 GB we had to wait 20 minutes. Under the hood FLR volumes are mounted, however the GUI hangs. We filed a SR for this one : #01730264

thanks

[lightsout]

I'll add with my Quantum DXi 4700 with 10Gb and Veeam v9's improvements (specifically the backup copy jobs reading from source) I've got no issues at all now.

[Gostev]

Veeam Backup Copy job you still have to do a transform eventually if you use GFS

Not if you enable active fulls for Backup Copy, as lightsout has just mentioned.

Is this topic still applicable with Veeam 9, or are there improvements ?

Not really applicable, when it comes to dedupe appliances v9 should be "night and day" comparing to the previous versions. In fact, good idea to rename the topic to keep v9 issues separate.

[ChrisSnell]

Of course one dedupe backup appliance, ExaGrid, has a unique architecture which helped avoid these issues anyway!

http://www.exagrid.com/wp-content/uploa ... -Sheet.pdf

[lowlander]

running the Veeam Backup Console on the gateway server in front of a StoreOnce improves the FLR process !