Why is Veeam v8 so terrible with Dedupe Appliances?

[luckyinfil]

I've recently implemented a HP StoreOnce VSA appliance as a target for Veeam backups. Using windows explorer, I'm able to access the CIFS share and read/write at maximum network speeds (this is copying and writing VBK files onto the storeonce appliance).

While backup writes to the appliance also result in full speeds, backup restore speeds are abysmal. It literally takes way too long before the FLR explorer times out and doing any other restores result in horrible speeds.

Is this the way Veeam is designed to work with dedupe appliances? I'm able to copy to and from the CIFS share no issues and able to achieve full speeds. The jobs all have no compression and inline dedupe as per all the best practices whitepapers for HP StoreOnce and Veeam backups.

Anyone implemented dedupe appliances and get good restore speeds?

[dellock6]

John, all the operations you listed that are giving you good results are pure sequential operations.
On the other side, any restore operation requires random access to different blocks stored into the dedup appliance, that first have to find them, then rehydrate them, and finally serve them to Veeam; with all this said, performances cannot be the same as a sequential operation.

[luckyinfil]

Can you explain why a sequential read operation does not need to rehydrate data where as a random read operation requires data to be rehydrated? In a sequential read operation, the data still has to find them on the dedup appliance, rehydrate them, and then serve it to windows.

Also, can you explain why Veeam requires random reads? especially if you are restoring an entire VM or even just opening the file index for guest OS FLR?

Lastly, even if it was related to the disk not being able to support the IOPS for random read access as you suggest, why am I seeing same disk latencies for sequential as well as "random reads" (your words) which would indicate that it's not an issue with the disks.

[luckyinfil]

Also, I tried a Copy+Paste using the Veeam file explorer and the files do not copy at all (0KB/s) while a windows explorer copy and paste works fine. I'm assuming this test eliminates your random read hypothesis out completely?

[michaelryancook]

During our testing we spun up a StoreOnce VSA as a repository target. We did not have the issues you have faced with restores. I didn't test the FLR from the VSA but did some full VM recoveries without issue. I don't have a record of the restore times and size but they were very similar to restoring from our local disk. We had used both dedupe friendly and extreme compression in our tests. Both with inline dedupe. We found the best results with dedupe friendly as far as time and resulting backup size. Sorry I cannot offer any advice on improving performance as we implemented a pretty straightforward VSA install with a single CIFS share. I have since then removed the VSA due to some environment restrictions so I cannot retry these tests either.

[luckyinfil]

Thanks for the reply. I've noticed that it works when I'm testing the backup of a small VM (40GB). For this specific backup job that doesn't work well, it's a backup job of a 3TB VM.

[michaelryancook]

You're welcome. The largest VM I restored during my tests only had 60 GB of data.

[andersonts]

Here's the "official" configuration document related to this: http://go.veeam.com/rs/veeam/images/Con ... Backup.pdf

You'll notice it gives more specifics about the IO that's generated. Copying files via Windows Explorer is not really an apples to apples comparison and really doesn't mean anything. I don't think there's a need to get too deeply technical as there's several related posts already out there that talk through the difference in a dedupe appliance as a target versus other disk types. I normally tell customers that if they want to use all of the virtual lab, instant restore, fast RTO features they may want to consider staging the backups to a faster disk type then using the dedupe appliance as more of an archive device. This is easily accomplished with backup copy jobs.

BTW I like the title of the post...it's very eye catching. Here's my google search results page for "slow restore dedupe appliance"....as you can see from this the restore issues from these types of devices are globally an issue regardless of the backup software / process: https://www.google.com/search?q=slow+re ... +appliance

[Gostev]

The issue highlighted by OP is more of an architectural issue than anything else.

Backing up directly to deduplicating storage appliance is simply a wrong thing to do. Deduplicating storage is built to be a replacement for tape, so it is optimized for similar workloads (streaming, as opposed to random I/O) and is generally not a good choice for primary backup storage. Even though some people may get acceptable restore performance required to meet their RTO (especially with higher end units), we still highly recommend using different storage architecture for VM backups > What is the ultimate VM backup architecture?

This is basically the only architecture that allows you to meet all modern backup requirements, namely:
1. Maintain at least two copies of your backups (3-2-1 rule of backups)
2. Ensure fastest restores from latest restore points (low RTO)
3. Achieve smallest possible backup window (low RPO)
4. Provide long term retention to meet organizational policies and external requirements on data archival
5. Deliver all of the above at the lowest possible cost for the entire solution

And while traditionally, majority of users are mostly concerned about #2 and #3 (just like in this case), the most important requirement is still #1. Unfortunately, the first time most customers try to restore from the backup when they have already lost their production storage. And if the only backup you have appears to be corrupted due to some backup storage problem you were not aware of, this means you will not be able to recover your data at all. And I have seen this happen too many times...

Now, specifically to HP StoreOnce VSA appliance (and any other storage VSA for that matter), it's performance will solely depend on underlying hardware and especially storage, which is why different results will be observed in different deployments, as feedback from Michael demonstrates. Apple to apple comparison requires hardware units, or at least absolutely the same server and storage setups for ESXi running VSA VM.

[ckbrou]

I really like the idea of the 3-2-1 ultimate VM backup architecture consisting of the primary target being fairly fast disks and the secondary target being a source for backup copy jobs for longer term retention as Rick outlines in the link posted above (What is the ultimate VM backup architecture?). Rick specifically references a Dedupe applicance being a good option for this secondary target. But is anyone actually doing this successfully? Here is a link to another post discussing Dedupe appliance terrible performance with backup copy jobs specifically (http://forums.veeam.com/veeam-backup-re ... 17785.html).

[b.vanhaastrecht]

I have a DataDomain DD640, and my experiance with copy jobs is terrible to. The transform at the end of the copy job is the problem, it's to much read&write IO for the DataDomain to handle in an acceptable timeframe. We have two site's, and on the secondary site the DataDomain resides. We created a uncompressed repository on the secondary site, and then with scripts at the end of those copy jobs we copy the data to the DataDomain. This way the DD gets a nice flow of data and good performance. Disadvantage: scripts, sensative for errors, bad monitoring, not GUI intergrated.

I've tested with Veaam 8 beta and DD-Boost plugin. And I've messured 50% improvement on the transforms!

[lightsout]

I've had the same issues with a Quantum DXi deduplicator as well.

Its a Linux based appliance, and runs Samba 3.6. I can see it was using SMB v1. Veeam performance to everything was great, apart from the dedup appliance. Like you, performance directly in Windows explorer to the dedup appliance was fine. As it happens, the Quantum also supports NFS. Using the Server 2012 NFS client wasn't much better than CIFS. However, mounting the Quantum via a Linux server and then adding that as a backup repository worked great. We were able to get full wire speed this way. I spent a lot of time with both Veeam & Quantum's support on this.

To be fair, not ideal to have this Linux server inline, but it did resolve it for us. So I'd suggest NFS as an option right now.

[luckyinfil]

The issue highlighted by OP is more of an architectural issue than anything else.

Backing up directly to deduplicating storage appliance is simply a wrong thing to do. Deduplicating storage is built to be a replacement for tape, so it is optimized for similar workloads (streaming, as opposed to random I/O) and is generally not a good choice for primary backup storage. Even though some people may get acceptable restore performance required to meet their RTO (especially with higher end units), we still highly recommend using different storage architecture for VM backups > What is the ultimate VM backup architecture?

This is basically the only architecture that allows you to meet all modern backup requirements, namely:
1. Maintain at least two copies of your backups (3-2-1 rule of backups)
2. Ensure fastest restores from latest restore points (low RTO)
3. Achieve smallest possible backup window (low RPO)
4. Provide long term retention to meet organizational policies and external requirements on data archival
5. Deliver all of the above at the lowest possible cost for the entire solution

And while traditionally, majority of users are mostly concerned about #2 and #3 (just like in this case), the most important requirement is still #1. Unfortunately, the first time most customers try to restore from the backup when they have already lost their production storage. And if the only backup you have appears to be corrupted due to some backup storage problem you were not aware of, this means you will not be able to recover your data at all. And I have seen this happen too many times...

Now, specifically to HP StoreOnce VSA appliance (and any other storage VSA for that matter), it's performance will solely depend on underlying hardware and especially storage, which is why different results will be observed in different deployments, as feedback from Michael demonstrates. Apple to apple comparison requires hardware units, or at least absolutely the same server and storage setups for ESXi running VSA VM.

Hi Gostev,

Can you explain why a simple file restore will have HIGH IO? And by "High IO" are you referring to random reads or random writes or both. Can you explain the technical details on how a FLR works?

[luckyinfil]

Also, when doing a file copy operation using the veeam explorer between the dedupe appliance and a SAN disk/local drive, I get significantly worse performance compared to a direct copy over windows explorer. Care to explain how that works as well?

[tsightler]

Also, when doing a file copy operation using the veeam explorer between the dedupe appliance and a SAN disk/local drive, I get significantly worse performance compared to a direct copy over windows explorer. Care to explain how that works as well?

Can you explain what you mean here? Are you comparing simply copying files using Windows Explorer, to extracting files from VM backups within Veeam backups files? Or are you saying that simply using Windows Explorer with Veeam to perform FLR (by copying files directly from the C:\VeeamFLR mount point) is faster than using Veeam Backup Browser window?

If you are comparing simply copying files to/from the appliance with Explorer vs Veeam extracting files from VM images, that's a drastically different process and I will be glad to explain that in detail, but I just want to make sure I'm answering the correct comparison before I do that.

[luckyinfil]

It's a file copy vs file copy, not extracting backup files. I'm talking about using the Veeam File explorer (Under the "Files" section between "Virtual Machines" and "Backup Infrastructure". Copying a VBK file from the dedupe appliance (through a mapped drive) to a local disk (SAN Storage) on the same server results in much slower speeds than using Windows Explorer to do a direct copy. Now if this isn't an apples to apples comparison, then I don't know what is. Why would copying a VBK through the Veeam Explorer be slower than Windows Explorer? There must be something different that Veeam is doing even for a simple file copy.

[Gostev]

This File Copy functionality comes from FastSCP times 6 years ago. Unlike Windows Explorer, this was built and optimized for copying files between backup server and ESX(i) host datastores. As such, it uses special I/O patterns that provide for faster performance with this particular use case (which is exactly what made FastSCP a few times faster than existing tools, and huge success). Perhaps deduplicating storage simply does not "like" these I/O patterns. We did not have a goal to build another Windows Explorer that is good with copying files from CIFS shares locally, because it already exists

[luckyinfil]

I'll do a test with FastSCP. Can you explain the technical details (of the read and writes) when a

1) FLR is initiated - How does it generate the file structures
2) FLR is executed - How does it restore single files

[Gostev]

File Copy functionality = Former FastSCP, so you are already testing it.

1. FLR process reads backup file blocks holding all MFT areas in the volume inside of the virtual disk.

2. FLR process reads backup file blocks holding areas occupied by the restored file in the volume inside of the virtual disk. Fragmented files may require a lot of random I/O, and significant overhead in terms of extra read I/O. Since backup file block size is whatever 1MB of source data compresses to, FLR process needs to read the entire block even when trying to get some small 4KB fragment of the restored file sitting inside of the given backup file block.

In case restoration is performed from an incremental restore point, this also requires accessing multiple backup files in the process, of course.

[tsightler]

While it's true in general that dedupe appliance cause slower restores, the original post in this thread referred to FLR timing out and other issues that seemed beyond the "usual" slowdown. As mentioned by Gostev in the weekly forum newsletter, this weekend I spent a lot of time testing with Veeam V7 and the StoreOnce VSA as well as a small StoreOnce physical appliance (I believe it's a 2620, a very entry level device). What I discovered was that restore performance was indeed quite abysmal with using CIFS to access the StoreOnce, both on the VSA and the physical device. Backups would run around 40-50MB/s, but full VM restores were roughly 1/10th of that speed.

I decided to install a Linux server to access the physical StoreOnce via NFS and interestingly the StoreOnce restores suddenly went to 45MB/s, an increase of approximately 10x. Unfortunately, the VSA does not come with NFS support so this wasn't an option there, however, out of curiosity I decided to try mounting the VSA using the native Linux CIFS client and surprisingly this also resulted in greatly improved performance of around 40MB/s for restores.

This really didn't make any sense to me, I was performing restores from the very same backups files, stored on the very same CIFS share on the StoreOnce, the only difference being that I was using the native Linux CIFS client instead of Windows 2012 CIFS. This got me thinking, what if there was some issue with the interaction of Veeam, Windows 2012 CIFS, and the StoreOnce VSA and I decided to try some older versions. My first test was with Windows 2008R2, and, interestingly enough, it was about 2x faster than the Windows 2012 box. This still wasn't very fast, but certainly got me thinking I was on to something so I moved on to testing with an old 32-bit Windows 2003 VM as the CIFS gateway. Amazingly, this setup provided performance that was virtually identical to the setup using Linux server via NFS, ~45MB/s (this lab is somewhat resource constrained, with only 1GbE thus restores from "normal" disk only run at around 55MB/s so 45MB/s is quite respectable, especially considering the entry level StoreOnce hardware).

I continued testing with Windows 2008 (not the 2008R2 version), both 32-bit and 64-bit variants, and performance was very similar to the Windows 2003 test. I also repeated these test in my own home lab using the StoreOnce VSA and, while performance was somewhat lower with the VSA compared to the physical appliance, the pattern of results was the same. Windows 2012/2012R2 was very slow (3-4MB/s), Windows 2008R2 was slightly faster (15-20MB/s) and Windows 2003/2008 was the fastest (30-35MB/s).

Also interesting is that when running 2008R2, even though the restore is much slower that with 2003/2008, the CPU usage on the StoreOnce VSA is 2x higher, and when using 2012/2012R2 the restores are even slower, while the CPU increases even more. There certainly seems to be some significant issues with the interaction of these OS versions, Veeam, and the current versions of the StoreOnce firmware. While the test results above are all talking about a full VM restore, this performance also significantly impacted FLR restores from the devices as well.

So for now the recommendation is to either use a Linux server to connect to the StoreOnce via NFS (this seems to be the highest performing option), or to set a Windows 2003/2008 server and configure it as the CIFS proxy for all Veeam I/O to the StoreOnce. For larger environments it's possible to have multiple servers for CIFS proxies, although only one per repository. We'll certainly be sharing these results with HP and continue to work with them on more testing and research into the issues, but for now these workarounds should provide acceptable performance.

[HPersson]

Have you done any testing against different generations of the StoreOnce software (V2.x and V3.x)?

[b.vanhaastrecht]

Very interesting. Could you perhaps perform your tests when you tell Windows 2012 /2012 R2 to use SMB v1 instead of V2/V3:

Code: Select all

sc.exe config lanmanworkstation depend= bowser/mrxsmb10/nsi 
sc.exe config mrxsmb20 start= disabled

http://support.microsoft.com/kb/2696547

[tsightler]

The StoreOnce device doesn't actually support SMB v2, but indeed during my testing, I did try disabling SMB v2 on the Windows client anyway, just to be thorough. Actually, I tried about a million tweaks (disabling signing, plus tons of TCP tweaks on the Windows and Linux/StoreOnce side), but haven't been able to change the performance more than marginally. When I again have some "spare" time I plan to run some packet traces as well and see if that gives any clues.

Also, testing was only done with fairly current 3.x firmware, specifically 3.7.3 for the VSA, and I believe 3.9.0 for the 2610 (I had previously inaccurately reported that this was a 2620). I had many years ago performed testing with older StoreOnce equipment using older 2.x firmware and had never noticed any such issues, but in the field we mostly see 3.x firmware and my goal was to attempt to identify any possible workarounds for customers struggling with this, and to hopefully uncover some paths that we can pursue further with HP. I think I've accomplished that for now, even though the workarounds might be less than ideal.

[luckyinfil]

How large are the backups you are restoring from? I tried to use a 2003 server as the front end for the CIFS share and I can not even open the FLR explorer window within 40 minutes. This is for a 3TB VBK file which consists of 1 VM in the backup.

Using the VSA appliance to test right now as well.

[luckyinfil]

Just tested the FLR with a 3TB file and the logs show that it took 32 minutes to mount the FLR drives:

Code: Select all

[02.06.2014 11:18:25][/b] <08> Info     WinFLR: Mounting restore point...
[02.06.2014 11:50:20][/b] <08> Info     WinFLR: Restore point was mounted

Has anyone done testing with decent sized files on dedupe appliances? When I try the same thing with a VBK file that's 40GB, all is fine and dandy but as it get's larger things have more issues. What's the largest VBK that veeam has tested their functionality with?

[luckyinfil]

Also, just to add, I can confirm that using a 2003 front end server for CIFS share appears to resolve the slowness associated with non FLR restores and write to tapes. What's seems to still have issues is how FLR interacts with dedupe appliances when dealing with large files. As I'm currently testing with the VSA, I can easily tell that the storage behind the VSA is not being stressed as the latency times are ~7-8ms.

[tsightler]

Hi John. The testing I've been doing has been relatively limited in size, with a VBK of ~300GB being the largest size tested are part of my own work. Note that this isn't "official" Veeam testing per-se, although I'm a Veeam employee. This is largely me testing and trying understand issues customers are seeing in the field with this configuration, and hopefully find workarounds or help escalate to R&D teams findings that might help get to the bottom of this specific issue.

Regarding the maximum size tested, I certainly have other customers that have large VBK files on StoreOnce and other dedupe appliances and indeed restores from those volumes can be very slow. This is not really unique to the StoreOnce but rather due to the way the FLR process works and how it accesses data. This wasn't really part of the scope of my testing, and while I did see a significant improvement in FLR, especially for the restore of large files, I didn't expect this would significantly reduce the time required for random operations.

When Veeam performs an FLR we basically simply mount the disk image to a Windows machine. This mounting process requires random reads from the MFT and the latency of the requests is a major factor. For large volumes with many files and large, fragmented MFTs, this can be a very slow process. Unlike the sequential process of a full VM restore or tape copy, the mounting of a disk cannot requests many, many blocks at a time, rather, we must request a block, parse it contents, and then request the next block based on it's link. While you are seeing only moderate disk latency on the StoreOnce appliance, it would perhaps be more interesting to see what the CIFS request latency from Veeams perspective is. My guess is it will be in the >50ms range for each block, perhaps even higher.

Also, are you using the default settings for the job, i.e. LAN block size, Veeam dedupe enabled, etc.? You should be able to significantly enhance the performance of FLR by disabling Veeam dedupe and increasing the blocks size to "Local Target (>16TB)". This will increase the amount of data being sent to the StoreOnce during backups, although the StoreOnce should just dedupe the data to the back to the same size, but I'd suspect it to have a significant benefit on FLR restores as, instead of having to make 16 requests to get 16 512KB blocks, we'd make 1 request to get 1 8MB block instead. Yes, this will lead to some additional data being read, but overall I'd suspect the performance would be much higher as the StoreOnce can generally retrieve a large block of data in roughly the same time that it retrieves a small block.

Would you be willing to share a few additional details regarding your VSA setup, i.e. how much memory have you allocated to it, and what's the total amount of disk and data assigned?

[luckyinfil]

Hi Tom,

I currently have dedupe enabled, compression set to none, and LAN target as recommended in the HP StoreOnce/Veeam configuration guide I found. As there is only 1 VM within the Job, I don't think the deduplication will make any difference. As for compression, would increasing the compression level be better for FLRs? The thinking behind this is that it will have fewer blocks to retrieve albeit at the cost of processing power to uncompress and less deduplication.

I'll give local target (16+ TB backup files a try).

The VSA is configured to use 32 GB of RAM, 10 TB of space (10x 1TB LUNs backed by a v3700 SAN), and 32GB RAM. It currently has about 30TB of backup which uses about 4TB of space (9 daily full VBK backups, no incrementals).

[luckyinfil]

We're actually speaking to some HP engineers and apparently there's some issues with the StoreOnce with Veeam. Don't know the exact details yet but when we asked HP to certify that the appliances work with Veeam, we were told that Veeam wouldn't certify it until some issues were resolved. I'll try to update if I find out more but anyone at Veeam know what might be the issue? Is it the CIFS issue?

[Gostev]

OK, that is simply not true. HP StoreOnce is being used by too many Veeam customers, big or small. Together with HP, we've been selling the joint solution of Veeam B&R + HP StoreOnce for probably over 2 years now, and of course it was certified back before we offered this as the joint solution. So I completely don't understand why those HP engineers you are talking to make it look like using StoreOnce with Veeam is something new.

Incidentally, I've just got a note from one very big and happy customer (well known UN organization), particularly stating "with two jobs running to both B6200 nodes I can achieve sustained 400MB/s and perhaps slightly better" (this is over NFS). That's pretty good for an inline dedupe storage target, if you ask me.