VEEAM v8 Restore from HP StoreOnce is slow

[Markus.K1985]

Hello,
Restore from StoreOnce is slow means a speed of 8 to 12 MBbs. Compared to restore from local disk with 140 MBbs.
When I read (with HP L&TT) Data from the StoreOnce Share which includes the VEEAM Backup-Data, the speed is 160 MBbs.

My Enviroment:
Components:
Backup Storage: StoreOnce 6500 (v3.11.4)
ESXi Version: 5.5.0 Build 1746018
-> Everything 10 GbE
Primary Storage: HP 3PAR 7400 (FC)
Design:
B&R Server: VMware VM - 2 vCPU’s / 4 GB RAM
B&R Proxy: VMware VM – 2 vCPU’s / 2 GB RAM
B&R DataMover: VMware VM – 4 vCPU’s / 8 GB RAM
Dedup is on the StoreOnce Share Enabled
Repository Setup:
-> Decompress data before Storing
Job Advanced Options:
Compression -> None
Storage optimizations -> LAN Target

Any suggestions, where the Problem can be located?

[Rick.Vanover]

The reads for a restore are "traversal" across the .VBK, meaning they read a bunch of different places. But if you were to copy the .VBK and put it on tape, it would be less such. You look like you have a good setup - are yu using Veeam v8? There are always improvements upwards.

[Markus.K1985]

OK, that’s what i thought... it’s not a sequential read during restore.

I already tried a Backup Copy from StoreOnce to local disk the Speed is 20 – 25 MBbs.

It’s a fresh VEEAM B&R v8 Installation.

[Gostev]

Any suggestions, where the Problem can be located?

The problem is located in StoreOnce itself: just like any other deduplicating storage it has poor random IOPS capacity.

Setting storage optimizations to "LAN Target (16+ TB backup files)" may help to improve performance, other than that there is little you can do, as it comes down to the storage speed.

[Markus.K1985]

I assume the Problem at StoreOnce to, at the moment… But I’m not sure.

Transfer Rate of 8 to 12 MBps in a Restore and d 20 to 25MBps in a BackupCopy is really poor.
I Read in a few other posts from StoreOnce Users about 50 MBps...

[Gostev]

It's important to realize that StoreOnce has many different models - from very low to very high end, and performance varies between them. I've certainly heard about even faster than 50 MB/s performance for high-end ones. I am not sure where 6500 stands in the whole StoreOnce line up though.

[tsightler]

Yep, I would expect much better performance from a 6500 than what you are seeing, as long as we're talking about full VM restore. The larger StoreOnce implementations I've worked with have typically been able to get between 70-100MBs for full VM restores. We've certainly seen some issues with CIFS/SMB performance although you appear to be running new enough firmware that this shouldn't be an issue. Still, if you have the possibility of standing up a Linux NFS repo it might be interesting to compare performance and see if that has anything to do with it.

[Markus.K1985]

I Tested a Linux DataMover with NFS Export from the StoreOnce.
It`s the Same result: 10 MBps during Restore

Here is my Linux DataMover Setup:
Debian 7.7 (VMware VM)
4 vCPUs
2 GB Memory
VMXNET3 Network Interface

The mount of my NFS Export:
172.20.7.2:/nas/VEEAM-TEST03 on /var/nfs type nfs (rw,relatime,sync,vers=3,rsize=1048576,wsize=1048576,namlen=255,hard,proto=tcp,timeo=600,retrans=2,sec=sys,mountaddr=172.20.7.2,mountvers=3,mountport=12204,mountproto=udp,local_lock=none,addr=172.20.7.2)

The Configuration of the Repository and the Job Setup is the same like the Other Tests. I Got the BestPractice from this Guide:
http://go.veeam.com/rs/veeam/images/Con ... napshot-lp

[Markus.K1985]

OK,
Update for the NFS Performance:
Now, it works great for Restores. It’s about 120 MBps.
CIFS is still bad… It is a HP StoreOnce Problem or a VEEAM Probem?

[tsightler]

So what changed from your report above where Linux/NFS was performing poorly as well? In the field I've consistently seen poor performance from CIFS with StoreOnce but I thought that the 3.11 firmware had fixed most of those issues for several customers. Based on that history I'd have to guess that the issue is on the StoreOnce side, but honestly I haven't seen these issues with the newer firmware so there could always be something environmental going on.

[chjones]

What did you do where you were able to achieve 120MBps restores?

We also use HP StoreOnce and see awesome performance during backups using forward incrementals (300-400MBps+) but terrible performance during a Full VM Restore (less than 4-5MBps). I even tested with a single VM in a backup job, disabled Veeam's in-line deduplication and compression (never have compression on for any storeonce jobs anyway) and even unchecked to exclude page file blocks. In this way, the restore should be a straight-forward read from the VBK file, there shouldn't been too much of Veeam having to do random reads. I understand the StoreOnce will perform random reads to reconstruct the VBK on the fly, but still should see some improvement. I saw no change whatsoever.

Our setup is:

StoreOnce 4430 (OS 3.11.7) with 72 x 2TB SATA Disks
Veeam Proxy is a HP ProLiant BL460c Gen8 with 2 x Eight Core Intel CPU, 128GB RAM, 8Gb FC HBA
ESXi Hosts are 7 x HP ProLiant BL460c Gen8 with 2 x Eight Core Intel CPU, 128GB RAM, 8Gb FC HBA in a DRS/HA Cluster
ESXi Storage is HP 3PAR 7400 Connected via 8Gb FC
All Servers and StoreOnce are connected via 10GbE via our Cisco Core Switches and HP FlexFabric 10Gb/24 Port Virtual Connect Modules for the blade servers

I can perform a straight windows file copy of the VBK off the StoreOnce CIFS Share onto a Windows Blade in the same enclosures and can read at 600-700MB/sec. When Veeam accessed the same VBK to restore the sole VM inside that VBK I get less than 5MB/sec restores.

I've tried using Direct-SAN restores from the proxy and via NBD and get the exact same results. I was hoping that Direct-SAN Restores with 8Gb FC to the datastores and 10GbE to the StoreOnce would at least give me something usable. I'm at a loss.

Would love to know what you did to get 120MBps restores working.

[emachabert]

I have many similar setups, from small to big. I've never seen a full VM restore running under 25MB/s or over 100MB/s, it depends of the storeonce model and generation. 5MB/s isn't what is expected, and the difference in speed between the windows copy and the veeam process is very big in your case. Did you open a support case ? Because there is something wrong here.

Dedupe appliances are good at sequential writes, ok at sequential read and horrible at random workloads.
With storeonce, be careful to run the latest firmware (>=3.11) because of a previous bug regarding SMB protocol (there is no CIFS in those appliances, this protocole is dead almost 10 years in the past and vendors are still using its name when speaking about SMB, it is annoying).

Let us know if support finds something.

[tsightler]

...(there is no CIFS in those appliances, this protocole is dead almost 10 years in the past and vendors are still using its name when speaking about SMB, it is annoying.)

While I get your point and I've been trying to change my own use as well, it's more difficult than it seems after years of being beaten over the head to call it CIFS back in the day, especially when vendors keep using it.

However, in the case of the StoreOnce it might actually be valid. Unless it's changed in the last 6 months (I doubt it), the Storeonce software is built on Redhat 5 and runs Likewise CIFS server to provide Windows compatible shares. While current versions of LikeWise CIFS server has some support for SMB 2.0, it was not enabled in the StoreOnce at least in the early 3.11.x firmware versions, rather they left with good old SMB 1.5 which can really be called CIFS and be accurate. Welcome back to 10-15 years ago.

During the troubleshooting last year on the Veeam+Storeonce performance issues with restores via SMB, one of the things I noted is that restores using a Windows 2003 or 2008 (non-R2) server as the gateway were 3-4x faster than restores when using Windows 2008R2 and newer as the gateway. At the time I suspected a bad interaction between the Likewise CIFS server and these newer Windows variants. HP made some changes in the 3.11.1 code that seemed to fix this, and it tested well both on the lab and in the field at some clients we were working with at the time, but I haven't done any additional testing with newer releases in the 3.11.x line. I assume it didn't break again along the way, but the 5MB/s does seem a lot like some of the performance we were seeing in those earlier releases where we were seeing problems.

One other thing to check, in some environments I've seen very bad reaction with LRO (large receive offload) on the gateway server. Other things to check is to make sure you've selected a gateway server that makes sense. If you use the default (automatic selection) one of the proxys will be used at random as the gateway server for backup each job. However, for restores, the Veeam server itself will be used as a gateway. If it's an all in one setup or if everything is local that's probably not an issue, but I've sometimes seen clients that used Veeam servers that were remote to the StoreOnce that weren't aware of this behavior so I thought it was worth mentioning.

[emachabert]

Hi Tom, i've been involved with hp labs (Bristol) in something similar regarding the two first generation of StoreOnce. I had a big issue with non english windows machine networks being able to crash the likewise CIFS daemon.
I confirm at the time of 1.x and 2.x, they were using the likewise daemon. But when Likewise was bought by EMC (it was included in the datadomain), they changed it to an HP based solution (that could be a fork of likewise ?). As far As I know, and I can be wrong but I got it from the dev team, they stopped using the oem daemon from likewise starting the 3.x branch of the firmware.

[tsightler]

I can say with 100% certainty that Likewise CIFS was still being used on 3.11.1, and I can't imagine them changing it on a minor revision. The specific version of the RPM was something like Likewise CIFS 5.6, and I couldn't find information on anything newer than 5.4 mentioned on the internet, so it's certainly possible it was an HP fork of the open source Likewise code, that seems totally reasonable, but it's still the base Likewise code.

[dellock6]

It still amazes me (and me and Tom discussed this for a while) why one should use Likewise instead of better known solutions like Samba...

[chjones]

We have been able to get our restore speeds from our HP StoreOnce 4430 devices from less than 5MB/sec up to on average 60-80MB/sec via SAN Restore. The problem is, I'm not quite sure how we did it. I apologise for the length of this post, but I don't know how else to explain this.

A couple of months ago, the SQL Server our Veeam Server uses was running on Windows 2012 R2 and SQL Server 2012 Standard SP2. We are working on deploying SharePoint 2013 using SQL 2014 at my company so we had been testing the application integration with Veeam and received warnings that for the SharePoint Explorer to work correctly the SQL Server hosting the Veeam Database should be SQL 2014 as well.

Rather than create a new SQL 2014 Server, and against my better judgement, I did an inplace SQL Upgrade which succeeded. Since then our restores have been terribly slow. Thinking it was related to SQL 2014 I created a new SQL 2012 Server using the "Generate Scripts" method to export the Veeam Database and recreate is on the SQL 2012 Server. Using the Veeam DB Config Tool we repointed the database and the Veeam Interface become much more responsive. However, restores speeds did not improve.

I then created a new SQL 2014 Server and moved the database to that using SQL backup and restore methods and again using the Veeam DB Config Tool. Restores then took off! Up to 60-80MB/sec.

I've since moved the Veeam Database back to the original SQL 2014 that was upgraded from SQL 2012 and the restores are still at the 60-80MB/sec range. I have no idea why this worked, perhaps something in the database schema itself got mucked up during the SQL upgrade and doing the migrations and backups and restores has resolved that?

We also, in further testing, found other settings that have big impacts on restore speeds. These, I assume, will seem pretty obvious to all the Veeam Staff and Gurus. Disabling Veeam's Inline dedupe resulted in faster restores as the Veeam Proxy doesn't have to rehydrate blocks that it retrieves from the StoreOnce. Also, changing the Storage Optimization setting in the backup job to "Local target (16TB+ backup files)" resulted in the fastest restores.

Using these two changes we were able to get 100-120MB/sec restores from the StoreOnce NAS Share.

There is a trade off with changing these settings. The StoreOnce will still perform it's dedupe and the resulting space used on the StoreOnce doesn't really increase, but we also write the VBK to tape each week after we do our weekly active full backups. When writing to tape the file is rehydrated so this results in much higher tape usage and longer time to write to tape.

For us, the writing to tape issues are a real concern as our entire backup and tape destage processes take a very long time and utilise many tapes already, so we'll live with the 60-80MB/sec restores for now.

[emachabert]

It is exactly what we do (and what we observed over time):
- if no destaging to tape => inline dedupe off
- if destaging to tape => inline dedupe on to fit into the tapes and reduce the write time

[Abigail Sugarman]

Interesting discussion, I work for IT Central Station, crowdsourced platform for IT reviews, and you may want to check out the comparisons between various backup solutions, including Storeonce: http://www.itcentralstation.com/product ... 1&cop=9623. Also might want to look at this reviewer who spoke specifically about his use of Veeam and Storeonce: http://www.itcentralstation.com/product ... 1&cop=9623

[Markus.K1985]

Sorry for my late update to this Problem but that wasn’t an easy Solution.
And only the combination did it:

- VEEAM v8 Update 2b (Issue in Update 2 with Dedup Appliances)
- StoreOnce Version 12.2.1 (NIC FW)
- HP Blade NIC 554FLB FW / Driver Update
Driver: elxnet 10.2.298.5 / Firmware Version: 10.2.340.19

[emachabert]

It is never too late for such an update. It can help others !

Thanks