Speed of backup/restore jobs

[briggs]

Morning all.
I've posted a few things on here, because we are literally in our second week of using Veeam...
I've got another thread on here about restore performance, because to me, it seems VERY slow to restore an entire VM. At the speeds I got testing it yesterday, it would take DAYS to get all our VM's restored and back online, not exactly progress from our previous tape backups scenario.

I want to know how to go about getting some accurate benchmark figures for the speed of backups and restores? At the moment, it seems very slow, but how do I know what is fast if I can't achieve it myself?

Our setup is VMWare essentials plus, 2 x DL380 hosts (2x2Ghz CPU, 12 cores total), 1 x HP P2000 SAN (2 x 10 disk RAID5 arrays), across HP 2910 switches. We have 20 VM's running, 1 of which is the Veeam VM. Veeam is on Windows 7, assigned 4 CPU, 4GB RAM.

We are backing up to a QNAP TS459 ProII NAS box, which is 4 x 2TB disks RAID5, 1Gbps network, using a normal share to access it.

Backups show the bottleneck as "Target" which I guess means the QNAP. The processing rate midway through the job is never more than 19MB/s. My restore yesterday, showed 12MB/s, which as it's coming from the QNAP, I guess won't ever reach more than 19MB/s. I can't understand why the QNAP is so slow, it's a decent spec NAS box.

What is a good throughput, when you are backing up to a networked storage device like the QNAP (ie <£500 NAS box)

Many thanks in advance for any advice that may follow.

Alan

[chrisdearden]

the problem is with 4 Sata discs in RAID 5 you are never going to be able to drive high Write IO's - you'll hit the limits of the spinning disc. Running the QNAP in RAID 10 should help a little as you wont get the RAID5 write penalties , though you would still be limited by the max IOPS of a pair of Sata discs.

If the QNAP supports iSCSI you may find it a little quicker to mount a backup volume via iSCSI rather than a share , its more efficient from a networking point of view.


My Home Nas is a readynas NV+ running 4 discs over Gigabit network , just ran a quick test now and on a full backup ( which is the only real indicator of speed you can compare ) , its running at about 14-19 MB/second. Target showing as the bottleneck.

[briggs]

Thanks for that, so I should expect no better speeds as long as I'm offloading to the QNAP? RAID10 would cut my storage down too much to maintain our retention policy. I guess I could try RAID0, though obviously I'm risking data a bit more than I'd like.
I can obviously experiment with this, but I don't really have the luxury of time.
Thanks, Alan

[foggy]

Alan, probably you'll find this thread to be helpful.

[dellock6]

Alan, another useful information would be to know which kind of backup mode you are using, I do hope you are going for the forward incremental backup, since on this type of device reverse incremental is too heavy for the capabilities they have. Remember, Qnap, even those based on Atom CPU rather than the Marvell ones, still rely on software RAID and have a limited amount of RAM. This also, it's used for all the operating system activities, so 1 GB of ram is not the same as having a dedicated memory for the raid controller.
This does not mean the Qnap are useless, it's the opposite, only you cannot ask them to perform out of their capabilities. Try first to convert the raid to 10, this will help for sure. !DO NOT! use Raid0, the Qnap is the saviour of your precious backups

Luca.

[briggs]

Many thanks guys, I'll look into this some more and let you know the outcome...

[briggs]

The QNAP now has RAID10 and 3GB of RAM. My budget has suffered, but so has my patience and sanity! After going to all that effort, the performance is still pretty dire. I've still got to do some more testing, to see if the bottleneck is changing, but I'm getting frustrated with all this now, Arcserve D2D only backed up Windows hosts, but that ran brilliantly, with smaller incrementals and little over half an hour max to run each incremental - Veeam doing the same servers, at the same time, is really not a match for it at the moment!

[Dima P.]

Hello Alan,

I am sorry to hear that you faced problems with performance on the target. According to users findings setting up RAID10 instead of RAID5 should double the performance on the target - could you please update this thread, with your test results so we could help you further!

Thank you.

[briggs]

OK, so the TS459Pro2 came with 4x2TB disks, RAID5.
I've just bought 4x4TB disks (!!!) and set it up as RAID10, hoping to cancel out this speed issue a little bit. To be honest, even doubling it is going to give me issues if I ever need to do a restore on a whole VM, some of my VM's are 800GB+
The 2 full backups I've tried since doing this, have run at 18MB/s, with the bottleneck showing as target, and 26MB/s with the bottleneck showing as source.
I've got nothing to go on with those 2 tests!!!

[Dima P.]

Alan,
Thank you for the update.

Could you please clarify did you set up additional proxies or you run Veeam B&R VM as a proxy itself? Also I am interested on how jobs you mentioned were set up, what level of compression you used?

I believe I saw the resource monitor under the Bandwidth Usage in the QNAP CP, could you please check the performance (send/receive) during the manual file upload to NAS share?

Thank you.

[briggs]

OK, Veeam B&R is on a Win7 64bit VM, on the cluster itself. This is a single server deployment of Veeam.
The jobs are optimised for LAN, with the standard level of compression. They are all incremental jobs now, I did initially see less than 10MB/s using reverse incremental, but a previous reply highlighted the fact that they are half the speed anyway.

The QNAP has a single 1Gbps NIC, the switch it's on is connected back to another 1Gbps switch, which is then connected back upto the main switches that the VM cluster sits on. No VLAN's setup for the QNAP.

We have 2 QNAP's and doing an RSYNC from one to the other (the other is the same model, but original RAID5 config, 4x2TB disks) I see the resource monitor reading a peak of 28.9MB/s. Though this RSYNC job I'm running is on a folder full of lots and lots of small files (Novell GroupWise archive databases).
I will try the same test in a moment, with a large single file from my desktop instead...

Alan

[briggs]

A single 3GB ISO image from my desktop results in the resource monitor of the QNAP showing 37.6MB/s

[dellock6]

Alan, just my 2 cents: Veeam Backup also uses deduplication and compression on the backup files it creates, and that reduces for sure the native speed of any storage it uses, since it has to compare data for the deduplication process, and it makes the process not completely sequential.
Said that, if the highest speed you can reach with a pure uncompressed compy is 37 MB/s, 26 Mb/s you saw in your backup does not sounds so unexpected. I'm not saying they are good, but if you read around the forums, those kind of NAS have speeds that ranges in these values...

Luca.

[veremin]

Yep, Luca is absolutely right here. Taking into account on-fly deduplication and compression, the difference in speeds (37 versus 6) seems to be more or less reasonable.

Thanks.

[Dima P.]

Alan,
Thank you for the provided info, it helps a lot.
As Luca mentioned Veeam B&R does compression/deduplication and here performance loss may occur.

You also said that you are using RSYNC between 2 QNAPS to replicate data; I am thinking that running RSYNC could significantly reduce the performance of the QNAP NAS as it requires huge resources to run.

Is it possible to shut the RSYNC down for a testing period and perform the same tests:
- Manual file copy to QNAP
- Veeam B&R same job re run
- Suggest also setting up additional job with no deduplication/compression enabled

According to this results with QNAP resources not being overloaded, I guess we would get the maximum possible performance.

[briggs]

Firstly, thanks for the prompt replies, this is an exceptional forum/community!
d.popov - The RSYNC was running for test purposes, as you say it consumes high CPU and has a fairly inconsistent throughput depending on the size/type/quantity of data you're sycing. This was running only while I was trying to get a "baseline" write speed figure. It was not, and is not running when Veeam does a backup job.
I will do a test job now, zero compression and turn off the option for dedup as well, to see what it achieves... Should I still set the optimisation for LAN in the job options?
Alan

[briggs]

Just a note to re-itterate something, this QNAP was slow as a RAID5, so I invested in faster disks to setup a RAID10 array instead, and saw a relatively minor improvement. I was expecting, certainly based on comments in the thread linked in an earlier reply (http://forums.veeam.com/viewtopic.php?f ... t=slow+nas), a huge improvement

[briggs]

OK, a Win2003 server, 31GB used, no compression or dedup, Veeam says bottleneck is target and shows 34MB/s. The QNAP resource monitor for the duration shows a spikey network resource graph, maxing out at 74.98, lows of less than 20MB/s. I can post a pic of the graph if necessary?

[briggs]

File copy to the QNAP from my Windows 7 desktop, to the network share, is terrible - 11MB/s. 1Gbps NIC in my machine, small desktop switch uplinked into a switch, which links eventually to a switch where the QNAP sits. Is it our network, why am I getting such vague results! Grrrr...

[Dima P.]

Thank you for the updates - highly appreciated!

Eventually we are getting close to the issue resolution, but there is one option you could also check on QNAP write back cache (or just write cache) under QNAP hardware settings. Is it available on your QNAP?

Cheers!

[briggs]

There is a setting "Enable write cache (EXT4 delay allocation) and that is ticked...

[Dima P.]

Please try to set write back cache off, and monitor the performance. Normally this option provides better I/O however I faced couple of mentioning’s that under QNAP write back cache works unstable.

[briggs]

I haven't changed the write back cache setting yet, and the QNAP forums seems to suggest you'll get far worse performance anyway, the issues with stability have gone in the recent firmware aparantly.
But, my restore speed, it's still pretty unworkable. Don't forget I've gone from RAID5, and 1GB RAM on the QNAP, to newer SATA disks in RAID10, 3GB RAM, and the restore speed WAS 35MB/s on a 60GB Win7 VM... It is now, wait for it, 36MB/s.
This is restoring on the same network switch, to a DL380 G7 server, with 8 x 6G SAS disks in it as RAID5. It can't possibly be down to the disks on the server, surely? enterprise SAS disks with dedicated raid controller?

[Dima P.]

Hello Alan,

I assume that the problem is only with the QNAP performance as when you did a manual copy resource monitor showed you 37,6 MB/s - and the backup with no compression/low deduplication got you close to these numbers. From this point, there could be several places to investigate:

- QNAP settings, firmware updates, limits under settings, other possible documented options that may affect or limit the I/O performance. Contact with QNAP support team if it possible - to get device set up best practice.
- Possibly the network/routing issues inside you LAN that affect the performance.

[briggs]

OK, an update of sorts...
I can now achieve 100MB/s copy speeds to and from the QNAP, from my desktop PC. The switch that the QNAP was plugged into has been swapped out. I'm not sure what the problem was though, same model switch with the same config (smart switch so no real detail to the config anyway) has replaced it, and the speeds are way better now.

The Veeam server though (a VM) cannot get better than 70MB/s when doing a simle file copy from the QNAP. I am guessing it's a bandwidth problem with the VM cluster holding it back slightly, but the restore speed remains very poor, showing 34MB/s at best. I've implemented another proxy server on another VMWare host (so different datastore too) and it was even slower.

If the QNAP can be read and written to from the Veeam server (VM) at 70MB/s, should I expect a restore to be close to that sort of speed? I know the restore job is doing more than just reading the backup files, but surely it can't be useable to the majority of people, if it's that slow? It takes 8 hours or more (estimates) to restore our main file server, there are 4 more critical servers that would need restoring in the event of a disaster, and with those figures, Veeam would get us back up and running no quicker, possibly slower, than our old LTO tapes?

Any advice? Many thanks, ALan

[dellock6]

you are forgetting Veeam has Instant VM recovery, and is design right to workaround slow restore time due to huge size of VMs or slow restore speeds. With it you can power on any VM in minutes, regardless its size, and than move it back to the production storage while it's already running.
About the restore times, given the way data are written into the VBK file (deduped, compressed, and blocks are also on different files if you are not requesting the last full version of the VM) yes the restore speed can vary and could be slower than the backup speed.

Luca.

[briggs]

Luca, I am not forgetting instant VM recovery at all. Whilst it is an amazing bit of technology, it's unworkable in our trials. Our restore speed is indicative of the bandwidth we have available to the QNAP, so when we do an instant VM recovery, it runs SOOOO SLLLOOOOOOWWWW. I am not exagerating, it's un-useable. I don't know if I am missing something here, but with a backup target of a 4 disk NAS box, even with our new RAID10 array, it's performance is terrible when trying instant VM recovery.
I'd welcome your comments on this though, should I be doing anything to get better instant VM recovery performance from it?
Recovery speed is therefore going to be very important to us in the event of loosing our cluster, or SAN, or similar serious outage. I am not sure what people are getting when restoring from a QNAP or similar NAS box? But 34MB/s is really not too good!

Many thanks, Alan

[dellock6]

Well, instant VM recovery is even worst than usual restore activity when it comes to speed. It basically runs a VM from a deduplicated and compressed backup file, so it's a complete random access to disk, and since it runs a production VM, the amount of I/O it can creates on the backup file can be high.
for sure, a configuration to be used is to use a VMFS datastore for the caching of Instant Recovery, so at least you avoid writes operation to hit the QNAP. Apart from this, I have small experience with that Qnap models and IR, never tried too much. I had a customer using it once, and I do remember restore operations where indeed slow...
Honestly, I've always used bigger qnap models or even better proper x86 servers as my primary backup repository, especially if IR is going to be considered.

Luca.

[softcomet]

Hello
I am new to this forum and to VEEAM. But I have the same performance issues that all you are having.

I believe, though, that there's an important point that hasn't been discussed and may be the culprit in most of the situations. I need to have your opinion (and ideally also VEEAM tech people's)

I have a number of VMWAre installations, most of them with VMWARE vSphere Essentials Plus suite. I also use various storage devices, from mid-hi performance ones (IBM DS5200, DELL MD3000i and 3200i) and some lower end ones (but still good): QNAP and Synology ones.

Now the common denominator of all these units (where I have my VMFS storage) is that they perform more than decently, and I/O operations speed is more than decent except - of course - for backups.

Talking about my vSPhere Essentials Plus installations, they range from version 4.0 (ESX) to 4.1 (ESXi) to 5/5.1(ESXi).
I also use various imaging software (in my endless search for performance).

This is what I discovered.
1) I have one installation of vSphere Essentials Plus 4.0. It is based on ESX (not ESXi) hosts. Backups and restores are FAST. The storage behind it is a DELL MD300i ISCSI, SAS 15K Rpm disks.
2) I had a second installation of this kind (different customer) and was fast too. Then I decided to upgrade to vSphere 4.1 (and passed from ESX to ESXi) and the performance dropped totally and was a shame. I opened various tickets with the imaging software vendor and no joy. I upgraded to version 5 and then 5.1 an the performance bottleneck remains.
3) I now have installed at a new customer site a new vSphere Essentials Plus environment, based on DELL MD3200i ISCSI storage and VEEAM Backup and Repl software (latest 6.5.0.109 version).
Now pay attention to this:
When we first installed this last virtual environment, we didn't immediately activate VMWARE Essentials Plus licenses, therefore we were working in trial mode (and therefore VMWARE features were unrestricted). We installed VEEAM (on the vCenter virtual machine) and backups were FAST.
The a few days later we applied the Essentials plus licenses, and suddenly the backups become extremely slow (70% performance drop).

This is due, I believe, to the fact that the Essentials Plus suite does not have some of the feature needed for fast backups, I believe the needed one is the Hot Add capability (to run a LAN free backup).

And this is a big problem. This limitation, of course, didn't exist on vSphere Essentials Plus 4.0 (and therefore I am NOT upgrading that customer until when I don't find a decent solution).

Se below the performance differences I had last week, before and after having applied the VMWare vSphere 5.1 Essentials Plus licenses

Backup of VM while vmware was in trial mode therefore with unrestricted features

Name Status Start time End time Size Read Transferred Duration
SVROMSPA Success 00:07:44 04:54:36 750,1 GB 534,7 GB 370,8 GB 4:46:51


Backup of VM after having applied the VMWare Essentials Plus licenses

Name Status Start time End time Size Read Transferred Duration
SERVER Success 22:01:03 08:55:03 (+1) 750,1 GB 541,4 GB 376,1 GB 10:53:59

What is your opinion on this?

Regards

Alex

[Vitaliy S.]

Hello Alex,

1) I have one installation of vSphere Essentials Plus 4.0. It is based on ESX (not ESXi) hosts. Backups and restores are FAST. The storage behind it is a DELL MD300i ISCSI, SAS 15K Rpm disks.
2) I had a second installation of this kind (different customer) and was fast too. Then I decided to upgrade to vSphere 4.1 (and passed from ESX to ESXi) and the performance dropped totally and was a shame. I opened various tickets with the imaging software vendor and no joy. I upgraded to version 5 and then 5.1 an the performance bottleneck remains.

This is totally expected, as ESXi hosts do not have service console enabled. See this topic for more information on this: Service console agent or NBD mode: questions

When we first installed this last virtual environment, we didn't immediately activate VMWARE Essentials Plus licenses, therefore we were working in trial mode (and therefore VMWARE features were unrestricted). We installed VEEAM (on the vCenter virtual machine) and backups were FAST.
The a few days later we applied the Essentials plus licenses, and suddenly the backups become extremely slow (70% performance drop).

VMware license type does not effect the backup/replication job performance, so there should be something else that have changed in the setup. On top of that, in order to start troubleshooting your backup job performance, we need to know bottleneck statistics that can be found in the corresponding job session.

Thank you!