Poor performance from DAS to NAS

[matarvai]

Hi, I am using Veeam backup 4.1 version to backup our virtual servers. I just started to use the product and I am having difficuties with performance. We have 10VM running on DAS (8x146GB SAS) and we want to backup those servers to our 6TB NAS. I have dedicated one NIC for this purpose. When we take backup from our servers it takes about 17h, and speed is only 5-11mb/s. What could cause this problem?? When we copy files from NAS to server or vice versa speed is MUCH better.

We use vmware storage api (network), without VSS, deduplication and optimal compression.

[tsightler]

What version of ESX? Since your using DAS the only method to access the VMDK files is via VMware Storage API (network) mode which is what you state you are using. With ESX 4 this method is known to be extremely slow due to a VMware issue. You might want to see this thread:

http://www.veeam.com/forums/viewtopic.p ... ork#p10859

The patch listed supposedly helps performance somewhat, but in my environment I saw very little improvement and performance is still only about 10-15MB/sec, whereas I get 50-60MB/sec with SAN mode. Incremental passes should be much better since it only has to copy changes however, this can help offset the performance issue somewhat.

I think you can use virtual appliance mode with DAS so that might be an option.

[matarvai]

I have ESXi 4 version. I will try the virtual appliance mode. Maybe that will help.

[Gostev]

Yes, virtual appliance mode will definitely help and improve the speed in a few times. Thanks Tom.

[matarvai]

Yes, virtual appliance mode will definitely help and improve the speed in a few times. Thanks Tom.

I have using the virtual appliance mode but still performance is really poor. I get about 11-13mb/s performance, it is little bit better than before. What could cause this problem? On my backup server CPU usage is about 20-35% during backup, there is memory available. NAS can handle much more data and it is receiving data much faster when I transfer files from Windows directly.

I only have one server with local disks. I have four SAS 10k disks on RAID-5 and four SAS 15k disks on RAID-5 where my production servers are running and these I want to backup. My backup server is running on same server, but on own RAID partition (three SATA drives on RAID-5). Do I need own server for backup? I don't think so. I only have five VM that I need to backup daily, but with speed 11-13mb/s it takes almost a day to backup those.

[Gostev]

I have using the virtual appliance mode but still performance is really poor. I get about 11-13mb/s performance, it is little bit better than before. What could cause this problem?

Did you give Veeam Backup VM 4 vCPU when testing the Virtual Appliance mode?

[jyarborough]

I am seeing the same basic problems you are but I have a SAN to NAS scenario. I created a new post http://www.veeam.com/forums/viewtopic.php?f=2&t=3546 that details my problems and findings. I am seeing the same 25% to 30% CPU per job when I have 4 vCPU's assigned to the virtual machine running Veeam. I did find that this seems to be per running job so if I have three running jobs, the total CPU is 75% (25% per VeeamAgent.exe instance). I'm still doing some testing but I do not feel that it is a bootleneck at either end of my storage system, but instead something going on with the VeeamAgent.exe. As with you, I can copy files to the NAS much faster through Windows than what Veeam is getting for backup speeds on some jobs.

[Gostev]

How did you specify your backup destination in the job wizard?

[matarvai]

How did you specify your backup destination in the job wizard?

I have destination as \\nas-ip-address\backup

[matarvai]

I have three jobs running at the same time and then I get CPU usage to about 80% with 4-cpu. But still speed is about 15mb/s all three together. One is 10mb/s, and two others are about 1mb/s

[Gostev]

Try to backup locally to Veeam Backup VM and see what performance is, this should help to isolate the performance issue.

[matarvai]

Try to backup locally to Veeam Backup VM and see what performance is, this should help to isolate the performance issue.

I tried to backup from DAS to DAS, different raid partitions. Then I get much better speed about 27mb/s

[matarvai]

When I copy one vmdk file from vSphere client to different partition I get almost three times the disk utilization then from backup.

[matarvai]

After first full to DAS to DAS second backup speed was 41mb/s

[Gostev]

I tried to backup from DAS to DAS, different raid partitions. Then I get much better speed about 27mb/s

That would indicate that the issue is with NAS performance, or network connection between Veeam Backup VM and NAS.

[matarvai]

I bought new iSCSI NAS and tried to copy files to that from Veeam backup VM. I get about 46000KB/s on NAS performance monitor, when normal Windows file transfer is on. When using Veeam backup I get only 11700-15600KB/S. Still the running backups takes more than one day when using Veeam. And what is strange, is that after first full, second backups still takes almost same time, even the backup file is much smaller.

[Gostev]

Hello, it is not clear from your description if you are using iSCSI NAS as production or backup storage, and how you have it connected to the backup server. Also, what backup mode are you using, Virtual Appliance? Did you set up Veeam Backup VM 4 vCPU?

I get about 46000KB/s on NAS performance monitor, when normal Windows file transfer is on. When using Veeam backup I get only 11700-15600KB/S.

If you are talking about NAS being backup target here, than this is normal, as Veeam heavily compresses and dedupes source data before writing it to target storage (up to 5-10 times). So for each 100MB of data read from source, only 10MB is written to backup target. So basically, full bandwidth NAS is capable to provide is simply not required.

And what is strange, is that after first full, second backups still takes almost same time, even the backup file is much smaller.

Do you see "with changed block tracking" in the sessions history for the corresponding job run?

[matarvai]

I am using this new iSCSI NAS only for backup storage, and it is connected with dedicated 1GB LAN to our server. Backup mode is virtual appliance mode, and VM is with 4xCPU. CPU usage doesn't go more than 27% percent when doing backup.

Backup mode: HOTADD with changed block tracking

[Gostev]

How do you present iSCSI NAS to Veeam Backup server in the job settings?
What is processing rate according to the job settings on the full and incremental passes?

I am curious as to why incremental passes take nearly same time as full, never seen anything like this...

[matarvai]

iSCSI NAS shows as normal Windows drive. I have given extra drive from Vmware console to this VM. And that vmdk file is on iSCSI NAS.

[matarvai]

I reinstalled my virtual server and Veeam Backup server. First 50gb virtual machine took 48minutes (3mb/s), Second VM backup 160GB took 8h 48min (5mb/s), but second backup from 160gb VM took only 1h51min on second day. Speed was 24m/s. It is still slow, but if the first backup only is so slow, I can backup all my five virtual machines daily. Last time backup cycle took more than day, so I couldn't backup daily every machine.

[Gostev]

Marko, only thing I can suspect from all the facts above are issues with your production storage performance... It looks like disk read speed is very low, because you are getting at least 10x times slower speed than expected for both full and incremental passes. 30MB/s is really minimal speed you should be getting for full backups, and you only get 3MB/s.

Have you been working with our support on this? What is the support case ID?

[matarvai]

When I copy data from another raid partition to another from ESXi console I get over 100000KBps speed on ESXi performance. But when doing backup it stays below 10000KBps. When copying same files from production storage to iSCSI I get over 46000KBps speed. So I really don't understand how the speed is low only on backup. I haven't create support case for this yet.

[Gostev]

Marlo, I was just troubleshooting some performance issue in my own lab and remembered about your issue... do you by any chance have some antivirus with real-time protection enabled on your Veeam Backup server?

Also, what do you mean by "copying data from ESXi console"? ESXi has service console access disabled, so I am not sure what are you saying here... what tool did you use to copy the data for this test?

[matarvai]

I have antivirus software on server, and what I meant earlier was that when I co to Vmware vSphere client, open storage browser and copy data VM from on datastore to another.