Slow backups on large VM

[mykl_74]

I'm having some trouble with backups of a particular VM. I want to basically know 1) Is the performance level I'm seeing expected? 2) What can I do to improve it if not.

This is a file server with a 1TB disk that has 600GB of data occupied. Prod storage is an IBM v3700, and this VM sits on 5x600gb 10K rpm SAS disks RAID5. Veeam backup storage is an IBM ds3300 with 11x450GB 10K rpm SAS disks RAID5 (and read/write cache is enabled- ds3300 pros will know what I mean here). I'm using Direct SAN for all jobs. Using iSCSI over 3560g 1GB switches. Veeam server is a Dell PE2950 8GB RAM and Xeon E5430 2.6 Ghz. So a mid range server, although I do have a brandy new server on order to replace this one.

I feel like my hardware is all pretty good. Trouble is- an Active Full backup takes forever on this large VM. Previous attempt failed after running for 14 hours and getting roughly 5MB/s. Most recent attempt finished, and took 27 hours and got 6MB/s. This seems like poor performance to me. I'm sure I can copy 600GB of data a lot faster than 27 hours using over methods. My bottlenecks usually look like this:
Source 16% > Proxy 42% > Network 1% > Target 99%
Source 2% > Proxy 23% > Network 1% > Target 97%

I had configured Windows iSCSI initiator to use MPIO while running the job that failed. After reading on the forums that Veeam/VCB don't like MPIO, I uninstalled it and reconfigured my iSCSI targets with a single path, although the v3700 & the DS3300 are using separate NICs. The 2nd job I ran that completed didn't use MPIO, so MPIO didn't seem to make any difference but still was pretty darn slow.

[tsightler]

5MB/s is pretty horrible and the fact that your seeing 99% target bottleneck says that something is wrong with the performance talking to the target storage (the DS3300). Have you run some simple benchmarks against this storage (something like CystalDiskMark perhaps)? Just as a guess I'd suspect some type of duplex mismatch on the iSCSI side, a NIC running at 100Mb unexpectedly, bad jumbo frame performance, or perhaps a bad network cable even. With the hardware you have I'd suspect at least 5-10x the speed you are seeing, perhaps even better.

[f.dapra@ifinet.it]

I had the same problems,
and after alot of time spent debugging was able to double the speed of the backup by disabling the "Dynamic cache read prefetch" on the source storage.

[mykl_74]

I ran some tests on my target storage using IOMeter. It gives the option to choose the block size used during the test. Does anyone know the block size Veeam uses so I can properly simulate disk I/O during my tests?

[foggy]

The block size is configurable and can be changed at the Storage step of the job settings (under Advanced > Storage > Storage optimizations). 1MB block is used for Local Target, 512KB for LAN Target and 256KB for WAN target.

[dellock6]

And depending on the type of backup job you want to simulate, please add some randomness (dedupe and compression, especially for incremental jobs) and remember for example reverse incremental have a 3x I/O operations on the storage. So you would need to simulate 66%read/33% write, but only the write speed would be used during backup obviously.

Luca.

[mykl_74]

Update: contrary to my previous statement, my ds3300's write cache somehow was disabled recently, so I enabled it yesterday. I then ran some simple disk IO tests using CrystalDisk. It has a 512KB block test built in which matches my Storage setting of LAN Target:
Without cache: Read 45MB/s Write 11.75MB/s
With cache: Read 45MB/s Write 38.68 MB/s

So the cache has helped greatly, at least for the test. I am running a full backup on the VM in question now. It seems better, but I have also noticed that the speed seems to be trending slower as the backup progresses. It's currently at 58% and is running 10MB/s. It was indicating ~15MB/s last night when I startedthe job. My 1GB iSCSI NICs are only using 5% bandwidth according to Task Manager. So, even with the cache enabled it's pretty slow overall.

[mykl_74]

Could the stripe size on the target storage be to blame? The device has a setting for what it calls segment size, which is set to 128KB. It has an option to use either 128KB or 256KB segment when you create a LUN.

[yizhar]

Hi.

I also have a DS3300 as backup repository, with 7 * 450 gb SAS 15k disks in RAID5 .
This device can perform faster , for example I can get about 100MB/sec during full backup.
I have a very fast 3PAR storage as production, but still the bottleneck is "source" in my case and not the DS3300 target.

One thing to recheck about "write cache" - check the write cache battery status as it stops functioning after 2 years and you need to reset it.
You can check it in DS Manager menu:
Tools - Change Battery Settings.
Consult with your IBM support if you aren't sure.

Also you should check If you have dual or single controllers, as single controller will disable write cache by default.

Yizhar

[mykl_74]

Hi Yizhar, thats good to know for the sake of comparison. I would be pretty happy if I can achieve even half the speed you're seeing. I did check the battery- I actually replaced the battery maybe a year ago or less, but I suppose it's possible it's bad again. And I have the dual controller model.

Out of curiosity, are you using iSCSI or a direct SAS connection to your DS3300? If iscsi, are you using MPIO or a single path to the DS?
I've seen mention here and there of using IBM provided storage drivers for the DS. Are you using the generic Windows iSCSI initiator or something IBM has provided?

Thanks for replying...

Hi.

I also have a DS3300 as backup repository, with 7 * 450 gb SAS 15k disks in RAID5 .
This device can perform faster , for example I can get about 100MB/sec during full backup.
I have a very fast 3PAR storage as production, but still the bottleneck is "source" in my case and not the DS3300 target.

One thing to recheck about "write cache" - check the write cache battery status as it stops functioning after 2 years and you need to reset it.
You can check it in DS Manager menu:
Tools - Change Battery Settings.
Consult with your IBM support if you aren't sure.

Also you should check If you have dual or single controllers, as single controller will disable write cache by default.

Yizhar

[yizhar]

Out of curiosity, are you using iSCSI or a direct SAS connection to your DS3300?
If iscsi, are you using MPIO or a single path to the DS?
I've seen mention here and there of using IBM provided storage drivers for the DS. Are you using the generic Windows iSCSI initiator or something IBM has provided?

I'm using ISCSI.

MPIO is installed and enabled, but I don't think that it is the key for better performance.
======================
C:\>mpclaim -e
"Target H/W Identifier " Bus Type MPIO-ed ALUA Support
-------------------------------------------------------------------------------
"IBM 1726-3xx FAStT " iSCSI YES ALUA Not Supported
"3PARdataVV " Fibre YES Implicit Only
======================

I'm using Windows generic drivers for ISCSI NIC (server is DELL R610).
NIC model is Broadcom BCM5709C NetXtreme II.
NIC driver version is 6.0.29.0

I'm not using any special IBM storage drivers, I just installed the management GUI .

Yizhar

[mykl_74]

I tested copying 137GB if Veeam files from a NAS to the DS3300. It sustained write speeds of 60-65MBps for about a half hour, so the device seems capable of writing the data fast enough. I suspect it's either processor bound or the block size is to blame.

[yizhar]

Hi.

Problem might be related to the Dell PE2950 server.
You can do the following test:

1. Configure a different server as Veeam proxy (either virtual or a different physical box).
Configure it with direct access (via ISCSI) to the DS3300 .

2. Try to run a similar backup job, with DS3300 as target.

3. You can also try a small job with PE2950 as the proxy, but to different repository other then the DS3300, for example a folder on local disks.
Compare and share the results.

Yizhar

[dellock6]

Sorry to repeat, but a sequential file copy is NOT a simulation of the performance you are going to have with Veeam. The only simulation is a Veeam Backup itself, that's another reason the trial is there for...

Luca.

[mykl_74]

Very true- I understand that. Just going through process of elimination, to see if the device is capable of a decent write speed.

Sorry to repeat, but a sequential file copy is NOT a simulation of the performance you are going to have with Veeam. The only simulation is a Veeam Backup itself, that's another reason the trial is there for...

Luca.

[yizhar]

Very true- I understand that. Just going through process of elimination, to see if the device is capable of a decent write speed.

Exactly.
I agree that file copy is not the same as backup job, but it is a good test among other things to check.

Here are results of file copy on my system -
from local disks on dell r610 backup server (local sas raid5), to DS3300 via ISCSI,
I can see the 1gbps network link maxed out at about 110MB/sec.

Yizhar

[mmonroe]

I just build a large file server to move our marketing files over to and I am having the exact same issue. My VM is about 1.4tb with about 800gig used. The VM is 2008 R2 SP1 and has two disks - the C-drive is about 30gig and the D-drive is about 1.4tb and has the 800 gig of data on it.

My Veeam (6.5 with patch #3 loaded) server is a standalone server with SAN access. I also have a Veeam "proxie" box that is a VM on the farm but is is never really used. I have about 40 other smaller VMs that this box and they all see "normal" speeds.

This new 1.4tb VM sees 80-100MB/s on the C-drive - which I think is normal. When it starts on the D-drive, the speeds start at 80-100MB/s and slowly erode to the point where I am seem <10MB/s. I tried using the proxie on the farm in virtual appliance mode to see if that would make a difference and it is slowing down just like the physical standalone server does in SAN mode.

When things start slowing down, the CPUs are mostly idle, the disk access is mostly idle, the network is mostly idle. The storage respository is a 30disk RAID 10 SAN and is very fast. I have run disk benchmarks and they are good. I checked the stats on the SAN directly when this slowing down occurs and the SAN is mostly idle.

As best I can tell, Veeam just starts slowing down and gets slower and slower as it tries to work its way through the VM and this larger D-drive. I have not been able to get a complete backup on the VM yet and am starting to get a bit nervous about it.

*** One more thing - this physical Veeam standalone server has local disk via fiber to the SAN - no ISCI. The same physical server has fiber access to the shared storage on the VMware farm - no ISCI. Basically there is no no network traffic at all involved with the backup data. The VMware farm is a 100% flash based SAN and the local storage space for the data is on the 30disk RAID 10 fiber SAN. All of it is fast and tests with excellent performance.

[alonsk]

I just build a large file server to move our marketing files over to and I am having the exact same issue. My VM is about 1.4tb with about 800gig used. The VM is 2008 R2 SP1 and has two disks - the C-drive is about 30gig and the D-drive is about 1.4tb and has the 800 gig of data on it.

My Veeam (6.5 with patch #3 loaded) server is a standalone server with SAN access. I also have a Veeam "proxie" box that is a VM on the farm but is is never really used. I have about 40 other smaller VMs that this box and they all see "normal" speeds.

This new 1.4tb VM sees 80-100MB/s on the C-drive - which I think is normal. When it starts on the D-drive, the speeds start at 80-100MB/s and slowly erode to the point where I am seem <10MB/s. I tried using the proxie on the farm in virtual appliance mode to see if that would make a difference and it is slowing down just like the physical standalone server does in SAN mode.

When things start slowing down, the CPUs are mostly idle, the disk access is mostly idle, the network is mostly idle. The storage respository is a 30disk RAID 10 SAN and is very fast. I have run disk benchmarks and they are good. I checked the stats on the SAN directly when this slowing down occurs and the SAN is mostly idle.

As best I can tell, Veeam just starts slowing down and gets slower and slower as it tries to work its way through the VM and this larger D-drive. I have not been able to get a complete backup on the VM yet and am starting to get a bit nervous about it.

*** One more thing - this physical Veeam standalone server has local disk via fiber to the SAN - no ISCI. The same physical server has fiber access to the shared storage on the VMware farm - no ISCI. Basically there is no no network traffic at all involved with the backup data. The VMware farm is a 100% flash based SAN and the local storage space for the data is on the 30disk RAID 10 fiber SAN. All of it is fast and tests with excellent performance.

Which OS is installed on your Backup repository server? I have the same problem and I think that reason in my case is that Backup target server operating system is Windows Server 2003.

[yizhar]

Hi.

Please provide more details about the hardware involved (brand, model, etc).

Do you use "thin provisioning" on the target repository?

Yizhar

[alonsk]

I resolved the problem by changing OS on backup repository server from Windows server 2003 SP2 to Windows Server 2008 R2 SP1. Now sustained backup speed on large VMs are constant and limited only by network speed, in my case link speed 1 Gbps, backup speed ~120 MB/s

[mykl_74]

Could the stripe size on the target storage be to blame? The device has a setting for what it calls segment size, which is set to 128KB. It has an option to use either 128KB or 256KB segment when you create a LUN.

I've finally been able to get good throughput with my Veeam backups. I'm now seeing 54-77MB/s, one VM even gets up to 116MB/s on full backups. We got a new server and I also updated to version 7. But I think the real reason for the speed increase is that I changed to using 2 proxies in appliance mode. I haven't been able to determine why, but direct SAN mode just does not work for me. For now, I am happy that proxy mode is working so well. I may spend some time troubleshooting direct san some day. But for now other projects need attention.