Comprehensive data protection for all workloads
Post Reply
mykl_74
Influencer
Posts: 21
Liked: 1 time
Joined: Feb 06, 2013 7:39 pm
Full Name: Mike Barnes
Contact:

Slow backups on large VM

Post by mykl_74 »

I'm having some trouble with backups of a particular VM. I want to basically know 1) Is the performance level I'm seeing expected? 2) What can I do to improve it if not.

This is a file server with a 1TB disk that has 600GB of data occupied. Prod storage is an IBM v3700, and this VM sits on 5x600gb 10K rpm SAS disks RAID5. Veeam backup storage is an IBM ds3300 with 11x450GB 10K rpm SAS disks RAID5 (and read/write cache is enabled- ds3300 pros will know what I mean here). I'm using Direct SAN for all jobs. Using iSCSI over 3560g 1GB switches. Veeam server is a Dell PE2950 8GB RAM and Xeon E5430 2.6 Ghz. So a mid range server, although I do have a brandy new server on order to replace this one.

I feel like my hardware is all pretty good. Trouble is- an Active Full backup takes forever on this large VM. Previous attempt failed after running for 14 hours and getting roughly 5MB/s. Most recent attempt finished, and took 27 hours and got 6MB/s. This seems like poor performance to me. I'm sure I can copy 600GB of data a lot faster than 27 hours using over methods. My bottlenecks usually look like this:
Source 16% > Proxy 42% > Network 1% > Target 99%
Source 2% > Proxy 23% > Network 1% > Target 97%

I had configured Windows iSCSI initiator to use MPIO while running the job that failed. After reading on the forums that Veeam/VCB don't like MPIO, I uninstalled it and reconfigured my iSCSI targets with a single path, although the v3700 & the DS3300 are using separate NICs. The 2nd job I ran that completed didn't use MPIO, so MPIO didn't seem to make any difference but still was pretty darn slow.
tsightler
VP, Product Management
Posts: 6035
Liked: 2860 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Slow backups on large VM

Post by tsightler »

5MB/s is pretty horrible and the fact that your seeing 99% target bottleneck says that something is wrong with the performance talking to the target storage (the DS3300). Have you run some simple benchmarks against this storage (something like CystalDiskMark perhaps)? Just as a guess I'd suspect some type of duplex mismatch on the iSCSI side, a NIC running at 100Mb unexpectedly, bad jumbo frame performance, or perhaps a bad network cable even. With the hardware you have I'd suspect at least 5-10x the speed you are seeing, perhaps even better.
f.dapra@ifinet.it
Novice
Posts: 4
Liked: never
Joined: Jul 24, 2012 12:10 pm
Full Name: Fabrizio Da Pra
Contact:

Re: Slow backups on large VM

Post by f.dapra@ifinet.it »

I had the same problems,
and after alot of time spent debugging was able to double the speed of the backup by disabling the "Dynamic cache read prefetch" on the source storage.
mykl_74
Influencer
Posts: 21
Liked: 1 time
Joined: Feb 06, 2013 7:39 pm
Full Name: Mike Barnes
Contact:

Re: Slow backups on large VM

Post by mykl_74 »

I ran some tests on my target storage using IOMeter. It gives the option to choose the block size used during the test. Does anyone know the block size Veeam uses so I can properly simulate disk I/O during my tests?
foggy
Veeam Software
Posts: 21139
Liked: 2141 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Slow backups on large VM

Post by foggy » 1 person likes this post

The block size is configurable and can be changed at the Storage step of the job settings (under Advanced > Storage > Storage optimizations). 1MB block is used for Local Target, 512KB for LAN Target and 256KB for WAN target.
dellock6
VeeaMVP
Posts: 6166
Liked: 1971 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Slow backups on large VM

Post by dellock6 » 1 person likes this post

And depending on the type of backup job you want to simulate, please add some randomness (dedupe and compression, especially for incremental jobs) and remember for example reverse incremental have a 3x I/O operations on the storage. So you would need to simulate 66%read/33% write, but only the write speed would be used during backup obviously.

Luca.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
mykl_74
Influencer
Posts: 21
Liked: 1 time
Joined: Feb 06, 2013 7:39 pm
Full Name: Mike Barnes
Contact:

Re: Slow backups on large VM

Post by mykl_74 »

Update: contrary to my previous statement, my ds3300's write cache somehow was disabled recently, so I enabled it yesterday. I then ran some simple disk IO tests using CrystalDisk. It has a 512KB block test built in which matches my Storage setting of LAN Target:
Without cache: Read 45MB/s Write 11.75MB/s
With cache: Read 45MB/s Write 38.68 MB/s

So the cache has helped greatly, at least for the test. I am running a full backup on the VM in question now. It seems better, but I have also noticed that the speed seems to be trending slower as the backup progresses. It's currently at 58% and is running 10MB/s. It was indicating ~15MB/s last night when I startedthe job. My 1GB iSCSI NICs are only using 5% bandwidth according to Task Manager. So, even with the cache enabled it's pretty slow overall.
mykl_74
Influencer
Posts: 21
Liked: 1 time
Joined: Feb 06, 2013 7:39 pm
Full Name: Mike Barnes
Contact:

Re: Slow backups on large VM

Post by mykl_74 »

Could the stripe size on the target storage be to blame? The device has a setting for what it calls segment size, which is set to 128KB. It has an option to use either 128KB or 256KB segment when you create a LUN.
yizhar
Service Provider
Posts: 182
Liked: 48 times
Joined: Sep 03, 2012 5:28 am
Full Name: Yizhar Hurwitz
Contact:

Re: Slow backups on large VM

Post by yizhar » 1 person likes this post

Hi.

I also have a DS3300 as backup repository, with 7 * 450 gb SAS 15k disks in RAID5 .
This device can perform faster , for example I can get about 100MB/sec during full backup.
I have a very fast 3PAR storage as production, but still the bottleneck is "source" in my case and not the DS3300 target.

One thing to recheck about "write cache" - check the write cache battery status as it stops functioning after 2 years and you need to reset it.
You can check it in DS Manager menu:
Tools - Change Battery Settings.
Consult with your IBM support if you aren't sure.

Also you should check If you have dual or single controllers, as single controller will disable write cache by default.

Yizhar
mykl_74
Influencer
Posts: 21
Liked: 1 time
Joined: Feb 06, 2013 7:39 pm
Full Name: Mike Barnes
Contact:

Re: Slow backups on large VM

Post by mykl_74 »

Hi Yizhar, thats good to know for the sake of comparison. I would be pretty happy if I can achieve even half the speed you're seeing. I did check the battery- I actually replaced the battery maybe a year ago or less, but I suppose it's possible it's bad again. And I have the dual controller model.

Out of curiosity, are you using iSCSI or a direct SAS connection to your DS3300? If iscsi, are you using MPIO or a single path to the DS?
I've seen mention here and there of using IBM provided storage drivers for the DS. Are you using the generic Windows iSCSI initiator or something IBM has provided?

Thanks for replying...
yizhar wrote:Hi.

I also have a DS3300 as backup repository, with 7 * 450 gb SAS 15k disks in RAID5 .
This device can perform faster , for example I can get about 100MB/sec during full backup.
I have a very fast 3PAR storage as production, but still the bottleneck is "source" in my case and not the DS3300 target.

One thing to recheck about "write cache" - check the write cache battery status as it stops functioning after 2 years and you need to reset it.
You can check it in DS Manager menu:
Tools - Change Battery Settings.
Consult with your IBM support if you aren't sure.

Also you should check If you have dual or single controllers, as single controller will disable write cache by default.

Yizhar
yizhar
Service Provider
Posts: 182
Liked: 48 times
Joined: Sep 03, 2012 5:28 am
Full Name: Yizhar Hurwitz
Contact:

Re: Slow backups on large VM

Post by yizhar »

mykl_74 wrote: Out of curiosity, are you using iSCSI or a direct SAS connection to your DS3300?
If iscsi, are you using MPIO or a single path to the DS?
I've seen mention here and there of using IBM provided storage drivers for the DS. Are you using the generic Windows iSCSI initiator or something IBM has provided?
I'm using ISCSI.

MPIO is installed and enabled, but I don't think that it is the key for better performance.
======================
C:\>mpclaim -e
"Target H/W Identifier " Bus Type MPIO-ed ALUA Support
-------------------------------------------------------------------------------
"IBM 1726-3xx FAStT " iSCSI YES ALUA Not Supported
"3PARdataVV " Fibre YES Implicit Only
======================

I'm using Windows generic drivers for ISCSI NIC (server is DELL R610).
NIC model is Broadcom BCM5709C NetXtreme II.
NIC driver version is 6.0.29.0

I'm not using any special IBM storage drivers, I just installed the management GUI .

Yizhar
mykl_74
Influencer
Posts: 21
Liked: 1 time
Joined: Feb 06, 2013 7:39 pm
Full Name: Mike Barnes
Contact:

Re: Slow backups on large VM

Post by mykl_74 »

I tested copying 137GB if Veeam files from a NAS to the DS3300. It sustained write speeds of 60-65MBps for about a half hour, so the device seems capable of writing the data fast enough. I suspect it's either processor bound or the block size is to blame.
yizhar
Service Provider
Posts: 182
Liked: 48 times
Joined: Sep 03, 2012 5:28 am
Full Name: Yizhar Hurwitz
Contact:

Re: Slow backups on large VM

Post by yizhar »

Hi.

Problem might be related to the Dell PE2950 server.
You can do the following test:

1. Configure a different server as Veeam proxy (either virtual or a different physical box).
Configure it with direct access (via ISCSI) to the DS3300 .

2. Try to run a similar backup job, with DS3300 as target.

3. You can also try a small job with PE2950 as the proxy, but to different repository other then the DS3300, for example a folder on local disks.
Compare and share the results.

Yizhar
dellock6
VeeaMVP
Posts: 6166
Liked: 1971 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Slow backups on large VM

Post by dellock6 »

Sorry to repeat, but a sequential file copy is NOT a simulation of the performance you are going to have with Veeam. The only simulation is a Veeam Backup itself, that's another reason the trial is there for...

Luca.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
mykl_74
Influencer
Posts: 21
Liked: 1 time
Joined: Feb 06, 2013 7:39 pm
Full Name: Mike Barnes
Contact:

Re: Slow backups on large VM

Post by mykl_74 »

Very true- I understand that. Just going through process of elimination, to see if the device is capable of a decent write speed.
dellock6 wrote:Sorry to repeat, but a sequential file copy is NOT a simulation of the performance you are going to have with Veeam. The only simulation is a Veeam Backup itself, that's another reason the trial is there for...

Luca.
yizhar
Service Provider
Posts: 182
Liked: 48 times
Joined: Sep 03, 2012 5:28 am
Full Name: Yizhar Hurwitz
Contact:

Re: Slow backups on large VM

Post by yizhar »

mykl_74 wrote: Very true- I understand that. Just going through process of elimination, to see if the device is capable of a decent write speed.
Exactly.
I agree that file copy is not the same as backup job, but it is a good test among other things to check.

Here are results of file copy on my system -
from local disks on dell r610 backup server (local sas raid5), to DS3300 via ISCSI,
I can see the 1gbps network link maxed out at about 110MB/sec.

Yizhar
mmonroe
Enthusiast
Posts: 75
Liked: 3 times
Joined: Jun 16, 2010 8:16 pm
Full Name: Monroe
Contact:

Re: Slow backups on large VM

Post by mmonroe »

I just build a large file server to move our marketing files over to and I am having the exact same issue. My VM is about 1.4tb with about 800gig used. The VM is 2008 R2 SP1 and has two disks - the C-drive is about 30gig and the D-drive is about 1.4tb and has the 800 gig of data on it.

My Veeam (6.5 with patch #3 loaded) server is a standalone server with SAN access. I also have a Veeam "proxie" box that is a VM on the farm but is is never really used. I have about 40 other smaller VMs that this box and they all see "normal" speeds.

This new 1.4tb VM sees 80-100MB/s on the C-drive - which I think is normal. When it starts on the D-drive, the speeds start at 80-100MB/s and slowly erode to the point where I am seem <10MB/s. I tried using the proxie on the farm in virtual appliance mode to see if that would make a difference and it is slowing down just like the physical standalone server does in SAN mode.

When things start slowing down, the CPUs are mostly idle, the disk access is mostly idle, the network is mostly idle. The storage respository is a 30disk RAID 10 SAN and is very fast. I have run disk benchmarks and they are good. I checked the stats on the SAN directly when this slowing down occurs and the SAN is mostly idle.

As best I can tell, Veeam just starts slowing down and gets slower and slower as it tries to work its way through the VM and this larger D-drive. I have not been able to get a complete backup on the VM yet and am starting to get a bit nervous about it.

*** One more thing - this physical Veeam standalone server has local disk via fiber to the SAN - no ISCI. The same physical server has fiber access to the shared storage on the VMware farm - no ISCI. Basically there is no no network traffic at all involved with the backup data. The VMware farm is a 100% flash based SAN and the local storage space for the data is on the 30disk RAID 10 fiber SAN. All of it is fast and tests with excellent performance.
alonsk
Lurker
Posts: 2
Liked: 2 times
Joined: Sep 01, 2013 2:37 pm
Full Name: AL
Contact:

Re: Slow backups on large VM

Post by alonsk »

mmonroe wrote:I just build a large file server to move our marketing files over to and I am having the exact same issue. My VM is about 1.4tb with about 800gig used. The VM is 2008 R2 SP1 and has two disks - the C-drive is about 30gig and the D-drive is about 1.4tb and has the 800 gig of data on it.

My Veeam (6.5 with patch #3 loaded) server is a standalone server with SAN access. I also have a Veeam "proxie" box that is a VM on the farm but is is never really used. I have about 40 other smaller VMs that this box and they all see "normal" speeds.

This new 1.4tb VM sees 80-100MB/s on the C-drive - which I think is normal. When it starts on the D-drive, the speeds start at 80-100MB/s and slowly erode to the point where I am seem <10MB/s. I tried using the proxie on the farm in virtual appliance mode to see if that would make a difference and it is slowing down just like the physical standalone server does in SAN mode.

When things start slowing down, the CPUs are mostly idle, the disk access is mostly idle, the network is mostly idle. The storage respository is a 30disk RAID 10 SAN and is very fast. I have run disk benchmarks and they are good. I checked the stats on the SAN directly when this slowing down occurs and the SAN is mostly idle.

As best I can tell, Veeam just starts slowing down and gets slower and slower as it tries to work its way through the VM and this larger D-drive. I have not been able to get a complete backup on the VM yet and am starting to get a bit nervous about it.

*** One more thing - this physical Veeam standalone server has local disk via fiber to the SAN - no ISCI. The same physical server has fiber access to the shared storage on the VMware farm - no ISCI. Basically there is no no network traffic at all involved with the backup data. The VMware farm is a 100% flash based SAN and the local storage space for the data is on the 30disk RAID 10 fiber SAN. All of it is fast and tests with excellent performance.

Which OS is installed on your Backup repository server? I have the same problem and I think that reason in my case is that Backup target server operating system is Windows Server 2003.
yizhar
Service Provider
Posts: 182
Liked: 48 times
Joined: Sep 03, 2012 5:28 am
Full Name: Yizhar Hurwitz
Contact:

Re: Slow backups on large VM

Post by yizhar »

Hi.

Please provide more details about the hardware involved (brand, model, etc).

Do you use "thin provisioning" on the target repository?

Yizhar
alonsk
Lurker
Posts: 2
Liked: 2 times
Joined: Sep 01, 2013 2:37 pm
Full Name: AL
Contact:

Re: Slow backups on large VM

Post by alonsk » 2 people like this post

I resolved the problem by changing OS on backup repository server from Windows server 2003 SP2 to Windows Server 2008 R2 SP1. Now sustained backup speed on large VMs are constant and limited only by network speed, in my case link speed 1 Gbps, backup speed ~120 MB/s
mykl_74
Influencer
Posts: 21
Liked: 1 time
Joined: Feb 06, 2013 7:39 pm
Full Name: Mike Barnes
Contact:

Re: Slow backups on large VM

Post by mykl_74 »

mykl_74 wrote:Could the stripe size on the target storage be to blame? The device has a setting for what it calls segment size, which is set to 128KB. It has an option to use either 128KB or 256KB segment when you create a LUN.
I've finally been able to get good throughput with my Veeam backups. I'm now seeing 54-77MB/s, one VM even gets up to 116MB/s on full backups. We got a new server and I also updated to version 7. But I think the real reason for the speed increase is that I changed to using 2 proxies in appliance mode. I haven't been able to determine why, but direct SAN mode just does not work for me. For now, I am happy that proxy mode is working so well. I may spend some time troubleshooting direct san some day. But for now other projects need attention.
Post Reply

Who is online

Users browsing this forum: Bing [Bot] and 66 guests