Are my backups slow?

[mongie]

I'm always struggling with our Exchange 2003 backup. It seems to take forever, and it causes heaps of issues for us when it comes time to remove the snapshot. I understand that backups with a large amount of changes will take longer... I do wonder though - are our backups slow? Running jobs on some servers will see 1-10MB/s speeds on the VMDK's. Is that acceptable speed?

We're using Veeam 6 (patch 3) on a Dell r710 physical box with 2x4c Xeons (with HT) and 16GB ram. I dont see any CPU bottleneck. Target is a 60TB RAID 50 volume on 2x Dell MD1200 DAS shelves connected via a H800 SAS controller. The physical backup box is connected via 2x10Gbit nic's to our iSCSI network (Dell Equallogic SANs - also 10Gbit). You never really see anything more than 5-6% usage on the 10Gbit nics (on the Veeam box).

I dont see any major performance issues with our SANs, but I'm not an expert. I just wonder if that is acceptable speeds? I would have hoped for faster... It would certainly make my backups easier.

Thanks,
Alex

[dellock6]

What are the bottlenek values shown in the job and the backup method effectively being used by the backup proxy?

[cag]

I have a similar setup. 2x Dell R710 24GB Ram. MD3200 DAS raid 5. 1.8TB
VM is exchange 2010 on Windows 2008 SBS.
Backup Time = 3.5 Hrs on first full backup = 330GB VM.
Backup target is a Linux server. access via gigbit network.

Second backup (INC) averages 30-40 Minutes.
Backup sizes Full = 330GB
inc backup averages = 40GB de-dupe size=18GB

I would say its slow.

[jessy5765]

We too have this issue. When we backup our File and Exchange servers the full backups take 5-6 hours. Then the incrementals are taking 3 hours. However I can say that Exchange and File server are both about 1 TB of data each. From what I understand Veeam is actually reading the HDD of the servers for all changes in data from the last backup. The actual size of the incremental is only 80GB

Newest Veeam B&R 6.0 Installed
Exchange Backup 950GB in drive size
Reverse Incremental with no active fulls
No SureBackup is setup for this job.

I currently have a ticket in to Veeam for this. Its saying that there is a 99% bottleneck at the target, however this is a Dell 2900 4 cores, 8GB RAM, Raid 5 12TB DataStore.

[tsightler]

I would have to say it's slow, but that's just a very general statement. You don't mention the size of your servers, or their total time. How big is your Exchange server, how long does a full backup take, and what MB/s do you see with that?

[J1mbo]

Just some general thoughts. Do you have flow-control enabled throughout? Have you looked at the TCP retransmit counters on the EqualLogic? What is the IOPS load on the EqualLogic in the backup window? Is the H800 cache configured for write-back?

[mongie]

Weirdly, I ran a manual full backup on my exchange server yesterday, and got really good performance.

Busy: Source 98%> Proxy 85%> Network 6%> Target 3%

Overall processing rate 245MB/s

Some VMDKs showed processing rates of 1GB/s.

My exchange backup is ~ 450 and the total voume size is ~ 2.1TB (although most of that is free thin provisioned disk).
File servers backup is ~ 5.2TB.

If I look at another backup of production servers (SAP) I see "HDD 1 - 40GB, 530MB read at 18MB/s", and similar figures for all disks.
This time, the bottleneck is the target apparently, even though the datastores are spread over the same three PS6000's.

I'll have a look at my configs, but J1mbo, are you asking me those questions because thats how I should have things configured? or becasue I shouldn't use those configurations?

[tsightler]

If I look at another backup of production servers (SAP) I see "HDD 1 - 40GB, 530MB read at 18MB/s", and similar figures for all disks.
This time, the bottleneck is the target apparently, even though the datastores are spread over the same three PS6000's.

So are you using reverse incremental backups? And based on your statement above your backup target is the same as the source storage? That's going to place a tremendous I/O load on the storage, especially if you are using a reverse incremental backup.

That being said, are you actually measuring the performance or are you just interpreting the "18MB/s" number and assuming that's bad? How much data did it actually have to read and transfer?

[mongie]

The source and target are totally different.

All VMWare datastores are spread over 3xEquallogic PS6000 SANs. The Backup server is a physical box with 2x Dell MD1200's running via a H800. It connects to the SAN network via 2x 10Gbit Intel nics and through some Dell 10Gbit switches.

I am interpreting the numbers shown in the Backup logs... to me it would seem that when you're talking about slow speeds, it doesn't make much difference if its synthetic or not. The fact its a synthetic speed only matters if its fast (empty space etc.) If its slow (e.g. 18MB/s) and its backed up a lot of empty space etc... then it just shows its slow. The "synthetic" calculation would only improve the speed, not make it look worse than it is. So when I'm seeing the slower numbers in the logs, its generally because its a more realistic figure, not impacted by the synthetic calculation.

Wow, that was hard to explain.

Thanks,
Alex

[tsightler]

Who said anything about synthetic? Reverse incremental vs forward incremental is a HUGE difference. For reverse incremental, the target storage will almost always be the bottleneck. This is explained in other threads, but primarily this is because it requires 3x the I/O of a full backup/incremental backup on the target storage, and, even worse, it's effectively 100% random I/O, not a nice sequential write like a full or incremental backup. This is because, to perform a reverse incremental, for each changed block read from the source storage, 3 blocks have to be moved on the target storage.

For servers which have a high change rate (typically transactional servers), forward incremental will generally be much faster than reverse incremental, typically 3-4x faster or more, with the penalty that you occasionally have to run a full backup, and of course this requires more space.

[chimera]

Alex, what firmware version are you running on the EqualLogic? I think the latest 5.2.1 seems to be a bit buggy, coz after upgrading the latency under SAN HQ has gone through the roof and Veeam backup performance has dropped from 70-130MB/sec to around 3-5MB/sec average. Oddly this only appears to be from the Veeam (Win2k8 R2) server, ESXi 5.0 latest updates seems to run fine.

[SunkistDavid]

Chimera, we are running EQ's with 5.2.1 and are also experiencing slow performance issues across the board with our backups. I don't want to point to just the EQ's as it could be many things in our complex network. Having said that have you upgraded to the latest version of the EQ software as they just released an updated version after 5.2.1? My avg latency is between 6.8 and 7 on Read and 2.4 and 3.9 ms on writes. I haven't upgraded yet as I hadn't heard about any issues with 5.2.1 until seeing your comment.

I may also have to re-examine how I'm backing-up. RIght now I have two clusters with each cluster having 3 ESX hosts. I have two different jobs pointed to the host via the cluster. I run reverse incrementals with no diffentiation for for Exchange 07 and SQL but all my jobs are slow not just SQL/EX. One problem I have is some guest VM's on a host have say a 10.10.x.x network and others 10.11.x.x network with the two networks seperated through a firewall and a router. I'm backing-up to an Overland NAS SnapServer DX2, a new NAS from Overland. I'm backing-up to a CIFS share on the Overland.

[chimera]

Hi David,

Latency under 5ms is generally considered acceptable, so you're not too bad - at least its excessive like my clients! Interestingly, its only read latency, writes are ok - although, we experienced this with the default Broadcom 57711 10GbE NIC driver that comes with ESX - it was buggy and needed upgrading. Prior to that upgrade, same symptoms with high read latency.

Only reason I mention it is our Veeam backups are running exceptionally slow since the 24th of March. I'm trying to go back through the logs to find out what changes took place on or after this date, as we made a few changes including upgrading Veeam to patch 3. In saying that though, I have patch 3 at other clients and its running fine. The 5.2.1 firmware was released February, and now Dell release 5.2.2 a month later. The release notes don't specifically state anything to do with performance, but that maybe a red herring - I will be upgrading the client to 5.2.2 soon and see if this makes a difference.

Cheers

[SunkistDavid]

Let's us know once you upgrade if that makes a difference. I'm currently testing with a proxy server on the same network as the Mgt. Nic of the ESX server and its making quite a difference. My VM's are processing at 25MB/s versus 3-6. Still not your top range but I'll take what I can get.

[ctchang]

You are not alone. The only difference I don't have any performance drop. (ESX 4.1 + Veeam B&R 5)

http://www.modelcar.hk/?p=5768