Really slow backups - Any suggestions?

[pauliem]

I'm having a problem backing up a VM with about 1Tb of data, it seems to start OK but when backing up the drive with a lot of data on it seems to get slower and slower.

Setup -

Dell R510 running ESXi 4.1 Essentials
24 GB Ram
4 x 15k SAS (Raid 10)
4 x 7,200 RPM SATA (Raid 10)
Only 1 Windows 2003 VM (32-Bit) Running within at the moment with about 1.5tb storage provisioned and 825gb used storage.

Dell 2850 running Windows 2003 64-Bit and VBR 5, backups are being stored on iSCSI NAS, but speed problem is the same when using local storage more or less.

Everything connected on Gigabit unmanaged switch.

Currently running a full backup of the fileserver and so far it has taken around 16 hours to process 924Gb, VBR estimates another 10 hours remaining. Processing speed is reported at 16MB/s. The "Processed Size" value is hardly ever updating now though and was definatley quicker at the start of the process.

We also have another Dell R510 exactly the same setup running a single VM which seems to be backing up withing a problem, although it has much less data.

Any suggestions, should I cancel this job and give up on it and try a different tactic?

The first time I have used V5, is there a V4 download that I can download to try - noticed that some others have done the same.

[Gostev]

If you do not have this issue on another Dell R510, then most likely there is some sort of network issue. Also, feel free to contact our support, as in some cases they may be able to say from logs on what operations are taking long times.

[pauliem]

I have logged a call - but I did not get an automated response, or an actual response. I will wait and hopefully one will come!

[Gostev]

You should get automated response immediately no matter what, make sure spam filter did not kill it.

[pauliem]

OK - I will have to wait for my spam report which comes through a little later.

I am just trying different variations on the job. I have excluded the large drive for now and backing up the other two drives in the system. Getting a very good rate of 60MB a second now with those two only. It seems to be a problem related to the large volume.

[Gostev]

Processing rate depends on specific virtual disk contents a lot, volumes with large amount of white (zeroed) blocks will backup much faster.

[pauliem]

Not getting very far with support. I've noticed that the backup really struggles once it gets past 1Tb, I reckon that I am only managing to backup about 100mb every ten minutes.

I really need to resolve this as quickly as possible. The two smaller drives backup no problem at all. I get the impression that the backup will never end - the backup file continues to grow in size even though the amount of processed data does not change.

It has taken 6 hours to do approx 200mb.

[m.novelli]

I had a similar issue when backupping a large Windows 2003 file server with Veeam 4.1.2
After updating to Veeam 5.0.0 the backup window decreased from 4h to 1h

Marco

[tsightler]

We are actually seeing very slow full backups of one of our large (~1.7TB) VM's as well, although not as bad as what your seeing, however, our system is made up of three individual virtual disks, the largest of which is 900GB, and it's the 900GB volume that is the slowest, so maybe if you have a single large virtual disk it might be even worse. That being said, we don't see much issue with incrementals, just full backups.

[pauliem]

The VM that I am having the problem with also has three drives. The largest one being the slowest.

So far I have:

Defragmented all of the drives.
Run Sdelete.
Migrated them to a different storage volume to shrink them.
Added a Windows XP VM so that I can use Virtual Appliance mode to backup. (Assigned 8 Cores)

All of the above does seem to have made things a little quicker, but not much. The small drive (c:) backs up like lightning, even when it is not incremental.

Still getting the very slow speeds. I am not sure what my next course of action should be because I don't have a good backup of this system. I am tempted to try an alternative product just to see if the speeds are any different, although I'd rather be using veeam as I am familiar with it. This is the only system

[pauliem]

To provide a comparison I ran two full backups, one containing only the system drive, and the other containing one of the drives that is slow to backup.

System drive (not slow)

Total VM size: 64.03 GB
Processed size: 64.03 GB
Processing rate: 39 MB/s
Backup mode: HOTADD/NBD with changed block tracking
Start time: 21/11/2010 13:50:32
End time: 21/11/2010 14:18:32
Duration: 0:27:59

Data Drive:

Total VM size: 400.01 GB
Processed size: 40.00 GB
Processing rate: 15 MB/s
Backup mode: HOTADD/NBD without changed block tracking
Start time: 21/11/2010 14:20:18
Time remaining: 07:07:49

I'm sure the data backup will continue to get gradually slower as it gets further through the process.

[Gostev]

Reading back through my response above, the content of virtual disk does directly affect its processing speed. For example, because of that, we do performance testing on disks full of random data (to make sure it does not have empty blocks, and compresses/dedupes really bad). So unless your disks are like that, there is no point in comparing processing speed between the two.

Do you have both disks sitting on the same LUN?

[pauliem]

I have just migrated the disk that was backing up fast on to the same array as the disks that are slow to backup, and it is also now slow to backup. So I presume there is some configuration problem with the array or disks.

It is getting around 13Mb/Sec. Presumably others can manage much faster speeds with SATA drives? or a RAID 10 SATA array in my case.

[Gostev]

Yes, I would expect it to be much faster (but please ignore processing speed in job stats, this is not throughtput, it is affected by other things).

Could you try adding another thick disk to the same VM temporarily, place this disk on SAS array, and try to copy some data from virtual disk located on SATA array to virtual disk located on SAS array within guest OS of this VM, and see what performance do you get. Then may by try playing RAID controller settings for SATA array (for example, try changing write-back to write-through), and retest. Or may be swap raid controller altogether?

[pauliem]

Hi - thanks for your continued help.

First of all,my message above was not accurate. The speed backing up the system drive ended up almost the same from the SATA array as the SAS array.

I just added a thick disk and copied a 9.3Gb Outlook PST from the SATA Array to the SAS array in 70 seconds - which seems pretty fast to me?
Now, this system has loads of these PST files on them, could they be a potential cause?

[Gostev]

Not really, specific content does not matter. What matter is number of empty (and, so, processed) blocks. You said that small disk backup speed up at the end - this is a sign to me that it has significant amount of empty blocks. Large disk may not have any. So while actual block processing rate is the same, small disk will exhibit higher "processing rate" (which, again, does not mean a thing for when comparing throughtput).

It would be interesting to compare throughtput using the log files for the job run when both disks were on the same LUN.

[tsightler]

This is very close to the issue we are seeing as well, although in our case we know the actual disk throughput is less when backing up this one system (we can produce the graphs from our storage monitoring solution). We don't really know why, but we don't have any other systems that are similar in size so we suspect it's something to do with the overall size of the VM, but it's just a guess. We also don't know if V4 had the issue because we had not run a full in a very long time until the switch to V5. It's mostly been a non-issue since it doesn't seem to impact incremental performance and we pretty much don't run full backups but I could probably manually run some fulls if it's worth looking into.

[pauliem]

Last night I tried running the VM Backup of the large volume only, ecxluding the rest of the VM. It is so dissapointing and I don't know what to try next. It is so much slower than traditional backup methods that it seems to be not fit for what we are trying to use it for:

Total VM size: 1.00 TB
Processed size: 542.72 GB
Processing rate: 10 MB/s
Backup mode: NBD with changed block tracking
Start time: 11/22/2010 5:44:14 PM
Time remaining: 13:22:33

The "processed size" is barely changing now.

Again, it started off with a reasonable processing speed. This drive has been defragged and had sdelete run and is only about half full. Somewhere around this stage I'd be expecting it to hit the part where it goes very quick but it isn't. I think the problem is with VBR but I have no way to prove this. I really must find a solution but what is the mechanism for getting someone to have a proper analysis of the problem?

[Alexey D.]

Pauliem,

It's very hard to guess why you are having this speed, it might be environmental.
These topics may give you a clue, just more information for reading:
Q: v5 Backup Speed, Job, CPU loading, Backup Time, etc,
v5 Backup Speed.

[pauliem]

As it is about 2 hours since my last post I thought I would post again showing the current slow progress:

Total size of VMs to backup: 1.00 TB
Processed size: 583.68 GB
Processing rate: 10 MB/s
Start time: 11/22/2010 5:43:20 PM
Time remaining: 13:03:40

I have sent the logs to support, hopefully they will come up with something.

The current situation really needs a support professional to diagnose the issue and not for me to keep trying different permutatiions of configurations. Is this level of support available from Veeam or am I on my own effectivly?

I don't beleive the problem is with the setup, but would be very happy to be proved wrong.

[Gostev]

Hi Paul, what is your support case number?

[pauliem]

Hi Gostev,

Thanks for your help, case number is 542274

Paul

[lewislampkin]

Hi - thanks for your continued help.

First of all,my message above was not accurate. The speed backing up the system drive ended up almost the same from the SATA array as the SAS array.

I just added a thick disk and copied a 9.3Gb Outlook PST from the SATA Array to the SAS array in 70 seconds - which seems pretty fast to me?
Now, this system has loads of these PST files on them, could they be a potential cause?

Are the PST's being accessed across the network? You do know that Microsoft does not support that, correct? As PST's are designed to access local storage, they're quite inefficient when accessed across the network. (I know, many people use them in this manner, including my own company! but doesn't make it right ...)

[Gostev]

Just to update (and Paul may know more from support by now - or not yet), but since this topic was bumped - lead dev mentioned to me briefly that he reviewed the logs and apparently the job for this problematic disk fails over to network processing mode (which we do not currently report happening in the session results, but will report in 5.0.1). So of course, backup speed over network will be much lower than in other modes.

[pauliem]

Lewis - The PST files are just archives of project emails. They are not actually used in an acitve way.

Gostev - Thanks for your continued help, at least that explains the reason why. I am getting this error in the log on the guest:

Event Type: Warning
Event Source: vmdebug
Event Category: None
Event ID: 3
Date: 21/11/2010
Time: 18:18:32
User: N/A
Description:
VMDebug driver (version 7.3.4.3) was not enabled. This driver is required by the replay debugging feature of VMware Workstation. If you are using other VMware products or not using replay debugging, please ignore this message.

Looks harmless, but is it helpful at all?

[Gostev]

Which VM? When do you get this error? Is it related to this discussion at all, or something completely different?

[pauliem]

The VM which has the large disk that is slow to backup. No, it may not be relevant - I was just checking.

Have not heard back from support but did finally get the backup to finish and then managed to do an incremental backup in a reasonable time frame. As part of my troubleshooting I ran sdelete which expanded the thin disk out to its full size. I would like to vmotion it to another data store and then back again so that it shrinks back to its original size. If I do this will the backup behave as it if it were a normal incremental or will it backup as if it were a new volume. I dont want to do it yet if I am going to hit the same speed problem.

Hopefully support will come back soon.

[TrevorBell]

Hi All,

Skimming through the post i thought i would post up my largest VM. The figures below are a first backup ( which i have always had at around 15-27 MB/s i just thought because its a large VM thats the normal kind of backup speed ) i think that speed is totally normal for the size of VM. I have 21gig VM on fast 15k sas storage and 400gig,300gig,250gig on slower sata storage ( EQL PS5000xv & EQL PS4000e )

7 of 7 files processed
Total VM size: 971.00 GB
Processed size: 971.00 GB
Processing rate: 27 MB/s
Backup mode: HOTADD/NBD with changed block tracking
Start time: 23/11/2010 20:12:55
End time: 24/11/2010 06:28:27
Duration: 10:15:31


For sure when i do incrementals i can get speeds around 400 MB/s or higher depending on the amount of changed data per day

Other smaller Vm`s from 12 gig to 100 gig backup first pass at around 30-70 MB/s

hope that helps

Trev

[slidek9]

Hi Guys,

I am seeing the same when backing up 1TB+ volumes.

It starts off really fast 40MB/s and then as it approaches 1TB barrier it slows down to about 5MB/s.
I can duplicate this with different VM's. Its all office type archive data.

It's going to get messy here when I have to start backing up our 1TB+ SQL databaes with high I/O

[Gostev]

Hi, the issue above was due to the job quietly failing over to network processing mode on large VMDKs. I recommend that you open a support case to confirm if you are having the same issue. Thanks!