Full backup speed decreases as it runs

VMware specific discussions

Full backup speed decreases as it runs

Veeam Logoby jcsd » Wed Jan 11, 2012 2:47 pm

Anyone else having this issue? I have a ticket open on it, 45381354

Image

Notice how it starts off high and then drops as it runs, this is with Veeam 6, 5 did the same thing. This particular image a SAN mode job, virtual applicance does the same thing. This is a 1.7TB drive that is getting backed up from a 50 drive pool in a EMC SAN to a 12 disk array.

Our previous backup product (file based) took 5 hours and 20 minutes to backup this drive, from the same source array. Veeam runs for over 13 hours before I cancel it. In the image above I canceled the job, it was only 50 some percent done. It does the same thing on other large VM's. Smaller VM's seem to have the same behaviour maybe, it just finishes them before they get too low in speed.
jcsd
Influencer
 
Posts: 11
Liked: never
Joined: Tue Dec 13, 2011 3:26 pm
Full Name: Wesley Bresson

Re: Full backup speed decreases as it runs

Veeam Logoby Gostev » Wed Jan 11, 2012 3:39 pm

Please let the job finish and include the 4 bottleneck statistics numbers. Thanks!
Gostev
Veeam Software
 
Posts: 21390
Liked: 2349 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Full backup speed decreases as it runs

Veeam Logoby tsightler » Wed Jan 11, 2012 6:13 pm

What is the hardware of the server that is running this job? What is the CPU load while this is occurring?
tsightler
Veeam Software
 
Posts: 4768
Liked: 1737 times
Joined: Fri Jun 05, 2009 12:57 pm
Full Name: Tom Sightler

Re: Full backup speed decreases as it runs

Veeam Logoby Gostev » Wed Jan 11, 2012 6:43 pm

Bottleneck stats numbers will tell ;)
Gostev
Veeam Software
 
Posts: 21390
Liked: 2349 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: Full backup speed decreases as it runs

Veeam Logoby tsightler » Wed Jan 11, 2012 6:53 pm

Agreed, but I didn't ask for that because you already did so I just asked for information that will help interpret what the bottleneck information highlights.
tsightler
Veeam Software
 
Posts: 4768
Liked: 1737 times
Joined: Fri Jun 05, 2009 12:57 pm
Full Name: Tom Sightler

Re: Full backup speed decreases as it runs

Veeam Logoby jcsd » Wed Jan 11, 2012 7:37 pm

In SAN mode the backup proxy is a dual quad Intel E5506. I just started the job back up and checked the proxy after it started transfering and its running somewhere between 40-50-60-70-80%, I saw in hit the 90+% once. As the job slows down though the CPU usage drops way down.

On the job I cancelled this morning it says source 2%, proxy 56%, network 99%, target 98%.
In a different instance of this job it says source 21% proxy 72% network 97% target 73%.

In the above two instances the proxy is a physcial server in the same site as the source storage. The destination storage is in a different site connected to it by a 2GB (2x1GB) fiber link.


Using a different machine as the proxy I got source 65%, proxy 56%, network 49%, target 50%. That was using the server connected to the destination storage as the proxy. It is a dual quad core as well.

The network between the two sites I have tested at 90+% of a 1GB connection. At that time when I was testing it was between the virtual appliance proxy and the destination backup server. Right now I just started the job 30 minutes or so ago so it is during the faster part of the backup and it's only using 30-50% of the 1GB.

This is ESXi 5 by the way, 8MB block datastores. I have 3 servers this size and larger that I can't get to backup at a decent speed. This particular server I can't get to backup in one weekend let alone the 5 hours it used to back up in (without the verify). From the same source storage and server, through the same network connection, to the same server, a different array in the image above. It's running right now, going to the same storage this time as the job that takes 5 hours (without the verify) on our old backup product. I have tested that before (in virtual appliance mode) I got the same decreasing speed over time.

I've tried running the job with no compression, no deduplication, no verify and got the same results. Upgraded to ESXi 5 from 4 and still have the same results. Upgraded to VMFS 5 and am still getting the same results. Tried a thick disk instead of eager thick and am getting the same results. Tried without multipathing and got the same results. The picture above is without multipating. A single 1GB link to the SAN active.
jcsd
Influencer
 
Posts: 11
Liked: never
Joined: Tue Dec 13, 2011 3:26 pm
Full Name: Wesley Bresson

Re: Full backup speed decreases as it runs

Veeam Logoby tsightler » Wed Jan 11, 2012 9:01 pm

jcsd wrote:In SAN mode the backup proxy is a dual quad Intel E5506. I just started the job back up and checked the proxy after it started transfering and its running somewhere between 40-50-60-70-80%, I saw in hit the 90+% once. As the job slows down though the CPU usage drops way down.


I would be very interested in the CPU usage when the backup is slow.

jcsd wrote:On the job I cancelled this morning it says source 2%, proxy 56%, network 99%, target 98%.
In a different instance of this job it says source 21% proxy 72% network 97% target 73%.


Interesting, so this would imply that the problem is all about the target storage. Have you monitored memory usage on the server during the backup, specifically lazy write cache and paging? I'm wondering if your having the same issues as viewtopic.php?f=2&t=8999 where backups start off fast, but as the write cache fills up and uses all of the memory the Windows machine slows to a crawl because it actually starts paging more and more. Normally this happens when the target is slower than the source.

Just to make sure I understand, when you are saying your destination is at a different site, are you using a repository at that site? How is the file being written to the remote site, via CIFS? What is the latency.

Have you tried simply copying a very large file?

Just looking at the information you have posted, it would seem that the storage at the remote site is "slow" (it may not be the storage itself, it may be the combination of protocol/latency if you're writing via CIFS). Please be specific as to the target storage and the how you are writing to it.
tsightler
Veeam Software
 
Posts: 4768
Liked: 1737 times
Joined: Fri Jun 05, 2009 12:57 pm
Full Name: Tom Sightler

Re: Full backup speed decreases as it runs

Veeam Logoby jcsd » Fri Jan 13, 2012 8:56 pm

The job failed after 48 hours (patch 2 hasn't been installed yet). The reported load was source 48%, proxy 55%, network 55%, target 50%.

The destination storage is local to the Veeam install, it isn't connecting via CIFS/SMB. It must be using the VeeamAgent which on the destination storage that process was using almost all of one core of processor. Only ~6% of the total server processor power but almost maxing out one core (one hyperthreaded core anyways)

The destination storage should actually be faster than the source when the source is only using 1x1GB connection. The destination is a local (to the Veeam install) array of 12 disks in a raid 5. The most the 1GB connection can do is 120MB some. I just tested the destination storage at around 130MB-150MB/sec seq. write.

The job did drop the free RAM on the destination storage server to 0, this server has 32GB of RAM. Most of that looked like it was used in the cache. The job isn't running now and still there is only about 350MB free.

Our previous backup product ran the job on this same server, coming from the same source, going to the same destination, and it didnt' exhibit this downward speed curve (or take nearly as long to run).

The destination server is a Windows 2008 64 bit. The proxy is a Windows 2008 R2.

I forgot to mention that after the job slowed down, when I checked it yesterday, the CPU load on the proxy was less than 10%..

Actually, the proxy I have been testing with lately is a Windows 2003 32 bit. The backup destination is a 2008 64 bit. I put the Windows Dynamic Cache service (and Veeam 6 path 2) on the destination and am getting the same result.
jcsd
Influencer
 
Posts: 11
Liked: never
Joined: Tue Dec 13, 2011 3:26 pm
Full Name: Wesley Bresson

Re: Full backup speed decreases as it runs

Veeam Logoby jcsd » Thu Jan 19, 2012 2:18 pm

See the screenshots below, these screenshots are from the slow throughput part of the backup. I've switched to a Windows 2003 32 bit destination and the backups are running faster now but still getting progressively slower as they run. Right now my guess is that the Veeamagent.exe on the destination storage server is the cause. When the backup job starts out on the larger drive of this VM (at 99% of 1GB throughput) the Veeamagent.exe process running at 3-4-5% CPU. Towards the end of the job when the throughput has slowed way down the Veeamagent.exe is using almost all of one core of CPU power. See the screenshot below, the second core from the right must be the Veeamagent.exe running. For some reason it isn't spreading across the cores and for some reason it's CPU use seems to be a lot higher when the job is slow compared to when it is fast.

Can you have someone there test a VM with 1.5TB of data on it backed up to a Windows destination with the Veeamagent?

Image
Image
Image
jcsd
Influencer
 
Posts: 11
Liked: never
Joined: Tue Dec 13, 2011 3:26 pm
Full Name: Wesley Bresson

Backup job slows down in the middle of the bacup.

Veeam Logoby Karl » Fri Jan 20, 2012 11:03 am

[merged]

I have been evaluating Veeam backup v. 6 on my client pc for some days. All have been running fine, so we ordered a license, and installed Veeam on another pc (Dell PowerEdge T100).

I then set up the backup job on the new pc, and started a full backup of 1 VM on VMWare ESXi 4.1 (VM=Windows 2008 R2, disk=600GB). I am using reversed incremental backup, like I was doing when evaluating Veeam on my client pc.

When starting the full backup job, all is well at the start. It uses almost 100% of the processor, and have a good network thrughput, but after backing up about 250GB everyting slows down to almost a complete stop. The processor is almost idle, and almost no network traffic anymore. If I remember correctly, a full backup used to take about 4 hours on my client pc (when evaluating). Today I stopped the full backup after more than 11 hours, as it was only at 70% finished. Tried a new full backup, and get the same result. All is fine for the first ~200GB, but then it slows down to almost a complete stop.

Our setup:
Server:
VMWare ESXi 4.1 on HP ML 350 G6 server. 4 cores, and running 5 virtual machines (we only backup 1 of the VM's), 38 GB ram.
This server is not heavily loaded. We run 5 VM's, but most of them are almost idle. 1 VM is our developement server, where we have our source code repository, and a Pervasive.SQL database used for developement, so it is not used that much.

Backup pc:
Dell Poweredge T100. Xeon processor with 4 cores, and 4GB ram. 1GB NIC.
Windows 7 professional, 64 bit.
Veeam backup & replication 6.

Backup repository:
External USB3 disk. The same disk, and USB3 controller that I used on my client machine when evaluating Veeam (I moved it yesterday).

Any idea what is happening ?

TIA
Karl
Karl
Novice
 
Posts: 4
Liked: never
Joined: Fri Jan 20, 2012 10:33 am
Full Name: Karl

Re: Full backup speed decreases as it runs

Veeam Logoby Karl » Fri Jan 20, 2012 2:32 pm

A little info update. From the last backup I cancelled, I can see that the performance info on the job is the following:
Source=98%
Proxy=54%
Network=1%
Target=0%
Karl
Novice
 
Posts: 4
Liked: never
Joined: Fri Jan 20, 2012 10:33 am
Full Name: Karl

Re: Full backup speed decreases as it runs

Veeam Logoby jcsd » Fri Jan 20, 2012 2:48 pm

So it worked ok during evaulation but now it doesn't Karl? Was the destination storage connected to a Windows machine in both cases? Is it the the same OS version?
jcsd
Influencer
 
Posts: 11
Liked: never
Joined: Tue Dec 13, 2011 3:26 pm
Full Name: Wesley Bresson

Re: Full backup speed decreases as it runs

Veeam Logoby Karl » Fri Jan 20, 2012 8:12 pm

Yes it worked fine during evaluation, and the destination storage was connected to Windows 7 64 bit professional in both cases (but different machines).
It looks as I got this problem sorted out now, by following the "How to fix broken CBT". The backup is now at 94 % and still running fine.
Thanks for answering! I am new to Veeam, so that may be the explanation:)
Karl
Novice
 
Posts: 4
Liked: never
Joined: Fri Jan 20, 2012 10:33 am
Full Name: Karl

Re: Full backup speed decreases as it runs

Veeam Logoby mbutch » Sun Jan 22, 2012 6:05 pm

I've having this exact issue.

It starts at 70MBps, then drops down to 15MBps.

I have one Veeam Server connected to two EqualLogic boxes. Manually copying files works fine.
mbutch
Novice
 
Posts: 3
Liked: never
Joined: Sun Jan 22, 2012 4:55 pm
Full Name: Matthew Butch

Re: Full backup speed decreases as it runs

Veeam Logoby Karl » Mon Jan 23, 2012 8:47 am

Have you tried the "solution" "How to fix broken CBT" at the bottom of this page:
Summary of known v6 issues

This fixed it for me.
Karl
Novice
 
Posts: 4
Liked: never
Joined: Fri Jan 20, 2012 10:33 am
Full Name: Karl

Next

Return to VMware vSphere



Who is online

Users browsing this forum: No registered users and 18 guests