Discussions specific to the VMware vSphere hypervisor
jcsd
Influencer
Posts: 11
Liked: never
Joined: Dec 13, 2011 3:26 pm
Full Name: Wesley Bresson
Contact:

Full backup speed decreases as it runs

Post by jcsd » Jan 11, 2012 2:47 pm

Anyone else having this issue? I have a ticket open on it, 45381354

Image

Notice how it starts off high and then drops as it runs, this is with Veeam 6, 5 did the same thing. This particular image a SAN mode job, virtual applicance does the same thing. This is a 1.7TB drive that is getting backed up from a 50 drive pool in a EMC SAN to a 12 disk array.

Our previous backup product (file based) took 5 hours and 20 minutes to backup this drive, from the same source array. Veeam runs for over 13 hours before I cancel it. In the image above I canceled the job, it was only 50 some percent done. It does the same thing on other large VM's. Smaller VM's seem to have the same behaviour maybe, it just finishes them before they get too low in speed.

Gostev
SVP, Product Management
Posts: 24169
Liked: 3295 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Full backup speed decreases as it runs

Post by Gostev » Jan 11, 2012 3:39 pm

Please let the job finish and include the 4 bottleneck statistics numbers. Thanks!

tsightler
VP, Product Management
Posts: 5328
Liked: 2175 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Full backup speed decreases as it runs

Post by tsightler » Jan 11, 2012 6:13 pm

What is the hardware of the server that is running this job? What is the CPU load while this is occurring?

Gostev
SVP, Product Management
Posts: 24169
Liked: 3295 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Full backup speed decreases as it runs

Post by Gostev » Jan 11, 2012 6:43 pm

Bottleneck stats numbers will tell ;)

tsightler
VP, Product Management
Posts: 5328
Liked: 2175 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Full backup speed decreases as it runs

Post by tsightler » Jan 11, 2012 6:53 pm

Agreed, but I didn't ask for that because you already did so I just asked for information that will help interpret what the bottleneck information highlights.

jcsd
Influencer
Posts: 11
Liked: never
Joined: Dec 13, 2011 3:26 pm
Full Name: Wesley Bresson
Contact:

Re: Full backup speed decreases as it runs

Post by jcsd » Jan 11, 2012 7:37 pm

In SAN mode the backup proxy is a dual quad Intel E5506. I just started the job back up and checked the proxy after it started transfering and its running somewhere between 40-50-60-70-80%, I saw in hit the 90+% once. As the job slows down though the CPU usage drops way down.

On the job I cancelled this morning it says source 2%, proxy 56%, network 99%, target 98%.
In a different instance of this job it says source 21% proxy 72% network 97% target 73%.

In the above two instances the proxy is a physcial server in the same site as the source storage. The destination storage is in a different site connected to it by a 2GB (2x1GB) fiber link.


Using a different machine as the proxy I got source 65%, proxy 56%, network 49%, target 50%. That was using the server connected to the destination storage as the proxy. It is a dual quad core as well.

The network between the two sites I have tested at 90+% of a 1GB connection. At that time when I was testing it was between the virtual appliance proxy and the destination backup server. Right now I just started the job 30 minutes or so ago so it is during the faster part of the backup and it's only using 30-50% of the 1GB.

This is ESXi 5 by the way, 8MB block datastores. I have 3 servers this size and larger that I can't get to backup at a decent speed. This particular server I can't get to backup in one weekend let alone the 5 hours it used to back up in (without the verify). From the same source storage and server, through the same network connection, to the same server, a different array in the image above. It's running right now, going to the same storage this time as the job that takes 5 hours (without the verify) on our old backup product. I have tested that before (in virtual appliance mode) I got the same decreasing speed over time.

I've tried running the job with no compression, no deduplication, no verify and got the same results. Upgraded to ESXi 5 from 4 and still have the same results. Upgraded to VMFS 5 and am still getting the same results. Tried a thick disk instead of eager thick and am getting the same results. Tried without multipathing and got the same results. The picture above is without multipating. A single 1GB link to the SAN active.

tsightler
VP, Product Management
Posts: 5328
Liked: 2175 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Full backup speed decreases as it runs

Post by tsightler » Jan 11, 2012 9:01 pm

jcsd wrote:In SAN mode the backup proxy is a dual quad Intel E5506. I just started the job back up and checked the proxy after it started transfering and its running somewhere between 40-50-60-70-80%, I saw in hit the 90+% once. As the job slows down though the CPU usage drops way down.
I would be very interested in the CPU usage when the backup is slow.
jcsd wrote: On the job I cancelled this morning it says source 2%, proxy 56%, network 99%, target 98%.
In a different instance of this job it says source 21% proxy 72% network 97% target 73%.
Interesting, so this would imply that the problem is all about the target storage. Have you monitored memory usage on the server during the backup, specifically lazy write cache and paging? I'm wondering if your having the same issues as http://forums.veeam.com/viewtopic.php?f=2&t=8999 where backups start off fast, but as the write cache fills up and uses all of the memory the Windows machine slows to a crawl because it actually starts paging more and more. Normally this happens when the target is slower than the source.

Just to make sure I understand, when you are saying your destination is at a different site, are you using a repository at that site? How is the file being written to the remote site, via CIFS? What is the latency.

Have you tried simply copying a very large file?

Just looking at the information you have posted, it would seem that the storage at the remote site is "slow" (it may not be the storage itself, it may be the combination of protocol/latency if you're writing via CIFS). Please be specific as to the target storage and the how you are writing to it.

jcsd
Influencer
Posts: 11
Liked: never
Joined: Dec 13, 2011 3:26 pm
Full Name: Wesley Bresson
Contact:

Re: Full backup speed decreases as it runs

Post by jcsd » Jan 13, 2012 8:56 pm

The job failed after 48 hours (patch 2 hasn't been installed yet). The reported load was source 48%, proxy 55%, network 55%, target 50%.

The destination storage is local to the Veeam install, it isn't connecting via CIFS/SMB. It must be using the VeeamAgent which on the destination storage that process was using almost all of one core of processor. Only ~6% of the total server processor power but almost maxing out one core (one hyperthreaded core anyways)

The destination storage should actually be faster than the source when the source is only using 1x1GB connection. The destination is a local (to the Veeam install) array of 12 disks in a raid 5. The most the 1GB connection can do is 120MB some. I just tested the destination storage at around 130MB-150MB/sec seq. write.

The job did drop the free RAM on the destination storage server to 0, this server has 32GB of RAM. Most of that looked like it was used in the cache. The job isn't running now and still there is only about 350MB free.

Our previous backup product ran the job on this same server, coming from the same source, going to the same destination, and it didnt' exhibit this downward speed curve (or take nearly as long to run).

The destination server is a Windows 2008 64 bit. The proxy is a Windows 2008 R2.

I forgot to mention that after the job slowed down, when I checked it yesterday, the CPU load on the proxy was less than 10%..

Actually, the proxy I have been testing with lately is a Windows 2003 32 bit. The backup destination is a 2008 64 bit. I put the Windows Dynamic Cache service (and Veeam 6 path 2) on the destination and am getting the same result.

jcsd
Influencer
Posts: 11
Liked: never
Joined: Dec 13, 2011 3:26 pm
Full Name: Wesley Bresson
Contact:

Re: Full backup speed decreases as it runs

Post by jcsd » Jan 19, 2012 2:18 pm

See the screenshots below, these screenshots are from the slow throughput part of the backup. I've switched to a Windows 2003 32 bit destination and the backups are running faster now but still getting progressively slower as they run. Right now my guess is that the Veeamagent.exe on the destination storage server is the cause. When the backup job starts out on the larger drive of this VM (at 99% of 1GB throughput) the Veeamagent.exe process running at 3-4-5% CPU. Towards the end of the job when the throughput has slowed way down the Veeamagent.exe is using almost all of one core of CPU power. See the screenshot below, the second core from the right must be the Veeamagent.exe running. For some reason it isn't spreading across the cores and for some reason it's CPU use seems to be a lot higher when the job is slow compared to when it is fast.

Can you have someone there test a VM with 1.5TB of data on it backed up to a Windows destination with the Veeamagent?

Image
Image
Image

Karl
Novice
Posts: 4
Liked: never
Joined: Jan 20, 2012 10:33 am
Full Name: Karl
Contact:

Backup job slows down in the middle of the bacup.

Post by Karl » Jan 20, 2012 11:03 am

[merged]

I have been evaluating Veeam backup v. 6 on my client pc for some days. All have been running fine, so we ordered a license, and installed Veeam on another pc (Dell PowerEdge T100).

I then set up the backup job on the new pc, and started a full backup of 1 VM on VMWare ESXi 4.1 (VM=Windows 2008 R2, disk=600GB). I am using reversed incremental backup, like I was doing when evaluating Veeam on my client pc.

When starting the full backup job, all is well at the start. It uses almost 100% of the processor, and have a good network thrughput, but after backing up about 250GB everyting slows down to almost a complete stop. The processor is almost idle, and almost no network traffic anymore. If I remember correctly, a full backup used to take about 4 hours on my client pc (when evaluating). Today I stopped the full backup after more than 11 hours, as it was only at 70% finished. Tried a new full backup, and get the same result. All is fine for the first ~200GB, but then it slows down to almost a complete stop.

Our setup:
Server:
VMWare ESXi 4.1 on HP ML 350 G6 server. 4 cores, and running 5 virtual machines (we only backup 1 of the VM's), 38 GB ram.
This server is not heavily loaded. We run 5 VM's, but most of them are almost idle. 1 VM is our developement server, where we have our source code repository, and a Pervasive.SQL database used for developement, so it is not used that much.

Backup pc:
Dell Poweredge T100. Xeon processor with 4 cores, and 4GB ram. 1GB NIC.
Windows 7 professional, 64 bit.
Veeam backup & replication 6.

Backup repository:
External USB3 disk. The same disk, and USB3 controller that I used on my client machine when evaluating Veeam (I moved it yesterday).

Any idea what is happening ?

TIA
Karl

Karl
Novice
Posts: 4
Liked: never
Joined: Jan 20, 2012 10:33 am
Full Name: Karl
Contact:

Re: Full backup speed decreases as it runs

Post by Karl » Jan 20, 2012 2:32 pm

A little info update. From the last backup I cancelled, I can see that the performance info on the job is the following:
Source=98%
Proxy=54%
Network=1%
Target=0%

jcsd
Influencer
Posts: 11
Liked: never
Joined: Dec 13, 2011 3:26 pm
Full Name: Wesley Bresson
Contact:

Re: Full backup speed decreases as it runs

Post by jcsd » Jan 20, 2012 2:48 pm

So it worked ok during evaulation but now it doesn't Karl? Was the destination storage connected to a Windows machine in both cases? Is it the the same OS version?

Karl
Novice
Posts: 4
Liked: never
Joined: Jan 20, 2012 10:33 am
Full Name: Karl
Contact:

Re: Full backup speed decreases as it runs

Post by Karl » Jan 20, 2012 8:12 pm

Yes it worked fine during evaluation, and the destination storage was connected to Windows 7 64 bit professional in both cases (but different machines).
It looks as I got this problem sorted out now, by following the "How to fix broken CBT". The backup is now at 94 % and still running fine.
Thanks for answering! I am new to Veeam, so that may be the explanation:)

mbutch
Novice
Posts: 3
Liked: never
Joined: Jan 22, 2012 4:55 pm
Full Name: Matthew Butch
Contact:

Re: Full backup speed decreases as it runs

Post by mbutch » Jan 22, 2012 6:05 pm

I've having this exact issue.

It starts at 70MBps, then drops down to 15MBps.

I have one Veeam Server connected to two EqualLogic boxes. Manually copying files works fine.

Karl
Novice
Posts: 4
Liked: never
Joined: Jan 20, 2012 10:33 am
Full Name: Karl
Contact:

Re: Full backup speed decreases as it runs

Post by Karl » Jan 23, 2012 8:47 am

Have you tried the "solution" "How to fix broken CBT" at the bottom of this page:
Summary of known v6 issues

This fixed it for me.

jcsd
Influencer
Posts: 11
Liked: never
Joined: Dec 13, 2011 3:26 pm
Full Name: Wesley Bresson
Contact:

Re: Full backup speed decreases as it runs

Post by jcsd » Jan 23, 2012 6:03 pm

If I use a Linux target instead of a Windows target the job runs correctly without the decreasing speed. I would perfer to use a Winodws target though.

Gostev
SVP, Product Management
Posts: 24169
Liked: 3295 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Full backup speed decreases as it runs

Post by Gostev » Jan 23, 2012 9:13 pm

Hmm, I would not expect the target type to matter, because the data processing agent is cross-platform (same code). What really matters is the performance of storage you are writing to. However, if both Linux and Windows server have exactly identical hardware, then I would say you should investigate this with our support. Thanks!

jcsd
Influencer
Posts: 11
Liked: never
Joined: Dec 13, 2011 3:26 pm
Full Name: Wesley Bresson
Contact:

Re: Full backup speed decreases as it runs

Post by jcsd » Jan 24, 2012 4:38 pm

same hardware, I removed Windows and put Linux on it

mbutch
Novice
Posts: 3
Liked: never
Joined: Jan 22, 2012 4:55 pm
Full Name: Matthew Butch
Contact:

Re: Full backup speed decreases as it runs

Post by mbutch » Jan 24, 2012 10:01 pm

Wow, it must be something with the CBT. I followed those instructions, and after 40 minutes I'm still at 84MBps. That's awesome.

What is not is the fact that I know how to do this to each VM.

jcsd
Influencer
Posts: 11
Liked: never
Joined: Dec 13, 2011 3:26 pm
Full Name: Wesley Bresson
Contact:

Re: Full backup speed decreases as it runs

Post by jcsd » Jan 25, 2012 2:41 pm

In my case, these jobs are a full backup, CBT shouldn't be affecting them, I don't think?

tsightler
VP, Product Management
Posts: 5328
Liked: 2175 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Full backup speed decreases as it runs

Post by tsightler » Jan 25, 2012 2:49 pm

Actually Veeam still attempts to use CBT for full backups to detect unused areas of the disk, etc. Also, even during a full backup the snapshot CBT file has to be updated so, while it does feel like a long shot, it might be worth a try since other seem to consistently see improvements.

jcsd
Influencer
Posts: 11
Liked: never
Joined: Dec 13, 2011 3:26 pm
Full Name: Wesley Bresson
Contact:

Re: Full backup speed decreases as it runs

Post by jcsd » Jan 25, 2012 2:52 pm

tsightler wrote:Actually Veeam still attempts to use CBT for full backups to detect unused areas of the disk, etc. Also, even during a full backup the snapshot CBT file has to be updated so, while it does feel like a long shot, it might be worth a try since other seem to consistently see improvements.
All I did was switch the destination storage from Windows to Linux and the problem went away, so CBT wouldn't be the issue than right?

tsightler
VP, Product Management
Posts: 5328
Liked: 2175 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Full backup speed decreases as it runs

Post by tsightler » Jan 25, 2012 4:33 pm

I would generally agree with that statement, but seeing multiple users report that it helped them make it worth it even as a long shot (perhaps a very long shot).

wahrheit2004
Influencer
Posts: 10
Liked: 2 times
Joined: Feb 27, 2012 7:33 am
Full Name: DW
Contact:

Re: Full backup speed decreases as it runs

Post by wahrheit2004 » Feb 27, 2012 7:47 am

Hi,

I have the same issue with direct SAN (Network Failback disabled) Veeam B&R 6.0 Patch 3 and KB1113 (How to reset CBT) fix.
I am Backing up a 4,9TB VM running "full backup" (Win 2008 R2 with 13 virtual Disks). SAN is connected by 2x 10GBit/s iSCSI (1 per controller), Backup Machine is
Physical Server Win2008 R2 with Backup proxy, accessing via MPIO iSCSI connected by 2x 10GBit/s to SAN. Server has 32GB RAM, 2x 4 Core XEON 2,66 Ghz.
Backup ist stored to DAS (16x 1TB SATA RAID 6). If I copy files from SAN to DAS I have 650MB/s.

Realtimestatistics says Bottleneck "target" but CPU is only used @ 1core and here 80% kernel time.
Current backup speed is slow 15MB/s.

Backing up a 2nd VM (Win2008, 1HDD, 167GB) as "full backup" took "just" 10 minutes and 2 sec.

I opened a Veeam Level 3 ticket @ ID 5175226 and sent them my logs.

Gostev
SVP, Product Management
Posts: 24169
Liked: 3295 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Full backup speed decreases as it runs

Post by Gostev » Feb 27, 2012 9:01 am

wahrheit2004 wrote:Realtimestatistics says Bottleneck "target" but CPU is only used @ 1core and here 80% kernel time.
Actually, "target" means the bottleneck is the target storage write speed. Nothing to deal with CPU load.

wahrheit2004
Influencer
Posts: 10
Liked: 2 times
Joined: Feb 27, 2012 7:33 am
Full Name: DW
Contact:

Re: Full backup speed decreases as it runs

Post by wahrheit2004 » Feb 28, 2012 7:18 am

Yes and no. what I did to verify is to copy a 4.3GB DVD ISO Image to the same lokal Volume Veeam uses for it's backups.
Copying the .ISO from a remote LAN_Server with a 1GBit/s LAN Card I reach 60MB/s write.
If I duplicate it from local volume to local volume my speed is 46MB/s in write, handling read and write I/O from same Volume.

Btw.:
From my backup-job above the Real-Time statistics show 57% @ 10MB/s - Duration 45hrs 47min currently.

Gostev
SVP, Product Management
Posts: 24169
Liked: 3295 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Full backup speed decreases as it runs

Post by Gostev » Feb 28, 2012 9:34 am

Not sure what you mean by "yes and no". Regarding your test, you said your backup speed decreases with time, backing up very large files. So copying small 4.3GB ISO file won't really show anything?

Anyway, as far as I know your support case is being looked at.

wahrheit2004
Influencer
Posts: 10
Liked: 2 times
Joined: Feb 27, 2012 7:33 am
Full Name: DW
Contact:

Re: Full backup speed decreases as it runs

Post by wahrheit2004 » Feb 29, 2012 7:19 am

Hi Gostev,
Gostev wrote:So copying small 4.3GB ISO file won't really show anything?
I generated a 2,19 TB File in the volume of the Veeambackupfiles by using this command:
fsutil file createnew test.txt 2242880000000

Than I copied this file in same folder to "test - Kopie.txt" using windows Explorer.
Detail of copy process shows 2xxMB/s and Ressource manager shows 290MB/s in peeks up to 3xxMB/s.

Image

My 4,9 TB Veeam backup process is at 72% @ 8MB/s duration 69hrs 44 min...
Herefor I have a webex meeting tomorrow, hope we can find the issue.

wahrheit2004
Influencer
Posts: 10
Liked: 2 times
Joined: Feb 27, 2012 7:33 am
Full Name: DW
Contact:

Re: Full backup speed decreases as it runs

Post by wahrheit2004 » Mar 01, 2012 1:39 pm

Hi everyone,

I had the webex meeting today with veeam support. They identified it that is a issue deep in their B&R software.
Now developers should take it for deeper investigation and hopefully a fix soon.

What you can see ist, that the 1st HDDs are processed at a good speed, but than dramatically drop - see here:
Image

This is by running a full-backup. I hope that the reverse-incrmental will be processed a lot faster :D .

So waiting for fix 4 or 5....

Gostev
SVP, Product Management
Posts: 24169
Liked: 3295 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Full backup speed decreases as it runs

Post by Gostev » Mar 02, 2012 9:30 am

wahrheit2004 wrote:They identified it that is a issue deep in their B&R software.
News to me! I am on the same email thread between support and devs regarding the webex with you (already saw this picture above). At this time, the opinion is nothing but speculation of a specific support engineer. Multiple tests on very large VMs in our own labs could not reproduce this behavior, however it is true however that developers will take it for deeper investigation ;)

Post Reply

Who is online

Users browsing this forum: No registered users and 6 guests