Discussions specific to the VMware vSphere hypervisor
jcsd
Influencer
Posts: 11
Liked: never
Joined: Dec 13, 2011 3:26 pm
Full Name: Wesley Bresson
Contact:

Re: Full backup speed decreases as it runs

Post by jcsd » Jan 23, 2012 6:03 pm

If I use a Linux target instead of a Windows target the job runs correctly without the decreasing speed. I would perfer to use a Winodws target though.

Gostev
SVP, Product Management
Posts: 24014
Liked: 3250 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Full backup speed decreases as it runs

Post by Gostev » Jan 23, 2012 9:13 pm

Hmm, I would not expect the target type to matter, because the data processing agent is cross-platform (same code). What really matters is the performance of storage you are writing to. However, if both Linux and Windows server have exactly identical hardware, then I would say you should investigate this with our support. Thanks!

jcsd
Influencer
Posts: 11
Liked: never
Joined: Dec 13, 2011 3:26 pm
Full Name: Wesley Bresson
Contact:

Re: Full backup speed decreases as it runs

Post by jcsd » Jan 24, 2012 4:38 pm

same hardware, I removed Windows and put Linux on it

mbutch
Novice
Posts: 3
Liked: never
Joined: Jan 22, 2012 4:55 pm
Full Name: Matthew Butch
Contact:

Re: Full backup speed decreases as it runs

Post by mbutch » Jan 24, 2012 10:01 pm

Wow, it must be something with the CBT. I followed those instructions, and after 40 minutes I'm still at 84MBps. That's awesome.

What is not is the fact that I know how to do this to each VM.

jcsd
Influencer
Posts: 11
Liked: never
Joined: Dec 13, 2011 3:26 pm
Full Name: Wesley Bresson
Contact:

Re: Full backup speed decreases as it runs

Post by jcsd » Jan 25, 2012 2:41 pm

In my case, these jobs are a full backup, CBT shouldn't be affecting them, I don't think?

tsightler
VP, Product Management
Posts: 5290
Liked: 2144 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Full backup speed decreases as it runs

Post by tsightler » Jan 25, 2012 2:49 pm

Actually Veeam still attempts to use CBT for full backups to detect unused areas of the disk, etc. Also, even during a full backup the snapshot CBT file has to be updated so, while it does feel like a long shot, it might be worth a try since other seem to consistently see improvements.

jcsd
Influencer
Posts: 11
Liked: never
Joined: Dec 13, 2011 3:26 pm
Full Name: Wesley Bresson
Contact:

Re: Full backup speed decreases as it runs

Post by jcsd » Jan 25, 2012 2:52 pm

tsightler wrote:Actually Veeam still attempts to use CBT for full backups to detect unused areas of the disk, etc. Also, even during a full backup the snapshot CBT file has to be updated so, while it does feel like a long shot, it might be worth a try since other seem to consistently see improvements.
All I did was switch the destination storage from Windows to Linux and the problem went away, so CBT wouldn't be the issue than right?

tsightler
VP, Product Management
Posts: 5290
Liked: 2144 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Full backup speed decreases as it runs

Post by tsightler » Jan 25, 2012 4:33 pm

I would generally agree with that statement, but seeing multiple users report that it helped them make it worth it even as a long shot (perhaps a very long shot).

wahrheit2004
Influencer
Posts: 10
Liked: 2 times
Joined: Feb 27, 2012 7:33 am
Full Name: DW
Contact:

Re: Full backup speed decreases as it runs

Post by wahrheit2004 » Feb 27, 2012 7:47 am

Hi,

I have the same issue with direct SAN (Network Failback disabled) Veeam B&R 6.0 Patch 3 and KB1113 (How to reset CBT) fix.
I am Backing up a 4,9TB VM running "full backup" (Win 2008 R2 with 13 virtual Disks). SAN is connected by 2x 10GBit/s iSCSI (1 per controller), Backup Machine is
Physical Server Win2008 R2 with Backup proxy, accessing via MPIO iSCSI connected by 2x 10GBit/s to SAN. Server has 32GB RAM, 2x 4 Core XEON 2,66 Ghz.
Backup ist stored to DAS (16x 1TB SATA RAID 6). If I copy files from SAN to DAS I have 650MB/s.

Realtimestatistics says Bottleneck "target" but CPU is only used @ 1core and here 80% kernel time.
Current backup speed is slow 15MB/s.

Backing up a 2nd VM (Win2008, 1HDD, 167GB) as "full backup" took "just" 10 minutes and 2 sec.

I opened a Veeam Level 3 ticket @ ID 5175226 and sent them my logs.

Gostev
SVP, Product Management
Posts: 24014
Liked: 3250 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Full backup speed decreases as it runs

Post by Gostev » Feb 27, 2012 9:01 am

wahrheit2004 wrote:Realtimestatistics says Bottleneck "target" but CPU is only used @ 1core and here 80% kernel time.
Actually, "target" means the bottleneck is the target storage write speed. Nothing to deal with CPU load.

wahrheit2004
Influencer
Posts: 10
Liked: 2 times
Joined: Feb 27, 2012 7:33 am
Full Name: DW
Contact:

Re: Full backup speed decreases as it runs

Post by wahrheit2004 » Feb 28, 2012 7:18 am

Yes and no. what I did to verify is to copy a 4.3GB DVD ISO Image to the same lokal Volume Veeam uses for it's backups.
Copying the .ISO from a remote LAN_Server with a 1GBit/s LAN Card I reach 60MB/s write.
If I duplicate it from local volume to local volume my speed is 46MB/s in write, handling read and write I/O from same Volume.

Btw.:
From my backup-job above the Real-Time statistics show 57% @ 10MB/s - Duration 45hrs 47min currently.

Gostev
SVP, Product Management
Posts: 24014
Liked: 3250 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Full backup speed decreases as it runs

Post by Gostev » Feb 28, 2012 9:34 am

Not sure what you mean by "yes and no". Regarding your test, you said your backup speed decreases with time, backing up very large files. So copying small 4.3GB ISO file won't really show anything?

Anyway, as far as I know your support case is being looked at.

wahrheit2004
Influencer
Posts: 10
Liked: 2 times
Joined: Feb 27, 2012 7:33 am
Full Name: DW
Contact:

Re: Full backup speed decreases as it runs

Post by wahrheit2004 » Feb 29, 2012 7:19 am

Hi Gostev,
Gostev wrote:So copying small 4.3GB ISO file won't really show anything?
I generated a 2,19 TB File in the volume of the Veeambackupfiles by using this command:
fsutil file createnew test.txt 2242880000000

Than I copied this file in same folder to "test - Kopie.txt" using windows Explorer.
Detail of copy process shows 2xxMB/s and Ressource manager shows 290MB/s in peeks up to 3xxMB/s.

Image

My 4,9 TB Veeam backup process is at 72% @ 8MB/s duration 69hrs 44 min...
Herefor I have a webex meeting tomorrow, hope we can find the issue.

wahrheit2004
Influencer
Posts: 10
Liked: 2 times
Joined: Feb 27, 2012 7:33 am
Full Name: DW
Contact:

Re: Full backup speed decreases as it runs

Post by wahrheit2004 » Mar 01, 2012 1:39 pm

Hi everyone,

I had the webex meeting today with veeam support. They identified it that is a issue deep in their B&R software.
Now developers should take it for deeper investigation and hopefully a fix soon.

What you can see ist, that the 1st HDDs are processed at a good speed, but than dramatically drop - see here:
Image

This is by running a full-backup. I hope that the reverse-incrmental will be processed a lot faster :D .

So waiting for fix 4 or 5....

Gostev
SVP, Product Management
Posts: 24014
Liked: 3250 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Full backup speed decreases as it runs

Post by Gostev » Mar 02, 2012 9:30 am

wahrheit2004 wrote:They identified it that is a issue deep in their B&R software.
News to me! I am on the same email thread between support and devs regarding the webex with you (already saw this picture above). At this time, the opinion is nothing but speculation of a specific support engineer. Multiple tests on very large VMs in our own labs could not reproduce this behavior, however it is true however that developers will take it for deeper investigation ;)

Post Reply

Who is online

Users browsing this forum: csydas, restore-helper and 20 guests