Comprehensive data protection for all workloads
Post Reply
ThinkGrid
Enthusiast
Posts: 55
Liked: 2 times
Joined: Jul 11, 2011 10:18 am
Contact:

Source bottleneck, what does it mean? (Source 99%)

Post by ThinkGrid »

Hi

I'm getting 99% of bottleneck for my Source but what does it actually mean ? The v6 guide said this could mean that the majority of processing is being done at the source and infact there is nothing wrong with the infrastructure

But what is the source in this instance? ESXi hosts? Backup server itself? SAN ?

Also, what metrics does Veeam measure so that we can go back to those metrics in our environment and look to troubleshoot ?

Many thanks
Petrit
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: source bottleneck, what does it mean?

Post by Gostev »

Hi, this is covered in great details in the sticky FAQ topic.
ThinkGrid
Enthusiast
Posts: 55
Liked: 2 times
Joined: Jul 11, 2011 10:18 am
Contact:

Re: source bottleneck, what does it mean?

Post by ThinkGrid »

Thanks, had a read

But it would have been so much cooler if it was able to tell us the individual aspects of the bottleneck such as driver out of date or VMXNET3 driver is better then E1000 and stuff like that

Thanks anyway
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: source bottleneck, what does it mean?

Post by Gostev »

I know there are dedicated performance analysis and troubleshooting tools, but not sure even they can go as deep as you desire... this would require truly immense amount of built-in knowledge. By the way, it would be cool if those performance analysis tools also did backups, but none of them do ;) likewise, deep performance analysis is simply out of scope of our backup and recovery product...
WinstonWolf
Veteran
Posts: 284
Liked: 11 times
Joined: Jan 06, 2011 8:33 am
Contact:

[MERGED] Veeam Backup 6.5, Bottleneck Source 99%

Post by WinstonWolf »

Hello ,

My Installation is as followed :

2 x ESX 5.0 U1 Hosts , IBM x3650 Server , per Server 2 Network Cards for the VM Network
1 x DS 3300 Storage connected via ISCSI with the Servers
1 x ML330 ( Quad Core, 6 GB RAM ) used as Veeam Backup Proxy , on this Server only one Network Card , and an NETGEAR ReadyNas Pro 6 via ISCSI Initiator in W2008 R2 connected .

When my Backup Jobs are finished , i see the followed : Load Source 99% , Proxy 55% , Network 1% , Target 2%

What can be the reasin for the very High Source load ?

What can i control or make better . Thanks for Suggestions

Michael
gollem
Enthusiast
Posts: 33
Liked: 7 times
Joined: Jun 16, 2012 7:26 pm
Full Name: Erik E.
Contact:

Re: Veeam Backup 6.5, Bottleneck Source 99%

Post by gollem »

Do not worry Micheal,

I'm running a similar setup but with a IBM v7000 (22x600gb 10k sas + 2x300GB SSD) on a dual (4 FC in the storage, 2 switches, 2 FC per server) 8gbit/second fiberchannel setup.

My bottleneck is also the source (99%)

What is your average backup speed?


Mine:

Code: Select all

Success 1 Start time 18:00:16 Total size 738,5 GB Backup size 6,6 GB  
Warning 0 End time 18:14:14 Data read 13,2 GB Dedupe 1,0x 
Error 0 Duration 0:13:58  Transferred 6,5 GB Compression 2,0x 
Details  
Name Status Start time End time Size Read Transferred Duration Details 
HOSTNAME Success 18:02:24 18:14:05 738,5 GB 13,2 GB 6,5 GB 0:11:41  
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Veeam Backup 6.5, Bottleneck Source 99%

Post by Vitaliy S. »

Hi Micheal, source bottleneck means that your production VM storage cannot provide data to ESXi host any faster. Can you please tell me what backup mode you're using? Thanks!
Shralok
Enthusiast
Posts: 51
Liked: 5 times
Joined: Aug 29, 2012 5:36 pm
Full Name: Shralok
Contact:

Re: Source bottleneck, what does it mean? (Source 99%)

Post by Shralok »

I have the same issue. Backups are taking hours because I have a source bottleneck (99%). Jobs are between 7-10 MB/s. :evil: :x :(

Is there a way to improve this bottleneck? Or is my SAN connectivity really that bad?
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Source bottleneck, what does it mean? (Source 99%)

Post by Gostev »

Actually, it does not look like direct SAN access mode is used in your case... looks more like network mode. Can you confirm if virtual disks are processed in [SAN] mode?
Shralok
Enthusiast
Posts: 51
Liked: 5 times
Joined: Aug 29, 2012 5:36 pm
Full Name: Shralok
Contact:

Re: Source bottleneck, what does it mean? (Source 99%)

Post by Shralok »

Gostev wrote:Actually, it does not look like direct SAN access mode is used in your case... looks more like network mode. Can you confirm if virtual disks are processed in [SAN] mode?
Which post is this reply directed to?

Thx
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Source bottleneck, what does it mean? (Source 99%)

Post by Gostev »

To your post xD
Shralok
Enthusiast
Posts: 51
Liked: 5 times
Joined: Aug 29, 2012 5:36 pm
Full Name: Shralok
Contact:

Re: Source bottleneck, what does it mean? (Source 99%)

Post by Shralok »

Gostev wrote:Actually, it does not look like direct SAN access mode is used in your case... looks more like network mode. Can you confirm if virtual disks are processed in [SAN] mode?
Gostev wrote:To your post xD
I'm not following your question unfortunately - could you point me in the right direction to check on this?
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Source bottleneck, what does it mean? (Source 99%)

Post by Vitaliy S. »

Do you see [SAN] tag next to processed disk name in the VM session log?
Shralok
Enthusiast
Posts: 51
Liked: 5 times
Joined: Aug 29, 2012 5:36 pm
Full Name: Shralok
Contact:

Re: Source bottleneck, what does it mean? (Source 99%)

Post by Shralok »

Vitaliy S. wrote:Do you see [SAN] tag next to processed disk name in the VM session log?
No, it is currently [onhost] for all of my jobs.
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Source bottleneck, what does it mean? (Source 99%)

Post by Gostev »

Ah, so you are using Hyper-V and on-host backup mode? In that case, looks like the storage system itself is the bottleneck indeed... cannot find other explanation why it is so slow. What is your production storage? Is it heavily loaded?
Shralok
Enthusiast
Posts: 51
Liked: 5 times
Joined: Aug 29, 2012 5:36 pm
Full Name: Shralok
Contact:

Re: Source bottleneck, what does it mean? (Source 99%)

Post by Shralok »

Gostev wrote:Ah, so you are using Hyper-V and on-host backup mode? In that case, looks like the storage system itself is the bottleneck indeed... cannot find other explanation why it is so slow. What is your production storage? Is it heavily loaded?
We use a EMC CLARiiON AX4 connected via ethernet. About 70% loaded with some VMs on CSVs and some not. I have yet to find a correlation relating to speed when comparing the VMs on CSV vs. VMs not.

Going on 19 hours for a 770 GB VM :shock:

Is there any way to tell what it is in "source" that is taking so long?

Backing up to local storage on the physical Veeam server.
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Source bottleneck, what does it mean? (Source 99%)

Post by Vitaliy S. »

Install Veeam ONE to check the performance (IOPs, read/write rate etc.) of your cluster shared volumes during the backup winwow. It should be a good starting point of your investigation.
WinstonWolf
Veteran
Posts: 284
Liked: 11 times
Joined: Jan 06, 2011 8:33 am
Contact:

Re: Source bottleneck, what does it mean? (Source 99%)

Post by WinstonWolf »

Hello ,

My Backup Proxy was set to "Network Mode" , yesterday i set the Backup Proxy to "Automatic" and just in the Moment i controll the Backup result :

Image

27.11.2012 02:22:56 :: Direct SAN connection is not available, failing over to network mode...

27.11.2012 02:25:53 :: Unable to establish direct connection to the shared storage (SAN).
Please ensure that:
- HBA is properly installed in the Veeam Backup server computer, or software iSCSI initiator is configured correctly.
- SAN volume can be seen by operating system in the Windows Disk Management snap-in on the Veeam Backup server.
- Read access is allowed for the Veeam Backup server computer on the corresponding LUN (refer to your SAN documentation).

The Backup trys to start in direct SAN acces Mode , but fails with it ? What can be the reason ?
Or is it the fact , that in my environment only "network mode" is possible ?

Michael
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Source bottleneck, what does it mean? (Source 99%)

Post by Vitaliy S. »

Possible reasons are mentioned in the info message you've quoted above. It seems like you just haven't configured your proxy server properly, thus it cannot process your VMs in the direct SAN mode.
dellock6
Veeam Software
Posts: 6137
Liked: 1928 times
Joined: Jul 26, 2009 3:39 pm
Full Name: Luca Dell'Oca
Location: Varese, Italy
Contact:

Re: Source bottleneck, what does it mean? (Source 99%)

Post by dellock6 »

By saying "We use a EMC CLARiiON AX4 connected via ethernet" what do you mean? What kind of storage connection are you using on it, FC or iscsi?
And is Veeam proxy a virtual machine or a physical one?

Luca.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software

@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
Shralok
Enthusiast
Posts: 51
Liked: 5 times
Joined: Aug 29, 2012 5:36 pm
Full Name: Shralok
Contact:

Re: Source bottleneck, what does it mean? (Source 99%)

Post by Shralok »

dellock6 wrote:By saying "We use a EMC CLARiiON AX4 connected via ethernet" what do you mean? What kind of storage connection are you using on it, FC or iscsi?
And is Veeam proxy a virtual machine or a physical one?

Luca.
iSCSI connection
Veeam proxy is the Hyper-V host (we have 4 of them), all physical boxes

I installed Veeam ONE Monitor and I've been getting a ton of these (492 in 15 hours):

(taken from multiple emails but some of the emails have multiple lines of Warning/Error)

"Avg Disk sec/Read" (152 ms) is above a defined threshold (25 ms) for 'I:';
"Avg Disk sec/Write" (16 ms) is above a defined threshold (15 ms) for 'HarddiskVolume55';
"Avg Disk Queue Length" (2) is above a defined threshold (0) for 'K:';
"Avg Disk sec/Write" (22 ms) is above a defined threshold (15 ms) for 'HarddiskVolume19';
"Avg Disk sec/Write" (24 ms) is above a defined threshold (15 ms) for 'HarddiskVolume339';
"Avg Disk sec/Read" (60 ms) is above a defined threshold (25 ms) for 'I:';
"Avg Disk sec/Write" (18 ms) is above a defined threshold (15 ms) for 'F:';
"Avg Disk sec/Write" (19 ms) is above a defined threshold (15 ms) for 'U:';
"Avg Disk sec/Read" (160 ms) is above a defined threshold (25 ms) for 'I:';
"Avg Disk sec/Read" (53 ms) is above a defined threshold (25 ms) for 'I:';
"Avg Disk sec/Write" (23 ms) is above a defined threshold (15 ms) for 'D:';
"Avg Disk sec/Write" (22 ms) is above a defined threshold (15 ms) for 'HarddiskVolume10';
"Avg Disk sec/Write" (16 ms) is above a defined threshold (15 ms) for 'HarddiskVolume13';
"Avg Disk sec/Write" (20 ms) is above a defined threshold (15 ms) for 'HarddiskVolume14';
"Avg Disk sec/Write" (38 ms) is above a defined threshold (25 ms) for 'HarddiskVolume55';

Does anyone know what a CSV shows up as in terms of drive? I am assuming it is not going to show up as a letter - are the "HarddiskVolumeXX" the CSV?
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Source bottleneck, what does it mean? (Source 99%)

Post by Vitaliy S. »

Looks like you've nailed it. Do you also see the same latency values during non-backup hours?
Shralok wrote:Does anyone know what a CSV shows up as in terms of drive? I am assuming it is not going to show up as a letter - are the "HarddiskVolumeXX" the CSV?
You can find the correlation between "HarddiskVolumeXX" and actual drives via "Share and Storage Management" MMC snap-in. Hope this helps!
Shralok
Enthusiast
Posts: 51
Liked: 5 times
Joined: Aug 29, 2012 5:36 pm
Full Name: Shralok
Contact:

Re: Source bottleneck, what does it mean? (Source 99%)

Post by Shralok »

Yes, the latency is 24/7 and the alerts have been steady since installing Veeam ONE.

I suppose I will have to do some testing and possibly get the SAN guy involved. Although the alerts are not for the SAN, they are for the Hyper-V Host. So I am not sure where the problem exists.

It has been one issue after another with Veeam, unfortunately :(
Gostev
Chief Product Officer
Posts: 31460
Liked: 6648 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Source bottleneck, what does it mean? (Source 99%)

Post by Gostev » 2 people like this post

Shralok wrote:It has been one issue after another with Veeam, unfortunately :(
Honestly, it would be REALLY strange if backup and monitoring products did not uncover existing I/O capacity issues with the storage infrastructure. Implicitly in case with any backup product - as those create noticeable additional I/O load on the storage, and explicitly with any monitoring product - as those are designed to alert you about all the issues in your environment ;)
Shralok
Enthusiast
Posts: 51
Liked: 5 times
Joined: Aug 29, 2012 5:36 pm
Full Name: Shralok
Contact:

Re: Source bottleneck, what does it mean? (Source 99%)

Post by Shralok »

I agree and I am not blaming Veeam for this setback (well, this particular one), but it is delaying my Veeam deployment and possibly creating larger projects that I do not have in my plan or budget :|
Post Reply

Who is online

Users browsing this forum: Google [Bot] and 245 guests