-
- Enthusiast
- Posts: 55
- Liked: 2 times
- Joined: Jul 11, 2011 10:18 am
- Contact:
Source bottleneck, what does it mean? (Source 99%)
Hi
I'm getting 99% of bottleneck for my Source but what does it actually mean ? The v6 guide said this could mean that the majority of processing is being done at the source and infact there is nothing wrong with the infrastructure
But what is the source in this instance? ESXi hosts? Backup server itself? SAN ?
Also, what metrics does Veeam measure so that we can go back to those metrics in our environment and look to troubleshoot ?
Many thanks
Petrit
I'm getting 99% of bottleneck for my Source but what does it actually mean ? The v6 guide said this could mean that the majority of processing is being done at the source and infact there is nothing wrong with the infrastructure
But what is the source in this instance? ESXi hosts? Backup server itself? SAN ?
Also, what metrics does Veeam measure so that we can go back to those metrics in our environment and look to troubleshoot ?
Many thanks
Petrit
-
- Chief Product Officer
- Posts: 31779
- Liked: 7279 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: source bottleneck, what does it mean?
Hi, this is covered in great details in the sticky FAQ topic.
-
- Enthusiast
- Posts: 55
- Liked: 2 times
- Joined: Jul 11, 2011 10:18 am
- Contact:
Re: source bottleneck, what does it mean?
Thanks, had a read
But it would have been so much cooler if it was able to tell us the individual aspects of the bottleneck such as driver out of date or VMXNET3 driver is better then E1000 and stuff like that
Thanks anyway
But it would have been so much cooler if it was able to tell us the individual aspects of the bottleneck such as driver out of date or VMXNET3 driver is better then E1000 and stuff like that
Thanks anyway
-
- Chief Product Officer
- Posts: 31779
- Liked: 7279 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: source bottleneck, what does it mean?
I know there are dedicated performance analysis and troubleshooting tools, but not sure even they can go as deep as you desire... this would require truly immense amount of built-in knowledge. By the way, it would be cool if those performance analysis tools also did backups, but none of them do likewise, deep performance analysis is simply out of scope of our backup and recovery product...
-
- Veteran
- Posts: 284
- Liked: 11 times
- Joined: Jan 06, 2011 8:33 am
- Contact:
[MERGED] Veeam Backup 6.5, Bottleneck Source 99%
Hello ,
My Installation is as followed :
2 x ESX 5.0 U1 Hosts , IBM x3650 Server , per Server 2 Network Cards for the VM Network
1 x DS 3300 Storage connected via ISCSI with the Servers
1 x ML330 ( Quad Core, 6 GB RAM ) used as Veeam Backup Proxy , on this Server only one Network Card , and an NETGEAR ReadyNas Pro 6 via ISCSI Initiator in W2008 R2 connected .
When my Backup Jobs are finished , i see the followed : Load Source 99% , Proxy 55% , Network 1% , Target 2%
What can be the reasin for the very High Source load ?
What can i control or make better . Thanks for Suggestions
Michael
My Installation is as followed :
2 x ESX 5.0 U1 Hosts , IBM x3650 Server , per Server 2 Network Cards for the VM Network
1 x DS 3300 Storage connected via ISCSI with the Servers
1 x ML330 ( Quad Core, 6 GB RAM ) used as Veeam Backup Proxy , on this Server only one Network Card , and an NETGEAR ReadyNas Pro 6 via ISCSI Initiator in W2008 R2 connected .
When my Backup Jobs are finished , i see the followed : Load Source 99% , Proxy 55% , Network 1% , Target 2%
What can be the reasin for the very High Source load ?
What can i control or make better . Thanks for Suggestions
Michael
-
- Enthusiast
- Posts: 33
- Liked: 7 times
- Joined: Jun 16, 2012 7:26 pm
- Full Name: Erik E.
- Contact:
Re: Veeam Backup 6.5, Bottleneck Source 99%
Do not worry Micheal,
I'm running a similar setup but with a IBM v7000 (22x600gb 10k sas + 2x300GB SSD) on a dual (4 FC in the storage, 2 switches, 2 FC per server) 8gbit/second fiberchannel setup.
My bottleneck is also the source (99%)
What is your average backup speed?
Mine:
I'm running a similar setup but with a IBM v7000 (22x600gb 10k sas + 2x300GB SSD) on a dual (4 FC in the storage, 2 switches, 2 FC per server) 8gbit/second fiberchannel setup.
My bottleneck is also the source (99%)
What is your average backup speed?
Mine:
Code: Select all
Success 1 Start time 18:00:16 Total size 738,5 GB Backup size 6,6 GB
Warning 0 End time 18:14:14 Data read 13,2 GB Dedupe 1,0x
Error 0 Duration 0:13:58 Transferred 6,5 GB Compression 2,0x
Details
Name Status Start time End time Size Read Transferred Duration Details
HOSTNAME Success 18:02:24 18:14:05 738,5 GB 13,2 GB 6,5 GB 0:11:41
-
- VP, Product Management
- Posts: 27364
- Liked: 2794 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: Veeam Backup 6.5, Bottleneck Source 99%
Hi Micheal, source bottleneck means that your production VM storage cannot provide data to ESXi host any faster. Can you please tell me what backup mode you're using? Thanks!
-
- Enthusiast
- Posts: 51
- Liked: 5 times
- Joined: Aug 29, 2012 5:36 pm
- Full Name: Shralok
- Contact:
Re: Source bottleneck, what does it mean? (Source 99%)
I have the same issue. Backups are taking hours because I have a source bottleneck (99%). Jobs are between 7-10 MB/s.
Is there a way to improve this bottleneck? Or is my SAN connectivity really that bad?
Is there a way to improve this bottleneck? Or is my SAN connectivity really that bad?
-
- Chief Product Officer
- Posts: 31779
- Liked: 7279 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Source bottleneck, what does it mean? (Source 99%)
Actually, it does not look like direct SAN access mode is used in your case... looks more like network mode. Can you confirm if virtual disks are processed in [SAN] mode?
-
- Enthusiast
- Posts: 51
- Liked: 5 times
- Joined: Aug 29, 2012 5:36 pm
- Full Name: Shralok
- Contact:
Re: Source bottleneck, what does it mean? (Source 99%)
Which post is this reply directed to?Gostev wrote:Actually, it does not look like direct SAN access mode is used in your case... looks more like network mode. Can you confirm if virtual disks are processed in [SAN] mode?
Thx
-
- Chief Product Officer
- Posts: 31779
- Liked: 7279 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Source bottleneck, what does it mean? (Source 99%)
To your post xD
-
- Enthusiast
- Posts: 51
- Liked: 5 times
- Joined: Aug 29, 2012 5:36 pm
- Full Name: Shralok
- Contact:
Re: Source bottleneck, what does it mean? (Source 99%)
Gostev wrote:Actually, it does not look like direct SAN access mode is used in your case... looks more like network mode. Can you confirm if virtual disks are processed in [SAN] mode?
I'm not following your question unfortunately - could you point me in the right direction to check on this?Gostev wrote:To your post xD
-
- VP, Product Management
- Posts: 27364
- Liked: 2794 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: Source bottleneck, what does it mean? (Source 99%)
Do you see [SAN] tag next to processed disk name in the VM session log?
-
- Enthusiast
- Posts: 51
- Liked: 5 times
- Joined: Aug 29, 2012 5:36 pm
- Full Name: Shralok
- Contact:
Re: Source bottleneck, what does it mean? (Source 99%)
No, it is currently [onhost] for all of my jobs.Vitaliy S. wrote:Do you see [SAN] tag next to processed disk name in the VM session log?
-
- Chief Product Officer
- Posts: 31779
- Liked: 7279 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Source bottleneck, what does it mean? (Source 99%)
Ah, so you are using Hyper-V and on-host backup mode? In that case, looks like the storage system itself is the bottleneck indeed... cannot find other explanation why it is so slow. What is your production storage? Is it heavily loaded?
-
- Enthusiast
- Posts: 51
- Liked: 5 times
- Joined: Aug 29, 2012 5:36 pm
- Full Name: Shralok
- Contact:
Re: Source bottleneck, what does it mean? (Source 99%)
We use a EMC CLARiiON AX4 connected via ethernet. About 70% loaded with some VMs on CSVs and some not. I have yet to find a correlation relating to speed when comparing the VMs on CSV vs. VMs not.Gostev wrote:Ah, so you are using Hyper-V and on-host backup mode? In that case, looks like the storage system itself is the bottleneck indeed... cannot find other explanation why it is so slow. What is your production storage? Is it heavily loaded?
Going on 19 hours for a 770 GB VM
Is there any way to tell what it is in "source" that is taking so long?
Backing up to local storage on the physical Veeam server.
-
- VP, Product Management
- Posts: 27364
- Liked: 2794 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: Source bottleneck, what does it mean? (Source 99%)
Install Veeam ONE to check the performance (IOPs, read/write rate etc.) of your cluster shared volumes during the backup winwow. It should be a good starting point of your investigation.
-
- Veteran
- Posts: 284
- Liked: 11 times
- Joined: Jan 06, 2011 8:33 am
- Contact:
Re: Source bottleneck, what does it mean? (Source 99%)
Hello ,
My Backup Proxy was set to "Network Mode" , yesterday i set the Backup Proxy to "Automatic" and just in the Moment i controll the Backup result :
27.11.2012 02:22:56 :: Direct SAN connection is not available, failing over to network mode...
27.11.2012 02:25:53 :: Unable to establish direct connection to the shared storage (SAN).
Please ensure that:
- HBA is properly installed in the Veeam Backup server computer, or software iSCSI initiator is configured correctly.
- SAN volume can be seen by operating system in the Windows Disk Management snap-in on the Veeam Backup server.
- Read access is allowed for the Veeam Backup server computer on the corresponding LUN (refer to your SAN documentation).
The Backup trys to start in direct SAN acces Mode , but fails with it ? What can be the reason ?
Or is it the fact , that in my environment only "network mode" is possible ?
Michael
My Backup Proxy was set to "Network Mode" , yesterday i set the Backup Proxy to "Automatic" and just in the Moment i controll the Backup result :
27.11.2012 02:22:56 :: Direct SAN connection is not available, failing over to network mode...
27.11.2012 02:25:53 :: Unable to establish direct connection to the shared storage (SAN).
Please ensure that:
- HBA is properly installed in the Veeam Backup server computer, or software iSCSI initiator is configured correctly.
- SAN volume can be seen by operating system in the Windows Disk Management snap-in on the Veeam Backup server.
- Read access is allowed for the Veeam Backup server computer on the corresponding LUN (refer to your SAN documentation).
The Backup trys to start in direct SAN acces Mode , but fails with it ? What can be the reason ?
Or is it the fact , that in my environment only "network mode" is possible ?
Michael
-
- VP, Product Management
- Posts: 27364
- Liked: 2794 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: Source bottleneck, what does it mean? (Source 99%)
Possible reasons are mentioned in the info message you've quoted above. It seems like you just haven't configured your proxy server properly, thus it cannot process your VMs in the direct SAN mode.
-
- VeeaMVP
- Posts: 6165
- Liked: 1971 times
- Joined: Jul 26, 2009 3:39 pm
- Full Name: Luca Dell'Oca
- Location: Varese, Italy
- Contact:
Re: Source bottleneck, what does it mean? (Source 99%)
By saying "We use a EMC CLARiiON AX4 connected via ethernet" what do you mean? What kind of storage connection are you using on it, FC or iscsi?
And is Veeam proxy a virtual machine or a physical one?
Luca.
And is Veeam proxy a virtual machine or a physical one?
Luca.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software
@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
Principal EMEA Cloud Architect @ Veeam Software
@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
-
- Enthusiast
- Posts: 51
- Liked: 5 times
- Joined: Aug 29, 2012 5:36 pm
- Full Name: Shralok
- Contact:
Re: Source bottleneck, what does it mean? (Source 99%)
iSCSI connectiondellock6 wrote:By saying "We use a EMC CLARiiON AX4 connected via ethernet" what do you mean? What kind of storage connection are you using on it, FC or iscsi?
And is Veeam proxy a virtual machine or a physical one?
Luca.
Veeam proxy is the Hyper-V host (we have 4 of them), all physical boxes
I installed Veeam ONE Monitor and I've been getting a ton of these (492 in 15 hours):
(taken from multiple emails but some of the emails have multiple lines of Warning/Error)
"Avg Disk sec/Read" (152 ms) is above a defined threshold (25 ms) for 'I:';
"Avg Disk sec/Write" (16 ms) is above a defined threshold (15 ms) for 'HarddiskVolume55';
"Avg Disk Queue Length" (2) is above a defined threshold (0) for 'K:';
"Avg Disk sec/Write" (22 ms) is above a defined threshold (15 ms) for 'HarddiskVolume19';
"Avg Disk sec/Write" (24 ms) is above a defined threshold (15 ms) for 'HarddiskVolume339';
"Avg Disk sec/Read" (60 ms) is above a defined threshold (25 ms) for 'I:';
"Avg Disk sec/Write" (18 ms) is above a defined threshold (15 ms) for 'F:';
"Avg Disk sec/Write" (19 ms) is above a defined threshold (15 ms) for 'U:';
"Avg Disk sec/Read" (160 ms) is above a defined threshold (25 ms) for 'I:';
"Avg Disk sec/Read" (53 ms) is above a defined threshold (25 ms) for 'I:';
"Avg Disk sec/Write" (23 ms) is above a defined threshold (15 ms) for 'D:';
"Avg Disk sec/Write" (22 ms) is above a defined threshold (15 ms) for 'HarddiskVolume10';
"Avg Disk sec/Write" (16 ms) is above a defined threshold (15 ms) for 'HarddiskVolume13';
"Avg Disk sec/Write" (20 ms) is above a defined threshold (15 ms) for 'HarddiskVolume14';
"Avg Disk sec/Write" (38 ms) is above a defined threshold (25 ms) for 'HarddiskVolume55';
Does anyone know what a CSV shows up as in terms of drive? I am assuming it is not going to show up as a letter - are the "HarddiskVolumeXX" the CSV?
-
- VP, Product Management
- Posts: 27364
- Liked: 2794 times
- Joined: Mar 30, 2009 9:13 am
- Full Name: Vitaliy Safarov
- Contact:
Re: Source bottleneck, what does it mean? (Source 99%)
Looks like you've nailed it. Do you also see the same latency values during non-backup hours?
You can find the correlation between "HarddiskVolumeXX" and actual drives via "Share and Storage Management" MMC snap-in. Hope this helps!Shralok wrote:Does anyone know what a CSV shows up as in terms of drive? I am assuming it is not going to show up as a letter - are the "HarddiskVolumeXX" the CSV?
-
- Enthusiast
- Posts: 51
- Liked: 5 times
- Joined: Aug 29, 2012 5:36 pm
- Full Name: Shralok
- Contact:
Re: Source bottleneck, what does it mean? (Source 99%)
Yes, the latency is 24/7 and the alerts have been steady since installing Veeam ONE.
I suppose I will have to do some testing and possibly get the SAN guy involved. Although the alerts are not for the SAN, they are for the Hyper-V Host. So I am not sure where the problem exists.
It has been one issue after another with Veeam, unfortunately
I suppose I will have to do some testing and possibly get the SAN guy involved. Although the alerts are not for the SAN, they are for the Hyper-V Host. So I am not sure where the problem exists.
It has been one issue after another with Veeam, unfortunately
-
- Chief Product Officer
- Posts: 31779
- Liked: 7279 times
- Joined: Jan 01, 2006 1:01 am
- Location: Baar, Switzerland
- Contact:
Re: Source bottleneck, what does it mean? (Source 99%)
Honestly, it would be REALLY strange if backup and monitoring products did not uncover existing I/O capacity issues with the storage infrastructure. Implicitly in case with any backup product - as those create noticeable additional I/O load on the storage, and explicitly with any monitoring product - as those are designed to alert you about all the issues in your environmentShralok wrote:It has been one issue after another with Veeam, unfortunately
-
- Enthusiast
- Posts: 51
- Liked: 5 times
- Joined: Aug 29, 2012 5:36 pm
- Full Name: Shralok
- Contact:
Re: Source bottleneck, what does it mean? (Source 99%)
I agree and I am not blaming Veeam for this setback (well, this particular one), but it is delaying my Veeam deployment and possibly creating larger projects that I do not have in my plan or budget
Who is online
Users browsing this forum: Google [Bot] and 95 guests