Source bottleneck, what does it mean? (Source 99%)

[ThinkGrid]

Hi

I'm getting 99% of bottleneck for my Source but what does it actually mean ? The v6 guide said this could mean that the majority of processing is being done at the source and infact there is nothing wrong with the infrastructure

But what is the source in this instance? ESXi hosts? Backup server itself? SAN ?

Also, what metrics does Veeam measure so that we can go back to those metrics in our environment and look to troubleshoot ?

Many thanks
Petrit

[Gostev]

Hi, this is covered in great details in the sticky FAQ topic.

[ThinkGrid]

Thanks, had a read

But it would have been so much cooler if it was able to tell us the individual aspects of the bottleneck such as driver out of date or VMXNET3 driver is better then E1000 and stuff like that

Thanks anyway

[Gostev]

I know there are dedicated performance analysis and troubleshooting tools, but not sure even they can go as deep as you desire... this would require truly immense amount of built-in knowledge. By the way, it would be cool if those performance analysis tools also did backups, but none of them do likewise, deep performance analysis is simply out of scope of our backup and recovery product...

[WinstonWolf]

Hello ,

My Installation is as followed :

2 x ESX 5.0 U1 Hosts , IBM x3650 Server , per Server 2 Network Cards for the VM Network
1 x DS 3300 Storage connected via ISCSI with the Servers
1 x ML330 ( Quad Core, 6 GB RAM ) used as Veeam Backup Proxy , on this Server only one Network Card , and an NETGEAR ReadyNas Pro 6 via ISCSI Initiator in W2008 R2 connected .

When my Backup Jobs are finished , i see the followed : Load Source 99% , Proxy 55% , Network 1% , Target 2%

What can be the reasin for the very High Source load ?

What can i control or make better . Thanks for Suggestions

Michael

[gollem]

Do not worry Micheal,

I'm running a similar setup but with a IBM v7000 (22x600gb 10k sas + 2x300GB SSD) on a dual (4 FC in the storage, 2 switches, 2 FC per server) 8gbit/second fiberchannel setup.

My bottleneck is also the source (99%)

What is your average backup speed?


Mine:

Code: Select all

Success 1 Start time 18:00:16 Total size 738,5 GB Backup size 6,6 GB  
Warning 0 End time 18:14:14 Data read 13,2 GB Dedupe 1,0x 
Error 0 Duration 0:13:58  Transferred 6,5 GB Compression 2,0x 
Details  
Name Status Start time End time Size Read Transferred Duration Details 
HOSTNAME Success 18:02:24 18:14:05 738,5 GB 13,2 GB 6,5 GB 0:11:41  

[Vitaliy S.]

Hi Micheal, source bottleneck means that your production VM storage cannot provide data to ESXi host any faster. Can you please tell me what backup mode you're using? Thanks!

[Shralok]

I have the same issue. Backups are taking hours because I have a source bottleneck (99%). Jobs are between 7-10 MB/s.

Is there a way to improve this bottleneck? Or is my SAN connectivity really that bad?

[Gostev]

Actually, it does not look like direct SAN access mode is used in your case... looks more like network mode. Can you confirm if virtual disks are processed in [SAN] mode?

[Shralok]

Actually, it does not look like direct SAN access mode is used in your case... looks more like network mode. Can you confirm if virtual disks are processed in [SAN] mode?

Which post is this reply directed to?

Thx

[Gostev]

To your post xD

[Shralok]

Actually, it does not look like direct SAN access mode is used in your case... looks more like network mode. Can you confirm if virtual disks are processed in [SAN] mode?

To your post xD

I'm not following your question unfortunately - could you point me in the right direction to check on this?

[Vitaliy S.]

Do you see [SAN] tag next to processed disk name in the VM session log?

[Shralok]

Do you see [SAN] tag next to processed disk name in the VM session log?

No, it is currently [onhost] for all of my jobs.

[Gostev]

Ah, so you are using Hyper-V and on-host backup mode? In that case, looks like the storage system itself is the bottleneck indeed... cannot find other explanation why it is so slow. What is your production storage? Is it heavily loaded?

[Shralok]

Ah, so you are using Hyper-V and on-host backup mode? In that case, looks like the storage system itself is the bottleneck indeed... cannot find other explanation why it is so slow. What is your production storage? Is it heavily loaded?

We use a EMC CLARiiON AX4 connected via ethernet. About 70% loaded with some VMs on CSVs and some not. I have yet to find a correlation relating to speed when comparing the VMs on CSV vs. VMs not.

Going on 19 hours for a 770 GB VM

Is there any way to tell what it is in "source" that is taking so long?

Backing up to local storage on the physical Veeam server.

[Vitaliy S.]

Install Veeam ONE to check the performance (IOPs, read/write rate etc.) of your cluster shared volumes during the backup winwow. It should be a good starting point of your investigation.

[WinstonWolf]

Hello ,

My Backup Proxy was set to "Network Mode" , yesterday i set the Backup Proxy to "Automatic" and just in the Moment i controll the Backup result :



27.11.2012 02:22:56 :: Direct SAN connection is not available, failing over to network mode...

27.11.2012 02:25:53 :: Unable to establish direct connection to the shared storage (SAN).
Please ensure that:
- HBA is properly installed in the Veeam Backup server computer, or software iSCSI initiator is configured correctly.
- SAN volume can be seen by operating system in the Windows Disk Management snap-in on the Veeam Backup server.
- Read access is allowed for the Veeam Backup server computer on the corresponding LUN (refer to your SAN documentation).

The Backup trys to start in direct SAN acces Mode , but fails with it ? What can be the reason ?
Or is it the fact , that in my environment only "network mode" is possible ?

Michael

[Vitaliy S.]

Possible reasons are mentioned in the info message you've quoted above. It seems like you just haven't configured your proxy server properly, thus it cannot process your VMs in the direct SAN mode.

[dellock6]

By saying "We use a EMC CLARiiON AX4 connected via ethernet" what do you mean? What kind of storage connection are you using on it, FC or iscsi?
And is Veeam proxy a virtual machine or a physical one?

Luca.

[Shralok]

By saying "We use a EMC CLARiiON AX4 connected via ethernet" what do you mean? What kind of storage connection are you using on it, FC or iscsi?
And is Veeam proxy a virtual machine or a physical one?

Luca.

iSCSI connection
Veeam proxy is the Hyper-V host (we have 4 of them), all physical boxes

I installed Veeam ONE Monitor and I've been getting a ton of these (492 in 15 hours):

(taken from multiple emails but some of the emails have multiple lines of Warning/Error)

"Avg Disk sec/Read" (152 ms) is above a defined threshold (25 ms) for 'I:';
"Avg Disk sec/Write" (16 ms) is above a defined threshold (15 ms) for 'HarddiskVolume55';
"Avg Disk Queue Length" (2) is above a defined threshold (0) for 'K:';
"Avg Disk sec/Write" (22 ms) is above a defined threshold (15 ms) for 'HarddiskVolume19';
"Avg Disk sec/Write" (24 ms) is above a defined threshold (15 ms) for 'HarddiskVolume339';
"Avg Disk sec/Read" (60 ms) is above a defined threshold (25 ms) for 'I:';
"Avg Disk sec/Write" (18 ms) is above a defined threshold (15 ms) for 'F:';
"Avg Disk sec/Write" (19 ms) is above a defined threshold (15 ms) for 'U:';
"Avg Disk sec/Read" (160 ms) is above a defined threshold (25 ms) for 'I:';
"Avg Disk sec/Read" (53 ms) is above a defined threshold (25 ms) for 'I:';
"Avg Disk sec/Write" (23 ms) is above a defined threshold (15 ms) for 'D:';
"Avg Disk sec/Write" (22 ms) is above a defined threshold (15 ms) for 'HarddiskVolume10';
"Avg Disk sec/Write" (16 ms) is above a defined threshold (15 ms) for 'HarddiskVolume13';
"Avg Disk sec/Write" (20 ms) is above a defined threshold (15 ms) for 'HarddiskVolume14';
"Avg Disk sec/Write" (38 ms) is above a defined threshold (25 ms) for 'HarddiskVolume55';

Does anyone know what a CSV shows up as in terms of drive? I am assuming it is not going to show up as a letter - are the "HarddiskVolumeXX" the CSV?

[Vitaliy S.]

Looks like you've nailed it. Do you also see the same latency values during non-backup hours?

Does anyone know what a CSV shows up as in terms of drive? I am assuming it is not going to show up as a letter - are the "HarddiskVolumeXX" the CSV?

You can find the correlation between "HarddiskVolumeXX" and actual drives via "Share and Storage Management" MMC snap-in. Hope this helps!

[Shralok]

Yes, the latency is 24/7 and the alerts have been steady since installing Veeam ONE.

I suppose I will have to do some testing and possibly get the SAN guy involved. Although the alerts are not for the SAN, they are for the Hyper-V Host. So I am not sure where the problem exists.

It has been one issue after another with Veeam, unfortunately

[Gostev]

It has been one issue after another with Veeam, unfortunately

Honestly, it would be REALLY strange if backup and monitoring products did not uncover existing I/O capacity issues with the storage infrastructure. Implicitly in case with any backup product - as those create noticeable additional I/O load on the storage, and explicitly with any monitoring product - as those are designed to alert you about all the issues in your environment

[Shralok]

I agree and I am not blaming Veeam for this setback (well, this particular one), but it is delaying my Veeam deployment and possibly creating larger projects that I do not have in my plan or budget