Slow Performance on Reverse Incremental since Upgrade to v6

[scott_mac]

Hi Folks,

Finally got round to upgrading to version 6 of Veeam recently - since then the performance of one of my servers is very slow. It is the VM that contains Veeam which may explain it, but before the upgrade it would run at around 500Mb/s and complete the 150Gb in a matter of minutes - it is now taking nearly 50.

I didn't have stats before but it was (and i believe still is) running in Direct SAN access mode. The stats now show for pretty much all of the jobs that have run as:

09/05/2012 20:13:31 :: Load: Source 98% > Proxy 17% > Network 1% > Target 6%
09/05/2012 20:13:31 :: Primary bottleneck: Source

It's the only job that I run as a Reverse Incremental. I was hoping that i would be able to increase the performance of RI's so I could use them for all servers but on this evidence, that won't be the case.

For the most part the other jobs that I run as a normal incremental are running at the same speed, with the load at source being around 65%. Though bizarrely one of them does take an age to complete - the job seems to progress at a decent rate but sits at 99% complete for about 30 minutes.

2 issues i'm fairly sure, but strange!

Any ideas??

Thanks

Scott

[dellock6]

Scott,
you talked about DirectSan access and Veeam in a VM at the same time, is it correct? If true, you are using iscsi storage and the Veeam VM has a network connection directly into the iscsi network? Trying to figure out your scenario... Please check on the job report which method was used...

Reverse Incremental puts an extra load (compared to forward) on the target storage, not on the source one.

[scott_mac]

Hi Luca,

You are correct with the scenario we use.

I cannot find out in v6 where it says what method was used.... when the job was setup in v5, Direct SAN was used so I presume it is still the same.

As you say Reverse usually loads the target - hence i'm a bit puzzled.

Thanks

Scott

[foggy]

I cannot find out in v6 where it says what method was used.... when the job was setup in v5, Direct SAN was used so I presume it is still the same.

Just select the particular VM to the left in the Realtime Statistics window and you will get the processing details for this VM in the right pane.

[scott_mac]

Hi foggy,

I'm not 100% certain, i believe the relevant bits are:

09/05/2012 19:31:10 :: Using source proxy VMware Backup Proxy [nbd]
09/05/2012 19:31:23 :: Preparing guest for hot backup
09/05/2012 19:32:08 :: Hard Disk 1 (150.0 GB) 144.0 GB read at 61 MB/s

Can't see anything stating a specific version.... though looking at another version that says:

09/05/2012 21:16:00 :: Using source proxy VMware Backup Proxy [hotadd;nbd]

Given that mentions hotadd, does that mean that my first has for some reason failed over to running in network mode (nbd?)

Scott

[Cokovic]

09/05/2012 19:31:10 :: Using source proxy VMware Backup Proxy [nbd]
09/05/2012 19:31:23 :: Preparing guest for hot backup
09/05/2012 19:32:08 :: Hard Disk 1 (150.0 GB) 144.0 GB read at 61 MB/s

Correct. This job has run in network mode.

[foggy]

Given that mentions hotadd, does that mean that my first has for some reason failed over to running in network mode (nbd?)

Exactly. And as you're experiencing the issue with the Veeam backup server VM (which is also a default proxy), this is explained by the fact that the proxy cannot hotadd itself (known limitation, covered in the release notes).

You either need to add another proxy server to backup this particular VM or use Direct SAN access mode, which is available in case of iSCSI connection to the storage (as Luca suspected you were using).

[scott_mac]

Thanks - my next obvious question is how to dictate what mode it runs in now?

It used to be a bullet point choice, but I can no longer see this choice - I can select the VM by going through either the ESX host, or through the vCenter selection.

Thanks

[foggy]

Thanks - my next obvious question is how to dictate what mode it runs in now?
It used to be a bullet point choice, but I can no longer see this choice - I can select the VM by going through either the ESX host, or through the vCenter selection.

The corresponding settings can now be found in the proxy server settings (right-click the proxy > Properties > Transport mode).

[scott_mac]

Thanks foggy - given that I only have one proxy at the moment I only seem to get the option to change it for the proxy (based on what you're saying)

It's currently set to 'Automatic' - given that we use iSCSI is Direct SAN the best option rather than Virtual Appliance (which i understand to be the hotadd mode?)?

[foggy]

Yes, but Direct SAN should be used by default if available and if it's set to Automatic. You could try to manually select the required datastore in the proxy settings if it's not detected automatically.

[scott_mac]

Ok, i've selected all of the available datastores manually so will see how it runs this evening.

It seems odd as it saw them immediately (v5 used to query them for a long time) which suggests it is aware of them all.

Given what you say above about it using Direct SAN automatically, what should be displayed in the properties - nbd = network, hotadd = Virtual Appliance & ??? = Direct SAN??

Thanks for your help

Scott

[foggy]

nbd = network, hotadd = Virtual Appliance & ??? = Direct SAN??

You could guess: san = Direct SAN

[scott_mac]

And the mystery unfolds....

I just ran a smaller job to see what result we'd get and got:

"Unable to establish direct connection to the shared storage (SAN). Please ensure that: - HBA is properly installed in the Veeam Backup server computer, or software iSCSI initiator is configured correctly. - SAN volume can be seen by operating system in the Windows Disk Management snap-in on the Veeam Backup server. - Read access is allowed for the Veeam Backup server computer on the corresponding LUN (refer to your SAN documentation).
Direct SAN connection is not available, failing over to network mode..."

Which explains why Direct SAN mode hasn't been working!

P.S. Thanks foggy.... that is fairly obvious!

[dellock6]

Have you configured correctly the software iscsi initiator in the Veeam VM, and the right ACL in the iscsi storage? That a look at this, even if running iscsi inside a VM sound weird to most people, it can work...

[scott_mac]

Thanks folks - am looking into why the Direct SAN isn't working and seeing the above, however, since changing the configuration to try and get that working I am now seeing:

"Unable to release guest. Error: VSSControl: Failed to freeze guest, wait timeout. Error: VSSControl: Failed to freeze guest, wait timeout"

Snapshots are being made and removed fine according to vSphere and everything else appears to be in order.

I'm on the latest patch (6.0.0.181 - 64 Bit) which seemed to be some suggestions when searching.... the SQL db containing Veeam is on this server and it is the vCenter server too. Seems odd that it a) worked in v5 and b) worked before i tried to change the backup method!

[scott_mac]

Err... ok. Veeam seems to be acting a bit like a spoilt child.

I've reported it on here and now it seems to be performing! The job seems to be running at the moment... whether it completes or not, i don't know but it's running which is more than it has been.

[Gostev]

Scott, kindly please do not mix all the different issues in the single topic, as this is going to make it extremely confusing for the future readers. Please continue posting about this new issue you are having in the existing topics dedicated to this error (that you have found through search), and I will remove the posts above from this topic. Thanks!

[scott_mac]

Hi Gostev,

Apologies - it seems that in trying to change the transport mode I had made the server look through VC rather than directly through the ESX host. I've tested this (and found info in another thread) and if you're trying to back up the Veeam server as a VM, then you cannot do it through VC - you get the above error if using Application Aware. Might be a useful one for others.

As it stands though I am aware of the issue I have and now need to work on getting iSCSI resolved.

Thanks for help from all

Scott

[Vitaliy S.]

I've tested this (and found info in another thread) and if you're trying to back up the Veeam server as a VM, then you cannot do it through VC - you get the above error if using Application Aware. Might be a useful one for others.

I never heard about this issue, but what you've just described should be used for vCenter Server backup jobs where application aware image processing is enabled.