Case # 01700972 - Hung backup and proxies used

[VeaamGuy]

Hi,
I logged a job with the Veeam helpdesk a few days ago for hung backup jobs. Have uploaded the logs as per instructions provided but received no reply even after 24 hours.
Was hoping for them to contact asap and resolve for a more permanent solution for the hung backups. In the end we couldn't wait any longer and rebooted the server after which the backups were finally stopped.

This now brings us to the 2nd issue which I need your help with:

We have 4 Veeam Proxy (2x virtual and 2x physical)
Since we have 2 VMware HA clusters, we have 1 virtual proxy and 1 physical proxy sitting in each. The physical proxy have now been connected to the SAN storage so should be able to backup in SAN mode.
Unfortuantely with 30 VMs in the job being backed up, the job only used the virtual proxies in hotadd mode while VMs were still waiting for the backup to be performed, can a Veeam job not use a mix of hotadd for some VMs and SAN for other VMs to speed up the backup process? If yes, does it need any setting to be enabled that I am missing?

Kind regards,
Rushad.

[alanbolte]

If your support case is not handled to your satisfaction, kindly log into the customer portal and click the "talk to a manager" link.

To the best of my knowledge, a single job will not use a mix of SAN and hotadd if it thinks it can use SAN for all VMs; it will wait to process all VMs by SAN mode instead. If you want to use both types of proxies, you should use some combination of:
1. Splitting up the jobs and specifying proxies for the jobs such that certain VMs are forced to back up using the hotadd proxies
2. Not configuring certain datastores for SAN mode, such that VMs on those datastores must be backed up by hotadd.

However, since it used hotadd for everything, there is probably some mistake with your attempt to configure for SAN mode. In some cases our support team can find the problem, but often the SAN vendor must be involved.

[VeaamGuy]

Hi Alan,

Thanks for your prompt reply, appreciate it.

I used the option in the customer portal for "Request an update" and then got a reply that they are investigating and will revert in a few hours.

SAN mode does work, I just ran a test and did a backup of the virtual proxy selecting the physical proxy (instead of the automatic proxy selection option) in the same VMware cluster and that went through fine. The same works too for the other VMware cluster too. So hopefully no need to contact SAN vendor.

Only one difference though I used a backup repository server instead of CIFS share for the backup location. Does that make a difference in any way?

Your 2 points are good options to help enable the use of all the proxies but that would kill the purpose of our design i.e. if one of the proxy servers in a VMware cluster had issues or is taken offline for maintenance the other one would pick up and continue the backup processing.

Also it seems quite inefficient that even though there are like 20VMs or more waiting for a backup to be taken, the job waits rather than using the available virtual proxy for processing some of the VMs via hot add and vice-versa with the SAN that I am facing!!

Would you consider the ability to use both modes in one job to process the VMs for future improvements or has this already been looked at and is not practically achievable? i.e. Hoping this is a reasonable request.

Thanks in advance.

Kind Regards,
Rushad.

[alanbolte]

Hmm... just to double-check, if you look at the job statistics for each VM, there's a line for each disk that says which proxy was used. It will have the transport mode listed in brackets. Are you seeing [NBD] or [SAN]?

I'm not on the product management team so I won't comment on the feature request.

[VeaamGuy]

Hi Alan,

Hmm... just to double-check, if you look at the job statistics for each VM, there's a line for each disk that says which proxy was used. It will have the transport mode listed in brackets. Are you seeing [NBD] or [SAN]?

I'm not on the product management team so I won't comment on the feature request.

Yes, that is exactly where I was getting the details from weather it is using SAN mode or hotadd mode.
Veeam support manually edited the proxy settings and changed it from "Automatic selection" to "Direct SAN access"
Now it is using the direct SAN proxies for everything but still we have the same problem, the job has so many other VMs to process but still waits on it instead of using the Veeam virtual proxy server.
The request is to process complete VMs within a job the same method i.e. not go down to the HDD level for different type of processing and I am hoping that would be easier to achieve.

In a job if there were some VMs that were not accessible by the physical proxy (direct SAN) would it then switch to NBD mode instead or use hotadd which is a faster option compared to NBD?

Also is it possible for you to find out from the product management team if this useful feature (which will make processing jobs a lot more quicker with limited amount of backup infrastructure) will be included in an update for v8 and if its already included in v9.


Kind Regards,
Rushad.

[Delo123]

Did you by any chance limit the number of concurrent tasks to the repository?
And yes, if you disable san access for some luns they will backup in a different mode (when allowed by proxy settings)

[VeaamGuy]

Hi Guido,

Thanks for your reply.

Did you by any chance limit the number of concurrent tasks to the repository?
And yes, if you disable san access for some luns they will backup in a different mode (when allowed by proxy settings)

I have changed the repository to have no restrictions now (it had a traffic restriction rather than concurrent tasks) also checked the network settings in Veeam and no throttling.
Another thing I changed was from a CIFS share to a windows server backup repository. Now I have 4 VMs in each VMware Cluster being backed up simultaneously like before but the surprising part is: 2 on each side are being backed up via SAN and 2 via hotadd, In fact I even saw a VM where 1 disk was backed up by SAN and the other 3 via hotadd.

Note the pending message on remaining VMs show:
Queued for processing at xx/xx/xxxx xx:xx:xx a.m.
Resource not ready: Active snapshots limit reached for datastore "

1. Is this restriction coming from VMware or is it a Veeam one and
2. Can it be tweaked to allow for more snapshots.

Kind regards,
Rushad.

[alanbolte]

vmware-vsphere-f24/active-snapshots-lim ... 19711.html

[Delo123]

Hi Rushad,

switching to a windows server backup repository should give you much better performance than cifs, good one
I do not think you should get an active snapshot limit message with "only" 4 vm's. Do you have everything on a single datastore? I can't imagine you hit active datastore limit on a single vm. What does support have to see about it?

Guido

[VeaamGuy]

Good day Guido,

Hi Rushad,

switching to a windows server backup repository should give you much better performance than cifs, good one
I do not think you should get an active snapshot limit message with "only" 4 vm's. Do you have everything on a single datastore? I can't imagine you hit active datastore limit on a single vm. What does support have to see about it?

Guido

The current job is setup to backup VMs on 2 datastores. I can see that that 4 VMs actively get backed up per datastore and all the other VMs have that message and are waiting to backup. I haven't tried that reg fix suggested by Alan yet, that's the next step.

Support is still working on why the backups were hung. They think it might be a VMware issue and have asked me to restart all ESXi hosts and vcenter server. But my concern is that no running VMs are affected, just the Veeam backup was hung. Have asked them for more information. Being a production system, quite a bit of documentation and signatures will need to be taken care of before I can go ahead and do that.