Resource not ready: VMware backup proxy

[mikegoodwin]

Hi,

I have a veeam deployment where I am continually seeing the message "Resource not ready: VMware backup proxy".

All of the veeam components are installed on one (virtual) server with 8 cores and 16gb ram.

In the properties of the "VMware backup proxy", I have max concurrent tasks set to 18 (yes, this is above 8 recommended for an 8 core server, but I have increased it trying to fix this issue)

As it stands right now, I have one job running (full backup of a large-ish fileserver), and 4 other jobs sat at 0% with the "Resource not ready: VMware backup proxy" message.

Is there anywhere else that limits could be applied? Or is there anyway to see more details in that error message as to what specific resource its waiting for?

Thanks,
Mike

[foggy]

Michael, how many disks does that VM have? Do you have parallel processing enabled? What type of repository do you have and how it is added to Veeam B&R? More details can be found in the job logs, you can ask support for assistance in reading them.

[mikegoodwin]

Hi Foggy,

The backup repositiory is hosted on a 8 disk QNAP storage presented via iSCSI. Total storage is 32TB, RAID6.
Current IOPS being used around 600 - 800, and the storage can do upto around 1750 - 2000.
I've not set any restrictions on the repository or the proxy in regards to disk IO, so hopefully this wont be causing a problem.

Parallel processing is enabled.

It seems that one of the "waiting" tasks is a backup of the SQL server that hosts the veeam DB (and therefore, I believe, is not a parallel job).
In this case, it seems that any jobs started after this non-parallel job see that there is a non-parallel job pending, and so "wait" for that job to complete.

The non-parallel job waits for the active job to complete (which because its a full of a 1.5tb fileserver, is going to take a good few hours). No other job will start whilst this non-parallel job is waiting, even though they could run and be finished before the larger job is finished.

Not sure if this is "by design". I can see it could be difficult to work out the queuing mechanism, as predicting how long the wait will be for the non-parallel job is going to be hard, so you don' want to start of another job that could last even longer....

I think I may re-design my backup job schedules to try and avoid this. Or move the Veeam DB onto its own DB server (so as not to cause this issue)

Interestingly, if I force stop the non-parallel job, the "stuck" jobs remain "stuck". But, any newly started (parallel enabled) jobs begin working immediately.....

Thanks,
Mike

[foggy]

There's indeed a logic that prevents tasks related to Veeam B&R server itself (or the server holding its database) from being processed in parallel with other tasks within a given job. Such tasks are placed at the bottom of the processing queue and wait for the resources assignment until all other job tasks are completed. However this logic is effective within a single job only and doesn't prevent other jobs from being processed in parallel.

You did not answer the question about the number of disks the file server VM has and another question is how many tasks do you see processed in parallel?

[mikegoodwin]

Ho Foggy,

Sorry, the fileserver has 2 disks (1 x 250gb OS, 1 x 2tb data). The task that processes this fileserver has processed those disks sequentially (i.e, not in parallel).

When there are multiple jobs running, I have seen as many as 4 or 5 run together, but it does seem that whenever the "database backup" job is pending, any newly started jobs also wait.

I proved this, by force stopping the non-parallel job, and then starting another job, which ran immediately.

Thanks,
Mike

[foggy]

This doesn't look expected, so I recommend contacting support for further investigation.

[PKaufmann]

was this solved? Have the same issue right now..
database backup job is running and waiting for VSS, for this time no other job runs

[foggy]

database backup job is running and waiting for VSS, for this time no other job runs

This looks different from the OP's problem. Anyway, it's always better to contact support directly with all technical issues.

[geets]

Hi,

sorry for digging up that old thread but I have a similar problem and stumbled upon it.
foggy asked about the tasks being processed in parallel:

[...] how many tasks do you see processed in parallel

How do I determine this? When I click on a job that is running and I see a list of VMs that this job will backup. Is the "number of tasks processed in parallel" the VMs that show a percentage in the status field?
Like:
vmx - Pending
vmy - success
vmz - 14%
vma - 99%

Thanks for the clarification!

Alex

[foggy]

More than that as the task is actually a VM disk, so if a VM has multiple disks all of them in theory could be processed in parallel.

[HelpMe]

Hello,
is there a solution now?

I got this on one Server (2 VMs getting backuped in this Job) after 12h one VM was like on 4% an the other showed: "Ressource not ready: backup proxy".
And this for 12h???
Can anyone explain or help me on this?

[MarkBoothmaa]

How many tasks is your repository set to? If I recall the default is 4 tasks, How many proxy slots and then how many disks are being processed on each VM?

[HelpMe]

Thanks for your answer @MarkBoothmaa !
I dont excatly know what do you actually need from me but i will give you some information. We have 2 Backup-Jobs. The first Job has 2 big VMs with 1-2 Virtual Disks. The second job has 8 VMs. 3 retries than it breaks up and as far as i know the backuping paralel.
Task one starts Mo-Fr at 22 o clock.
Task two start Mo-Fr at 20 o clock.
Feel free to ask everything you need to know.

[HelpMe]

Oh wait i found something might help you. Under Backup proxies i have one Type: Agent, Created by Veeam Backup & Replication and there is set: 2 on the Setting "Max concurrent tasks".
Is this what you need or something that helps you.

[PetrM]

Hello,

Most likely you don't have enough task slots on the backup proxy server and you need to increase the number of concurrent tasks. Please refer to this page of our help center to get a better idea of our resource scheduler logic.

Thanks!

[HelpMe]

Hello, i actually was reading this a few minutes before i checked this Thread to look for you answer.
I kindly get it but also i dont.
So i understand it that way:
Proxy: 1 concurrent task / 1 CPU core
Repo: 1 concurrent task / 1 GB RAM

Thats waht my research deliverd. But i also was reading at this site "2 tasks = 1 CPU core" (on Backup Repository). Is it than:
Proxy same
Repo: 2 concurrent task / 1 CPU core
?
Because that wouldnt make sense to me.
Could you mabe help me to understand this?

[HelpMe]

I mean I have the machine running these Veeam jobs. This machine is the backup proxy and also attaches the 2 RDXs (connected via USB 3.0) where the backups are stored, so in a way it's also the repository.
This machine has 16 CPU cores (32 logical) and 64 GB of RAM. (only as a marginal formation)

[HelpMe]

I also don't understand what the difference between the "Backup Proxy" and the "VMware Backup Proxy" is. (both Created by Veeam Backup & Replication)
BTW: these two RDX Backups are Hyper V Backup, so i conclude i only need "VMware Backup Proxy".

[HelpMe]

These VMs who getting backuped running on two seperate HyperV-Servers so one half runs on Hyper-V1 and one on Hyper-V2 (which are physical). They two physical have a cluster and get replicated to the server where the backup is running.

[veremin]

It might be a stupid question - but you are backing up Hyper-V VMs, right? If so, the role of the default proxy server is assigned to the source Hyper-V host, not to a dedicated machine (definitely not to the VMware proxy server). Thanks!

[HelpMe]

I do backup HyperV VMs that correct. I didnt changed anything at proxy or repository settings and as far as i know nobody else did it. So everything should be in the defaultsetting there.
I think you want to know who is the backupproxy.
As i metioned before in this thread i got two backupproxy-entries. (both "Created by Veeam Backup & Replication" so they were created automatically from Veeam)
And both of these entries has the same host. This host is the server where Veeam and the backups are running on.

To come back to your second sentence: What is the "source Hyper-V host"? The server where the Veeambackups are running? If so it is as i wrote up in this comment.
If you need a screenshot or something let me know.

[PetrM]

Hello,

You may try to increase the number of concurrent tasks in Hyper-V host settings. If it does not help, please open a support case and share a support case ID with us. It seems strange that the message about VMware backup proxy unreadiness occurs during Hyper-V backups.

Thanks!

[HelpMe]

Hello,
i know what you mean but what is the setting i change there? Is more RAM used than?

[MarkBoothmaa]

Adding additional tasks will result in an increase in RAM usage

To limit the number of concurrent tasks in a backup repository, you must enable the Limit maximum concurrent tasks to <N> option on the backup repository and define the necessary task limit.

The maximum number of concurrent tasks depends on the number of CPU cores available in the backup repository. It is strongly recommended that you define task limitation settings using the following rule: 2 tasks = 1 CPU core. It is recommended to configure 2 GB RAM for each concurrently processed machine or file share. In case of shared folder backup repositories, the same amount of resources is required for gateway servers. For example, for 1 VM with 4 virtual disks you need 2 CPU cores and 2 GB RAM. For 2 VMs with 2 virtual disks you need 2 CPU cores (as the number of tasks was not increased) and 4 GB RAM.

[veremin]

Either way, it's better to continue the investigation with the support team - you shouldn't get VMware-specific errors while backing up Hyper-V resources. Thanks!

[PetrM]

@HelpMe

2 GB of RAM for each concurrent task is required as it's stated in our release notes ( page 8 ).

Thanks!

[HelpMe]

@MarkBoothmaa okay so if i understand that right, Backup repository tasks using RAM and Backup proxies using CPU?
Leaves still my question: Which type of backup proxy do i need now?

@veremin it's not an error jsut a question an i already did that but this needs to much time.

@PetrM thank you also for your reply but still leaves the question like on @MarkBoothmaa.

[MarkBoothmaa]

@HelpMe each concurrent task will consume CPU & RAM on the proxy and the repository.

VMware backup proxy is for virtual VM backups. The backup proxy is for agents, NAS or file shares.

[veremin]

@veremin, it's not an error jsut a question an i already did that but this needs to much time.

The fact that you got VMware-specific messages in a Hyper-V environment made me think something is going wrong; thus, I recommended reaching the support team. Since you have done that already, continue working with them, as these forums are not replacements for technical support. Thanks!

[HelpMe]

@MarkBoothmaa i guess.

So i need to use the VMware backup proxy in my current situation?