Host-based backup of Microsoft Hyper-V VMs.
mikegoodwin
Novice
Posts: 6
Liked: never
Joined: Apr 17, 2014 7:04 pm
Full Name: Michael Goodwin
Contact:

Resource not ready: VMware backup proxy

Post by mikegoodwin »

Hi,

I have a veeam deployment where I am continually seeing the message "Resource not ready: VMware backup proxy".

All of the veeam components are installed on one (virtual) server with 8 cores and 16gb ram.

In the properties of the "VMware backup proxy", I have max concurrent tasks set to 18 (yes, this is above 8 recommended for an 8 core server, but I have increased it trying to fix this issue)

As it stands right now, I have one job running (full backup of a large-ish fileserver), and 4 other jobs sat at 0% with the "Resource not ready: VMware backup proxy" message.

Is there anywhere else that limits could be applied? Or is there anyway to see more details in that error message as to what specific resource its waiting for?

Thanks,
Mike
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Resource not ready: VMware backup proxy

Post by foggy »

Michael, how many disks does that VM have? Do you have parallel processing enabled? What type of repository do you have and how it is added to Veeam B&R? More details can be found in the job logs, you can ask support for assistance in reading them.
mikegoodwin
Novice
Posts: 6
Liked: never
Joined: Apr 17, 2014 7:04 pm
Full Name: Michael Goodwin
Contact:

Re: Resource not ready: VMware backup proxy

Post by mikegoodwin »

Hi Foggy,

The backup repositiory is hosted on a 8 disk QNAP storage presented via iSCSI. Total storage is 32TB, RAID6.
Current IOPS being used around 600 - 800, and the storage can do upto around 1750 - 2000.
I've not set any restrictions on the repository or the proxy in regards to disk IO, so hopefully this wont be causing a problem.

Parallel processing is enabled.

It seems that one of the "waiting" tasks is a backup of the SQL server that hosts the veeam DB (and therefore, I believe, is not a parallel job).
In this case, it seems that any jobs started after this non-parallel job see that there is a non-parallel job pending, and so "wait" for that job to complete.

The non-parallel job waits for the active job to complete (which because its a full of a 1.5tb fileserver, is going to take a good few hours). No other job will start whilst this non-parallel job is waiting, even though they could run and be finished before the larger job is finished.

Not sure if this is "by design". I can see it could be difficult to work out the queuing mechanism, as predicting how long the wait will be for the non-parallel job is going to be hard, so you don' want to start of another job that could last even longer....

I think I may re-design my backup job schedules to try and avoid this. Or move the Veeam DB onto its own DB server (so as not to cause this issue)

Interestingly, if I force stop the non-parallel job, the "stuck" jobs remain "stuck". But, any newly started (parallel enabled) jobs begin working immediately.....

Thanks,
Mike
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Resource not ready: VMware backup proxy

Post by foggy »

There's indeed a logic that prevents tasks related to Veeam B&R server itself (or the server holding its database) from being processed in parallel with other tasks within a given job. Such tasks are placed at the bottom of the processing queue and wait for the resources assignment until all other job tasks are completed. However this logic is effective within a single job only and doesn't prevent other jobs from being processed in parallel.

You did not answer the question about the number of disks the file server VM has and another question is how many tasks do you see processed in parallel?
mikegoodwin
Novice
Posts: 6
Liked: never
Joined: Apr 17, 2014 7:04 pm
Full Name: Michael Goodwin
Contact:

Re: Resource not ready: VMware backup proxy

Post by mikegoodwin »

Ho Foggy,

Sorry, the fileserver has 2 disks (1 x 250gb OS, 1 x 2tb data). The task that processes this fileserver has processed those disks sequentially (i.e, not in parallel).

When there are multiple jobs running, I have seen as many as 4 or 5 run together, but it does seem that whenever the "database backup" job is pending, any newly started jobs also wait.

I proved this, by force stopping the non-parallel job, and then starting another job, which ran immediately.

Thanks,
Mike
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Resource not ready: VMware backup proxy

Post by foggy »

This doesn't look expected, so I recommend contacting support for further investigation.
PKaufmann
Enthusiast
Posts: 51
Liked: 5 times
Joined: Oct 05, 2016 8:00 am
Contact:

Re: Resource not ready: VMware backup proxy

Post by PKaufmann »

was this solved? Have the same issue right now..
database backup job is running and waiting for VSS, for this time no other job runs :(
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Resource not ready: VMware backup proxy

Post by foggy »

PKaufmann wrote:database backup job is running and waiting for VSS, for this time no other job runs :(
This looks different from the OP's problem. Anyway, it's always better to contact support directly with all technical issues.
geets
Enthusiast
Posts: 25
Liked: 2 times
Joined: Mar 03, 2020 2:50 pm
Contact:

Re: Resource not ready: VMware backup proxy

Post by geets »

Hi,

sorry for digging up that old thread but I have a similar problem and stumbled upon it.
foggy asked about the tasks being processed in parallel:
foggy wrote: Jan 25, 2016 2:09 pm [...] how many tasks do you see processed in parallel
How do I determine this? When I click on a job that is running and I see a list of VMs that this job will backup. Is the "number of tasks processed in parallel" the VMs that show a percentage in the status field?
Like:
vmx - Pending
vmy - success
vmz - 14%
vma - 99%

Thanks for the clarification!

Alex
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Resource not ready: VMware backup proxy

Post by foggy »

More than that as the task is actually a VM disk, so if a VM has multiple disks all of them in theory could be processed in parallel.
HelpMe
Influencer
Posts: 21
Liked: 1 time
Joined: Aug 29, 2022 12:41 pm
Contact:

Re: Resource not ready: VMware backup proxy

Post by HelpMe »

Hello,
is there a solution now?

I got this on one Server (2 VMs getting backuped in this Job) after 12h one VM was like on 4% an the other showed: "Ressource not ready: backup proxy".
And this for 12h???
Can anyone explain or help me on this?
MarkBoothmaa
Veeam Legend
Posts: 179
Liked: 49 times
Joined: Mar 22, 2017 11:10 am
Full Name: Mark Boothman
Location: Darlington, United Kingdom
Contact:

Re: Resource not ready: VMware backup proxy

Post by MarkBoothmaa »

How many tasks is your repository set to? If I recall the default is 4 tasks, How many proxy slots and then how many disks are being processed on each VM?
HelpMe
Influencer
Posts: 21
Liked: 1 time
Joined: Aug 29, 2022 12:41 pm
Contact:

Re: Resource not ready: VMware backup proxy

Post by HelpMe »

Thanks for your answer @MarkBoothmaa !
I dont excatly know what do you actually need from me but i will give you some information. We have 2 Backup-Jobs. The first Job has 2 big VMs with 1-2 Virtual Disks. The second job has 8 VMs. 3 retries than it breaks up and as far as i know the backuping paralel.
Task one starts Mo-Fr at 22 o clock.
Task two start Mo-Fr at 20 o clock.
Feel free to ask everything you need to know.
HelpMe
Influencer
Posts: 21
Liked: 1 time
Joined: Aug 29, 2022 12:41 pm
Contact:

Re: Resource not ready: VMware backup proxy

Post by HelpMe »

Oh wait i found something might help you. Under Backup proxies i have one Type: Agent, Created by Veeam Backup & Replication and there is set: 2 on the Setting "Max concurrent tasks".
Is this what you need or something that helps you.
PetrM
Veeam Software
Posts: 3229
Liked: 520 times
Joined: Aug 28, 2013 8:23 am
Full Name: Petr Makarov
Location: Prague, Czech Republic
Contact:

Re: Resource not ready: VMware backup proxy

Post by PetrM »

Hello,

Most likely you don't have enough task slots on the backup proxy server and you need to increase the number of concurrent tasks. Please refer to this page of our help center to get a better idea of our resource scheduler logic.

Thanks!
HelpMe
Influencer
Posts: 21
Liked: 1 time
Joined: Aug 29, 2022 12:41 pm
Contact:

Re: Resource not ready: VMware backup proxy

Post by HelpMe »

Hello, i actually was reading this a few minutes before i checked this Thread to look for you answer.
I kindly get it but also i dont.
So i understand it that way:
Proxy: 1 concurrent task / 1 CPU core
Repo: 1 concurrent task / 1 GB RAM

Thats waht my research deliverd. But i also was reading at this site "2 tasks = 1 CPU core" (on Backup Repository). Is it than:
Proxy same
Repo: 2 concurrent task / 1 CPU core
?
Because that wouldnt make sense to me.
Could you mabe help me to understand this?
HelpMe
Influencer
Posts: 21
Liked: 1 time
Joined: Aug 29, 2022 12:41 pm
Contact:

Re: Resource not ready: VMware backup proxy

Post by HelpMe »

I mean I have the machine running these Veeam jobs. This machine is the backup proxy and also attaches the 2 RDXs (connected via USB 3.0) where the backups are stored, so in a way it's also the repository.
This machine has 16 CPU cores (32 logical) and 64 GB of RAM. (only as a marginal formation)
HelpMe
Influencer
Posts: 21
Liked: 1 time
Joined: Aug 29, 2022 12:41 pm
Contact:

Re: Resource not ready: VMware backup proxy

Post by HelpMe »

I also don't understand what the difference between the "Backup Proxy" and the "VMware Backup Proxy" is. (both Created by Veeam Backup & Replication)
BTW: these two RDX Backups are Hyper V Backup, so i conclude i only need "VMware Backup Proxy".
HelpMe
Influencer
Posts: 21
Liked: 1 time
Joined: Aug 29, 2022 12:41 pm
Contact:

Re: Resource not ready: VMware backup proxy

Post by HelpMe »

These VMs who getting backuped running on two seperate HyperV-Servers so one half runs on Hyper-V1 and one on Hyper-V2 (which are physical). They two physical have a cluster and get replicated to the server where the backup is running.
veremin
Product Manager
Posts: 20271
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Resource not ready: VMware backup proxy

Post by veremin »

It might be a stupid question - but you are backing up Hyper-V VMs, right? If so, the role of the default proxy server is assigned to the source Hyper-V host, not to a dedicated machine (definitely not to the VMware proxy server). Thanks!
HelpMe
Influencer
Posts: 21
Liked: 1 time
Joined: Aug 29, 2022 12:41 pm
Contact:

Re: Resource not ready: VMware backup proxy

Post by HelpMe »

I do backup HyperV VMs that correct. I didnt changed anything at proxy or repository settings and as far as i know nobody else did it. So everything should be in the defaultsetting there.
I think you want to know who is the backupproxy.
As i metioned before in this thread i got two backupproxy-entries. (both "Created by Veeam Backup & Replication" so they were created automatically from Veeam)
And both of these entries has the same host. This host is the server where Veeam and the backups are running on.

To come back to your second sentence: What is the "source Hyper-V host"? The server where the Veeambackups are running? If so it is as i wrote up in this comment.
If you need a screenshot or something let me know.
PetrM
Veeam Software
Posts: 3229
Liked: 520 times
Joined: Aug 28, 2013 8:23 am
Full Name: Petr Makarov
Location: Prague, Czech Republic
Contact:

Re: Resource not ready: VMware backup proxy

Post by PetrM »

Hello,

You may try to increase the number of concurrent tasks in Hyper-V host settings. If it does not help, please open a support case and share a support case ID with us. It seems strange that the message about VMware backup proxy unreadiness occurs during Hyper-V backups.

Thanks!
HelpMe
Influencer
Posts: 21
Liked: 1 time
Joined: Aug 29, 2022 12:41 pm
Contact:

Re: Resource not ready: VMware backup proxy

Post by HelpMe »

Hello,
i know what you mean but what is the setting i change there? Is more RAM used than?
MarkBoothmaa
Veeam Legend
Posts: 179
Liked: 49 times
Joined: Mar 22, 2017 11:10 am
Full Name: Mark Boothman
Location: Darlington, United Kingdom
Contact:

Re: Resource not ready: VMware backup proxy

Post by MarkBoothmaa »

Adding additional tasks will result in an increase in RAM usage

To limit the number of concurrent tasks in a backup repository, you must enable the Limit maximum concurrent tasks to <N> option on the backup repository and define the necessary task limit.

The maximum number of concurrent tasks depends on the number of CPU cores available in the backup repository. It is strongly recommended that you define task limitation settings using the following rule: 2 tasks = 1 CPU core. It is recommended to configure 2 GB RAM for each concurrently processed machine or file share. In case of shared folder backup repositories, the same amount of resources is required for gateway servers. For example, for 1 VM with 4 virtual disks you need 2 CPU cores and 2 GB RAM. For 2 VMs with 2 virtual disks you need 2 CPU cores (as the number of tasks was not increased) and 4 GB RAM.
veremin
Product Manager
Posts: 20271
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Resource not ready: VMware backup proxy

Post by veremin »

Either way, it's better to continue the investigation with the support team - you shouldn't get VMware-specific errors while backing up Hyper-V resources. Thanks!
PetrM
Veeam Software
Posts: 3229
Liked: 520 times
Joined: Aug 28, 2013 8:23 am
Full Name: Petr Makarov
Location: Prague, Czech Republic
Contact:

Re: Resource not ready: VMware backup proxy

Post by PetrM »

@HelpMe

2 GB of RAM for each concurrent task is required as it's stated in our release notes ( page 8 ).

Thanks!
HelpMe
Influencer
Posts: 21
Liked: 1 time
Joined: Aug 29, 2022 12:41 pm
Contact:

Re: Resource not ready: VMware backup proxy

Post by HelpMe »

@MarkBoothmaa okay so if i understand that right, Backup repository tasks using RAM and Backup proxies using CPU?
Leaves still my question: Which type of backup proxy do i need now?

@veremin it's not an error jsut a question an i already did that but this needs to much time.

@PetrM thank you also for your reply but still leaves the question like on @MarkBoothmaa.
MarkBoothmaa
Veeam Legend
Posts: 179
Liked: 49 times
Joined: Mar 22, 2017 11:10 am
Full Name: Mark Boothman
Location: Darlington, United Kingdom
Contact:

Re: Resource not ready: VMware backup proxy

Post by MarkBoothmaa »

@HelpMe each concurrent task will consume CPU & RAM on the proxy and the repository.

VMware backup proxy is for virtual VM backups. The backup proxy is for agents, NAS or file shares.
veremin
Product Manager
Posts: 20271
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Resource not ready: VMware backup proxy

Post by veremin »

@veremin, it's not an error jsut a question an i already did that but this needs to much time.
The fact that you got VMware-specific messages in a Hyper-V environment made me think something is going wrong; thus, I recommended reaching the support team. Since you have done that already, continue working with them, as these forums are not replacements for technical support. Thanks!
HelpMe
Influencer
Posts: 21
Liked: 1 time
Joined: Aug 29, 2022 12:41 pm
Contact:

Re: Resource not ready: VMware backup proxy

Post by HelpMe »

@MarkBoothmaa i guess.

So i need to use the VMware backup proxy in my current situation?
Post Reply

Who is online

Users browsing this forum: Google [Bot] and 22 guests