Issues with backups

[Rugbyfan]

Hi guys, quick query I have a ticket open with Veeam , finding it difficult to a remote session (bar the first one)

Veeam backup very slow the past few days, good few consolidation errors

3 ESXi Hosts, approx 50 vms

Now we have a backup window of 10pm but backup has been at 96% since 7.15pm , if the backup goes straight through till 10pm will it be affected?

Single B&R server

Just looking for advice on what to do (Previously there was a temporary snapshot on the Veeam server, which I couldnt remove, so had to restart management agent) I couldn't access the vm folder as part of /vmfs/volumes/datastore

I had asked for a remote session today serveral times on ticket but was finally told a few hours after asking that no slots available today

Any advice appreciated

[Mildur]

Hi Bill

Can you post the case number? Veeam officials can have a look at the state of the case if you post it.

Have you tried escalating the case to get a faster response and remote session?

Now we have a backup window of 10pm but backup has been at 96% since 7.15pm , if the backup goes straight through till 10pm will it be affected?

Backup will be cancelled at 10pm, if you have configured a backup window that ends at 10PM.

Just looking for advice on what to do (Previously there was a temporary snapshot on the Veeam server, which I couldnt remove, so had to restart management agent) I couldn't access the vm folder as part of /vmfs/volumes/datastore

Restarting veeam services when the backup jobs are running is never a good idea.
Can you tell us more about this temporary snapshot? Is your vbr server a VM and you are using hotadd (virtual appliance mode) as a backup method?

[Rugbyfan]

Thanks is it ok to post the ticket reference here or is there a private chat I can use, Veeam support changed it from nbd to hotadd as it was taking too long

It was never really followed up since last Friday , yes VBR server is a VM , below is the update from Friday from Veeam Support

In our webex session we have analyzed the latest backup job run and it was running correctly. The only issue I observed is that the job was running a bit slow.

This can be attributed to the transport mode, your backup job is using network (nbd) transport mode.

I would recommend switching to virtual appliance (hotadd) transport mode. I have already setup your proxy to use hotadd mode and fallback to nbd should hotadd not be available.

Also, here are the guides I was mentioning in our call:
• Test hotadd functionality: https://www.veeam.com/kb1184
• Hotadd requirements: https://www.veeam.com/kb1054

Backup is now at 97% is it just a case of waiting for it to finish? (Does it overlap for the scheduled 10pm job) Then removing the hot add disks, there are a good few disks attached, 13 at present

I saw the jobs for some of them were finished, does backup have to be finished before i can remove them from Veeam server and start consolidating?

Sorry forgot to say Veeam Version 10 is in use here , its a small setup just looking to get it back to how it was working

[Mildur]

Thanks is it ok to post the ticket reference here or is there a private chat I can use

It‘s ok to post it here. It‘s even the number one forum rule. Technical issues needs a case number posted (written in red text when you open a topic)
veeam-backup-replication-f2/rules-of-po ... -t755.html

It was never really followed up since last Friday , yes VBR server is a VM , below is the update from Friday from Veeam Support

Thanks for the case details.
Without knowing your infrastructure it‘s difficult to say, what the issue is.
From your text, i assume, you have removed the connection to the vm base disk when the backup was running of this disk. That could lead to strange behaviors in the backup job.
With hotadd, veeam tells the vmware server to create a snapshot of the vms. After that, the base disk of the backed up vm is mounted to the veeam proxy. In your case your vbr server. Is it possible that you have removed this connection when restarting the veeam service with a running backup job?

My recommendation, escalate the case to a manager, and ask for the remote session.

Second, for the slow job, you can use the bottleneck statistic to analyze which component is „slow“.
Bottleneck Analysis

[Rugbyfan]

Thanks the case reference is Case # 05131819, will i be contacted privately or should I just escalate in the morning??


Backup is still at 97%, is it a case of waiting for job to finish (tonight or whenever it finishes) , then removing the hot add disks from the Veeam Server?

Before the changes I often had to remove stuck disks, but it was always straightforward and after backup finished

Just to give you the overview, previously up to a few weeks ago ,backup finished at approx 3.30-4am nightly

Whats best thing to do? wait it out?

Thanks again for the advice

[Mildur]

Thanks the case reference is Case # 05131819, will i be contacted privately or should I just escalate in the morning??

Thanks, you should escalate it by yourself. You have the button in the veeam support portal.

Backup is still at 97%, is it a case of waiting for job to finish (tonight or whenever it finishes) , then removing the hot add disks from the Veeam Server?

Difficult to say without knowing your infrastructure and the status of the backup job. Over the forum, resolving technical issues is difficult.
How long is the job on 97%. Since you restarted the services? Then I would cancel the job, remove the hotadded disks from the proxy and try the job again.
Veeam KB1775 - Removing Stuck VMDK’s from the Veeam Proxy

If not, I would wait for Veeam Support or cancel the stop anyway, remove the stuck vm disks and start the stop again.

[Rugbyfan]

100% totally appreciate that, will look to escalate in the morning unless its best to cancel the job now, just the response so far this week wasn't great

Backup going fine till about 7.15pm I left it at 96% then, its moved to 97% just before 10pm and is still there

Thanks very much I have that KB article, (as I said used to removing stuck disks after backup completeled successfully, not sure when it doesnt in this case) what steps would you recommend?

Cancelling the job, and removing the hot added disks from the proxy? I'm just a little worried as saw KB articles that snapshots can be left standalone etc

If I do cancel the job, how confident would you be that i could remove the disks from the Veeam server and look to start again tomorrow? (Just looking for approx answer, I realise you cant be 100% as you don't know environment)

[Mildur]

Backup going fine till about 7.15pm I left it at 96% then, its moved to 97% just before 10pm and is still there

For the slow jobs, use the bottleneck information in the statistic. I have posted a link to get more infos about how to use that information to learn about the bottlenecks in your infrastructure.

If I do cancel the job, how confident would you be that i could remove the disks from the Veeam server and look to start again tomorrow? (Just looking for approx answer, I realise you cant be 100% as you don't know environment)

If I face this issue in my environment, I would cancel the job. Then see what happens and react on the outcome. I would do that, because I have experience with hanging hotadded disks (previous backup software, it was a pain).

I would‘t do it in your Environment, because I don‘t know the environment. And if I see hotadd backups getting stucked in the backup job, I would leave that to veeam support to decide what do to. My feeling tells me, that there is something not right. Messing around by myself could make the issue more badly.
Escalating is the right choice at the moment.

[Rugbyfan]

Thanks very much really appreciate it I spent a few nights on this already, so will wait and see what happens, hopefully some one from Veeam will be available in the morning to take a look. There is one last VM as part of job, it has gone from 54% to 61% since 9.30pm so will hopefully be finished by morning

I will check the bottleneck link as well,

[Rugbyfan]

Just another query if ok, there was a temporary snapshot on the Veeam server Monday night, which I thought was odd as the Veeam server is excluded from the backups. Thats where the issues started this week. Would that temporary snapshot have been because of the changes?

I followed the below KB for it and thats where the issues started, I had been able to do that for regular vms in the backup before,

https://www.veeam.com/kb1790

[Rugbyfan]

Hi thanks Mildur, I have requested escalation, still waiting, the backup completed last night at 2.01am , it failed

It then started again at 2.11 and finished at 2.52am with one success for an earlier failure, (never got notification mail for 2.01am)

I am unable to remove the stuck disks, from the Veeam Server, hopefully Veeam Support will contact me soon

[Mildur]

Hi Bill
Your welcome. Lets wait for a support manager to get in contact with you.