Agent Upgrade Process - Need to be fixed!!!

[adb98]

The backup agent upgrade process has a nasty issue and not sure if it is a bug or if it is as designed. Either way they need to put wordage in the setup at the least when having the check mark for auto update client in a protection group. Here is the nasty issue I ran into.

I upgraded our main VBR server and proxies to the latest version yesterday morning. All was great and running ok. I come in this morning and half of my physical servers didn't backup last night and each of them state "Host Reboot required". WTH!!!!

I started going thru it and apparently the default for the protection groups is to check box " install agent automatically" and to "upgrade the agent automatically". Well that is fine if its not going to break my backups but it did. The protection groups did the once daily scan last night and saw hey I have an upgrade and upgraded but it requires a reboot in order to go thru and will no longer take backups until that is done. Then Veeam is just failing.

This is a very very very bad situation to be in. The worst part is we have one physical file server that is the main file share for 4 hospitals and is backed up every 2 hours. Now we are not getting any backups at all. Now we are scrambling and doing an emergency change (which directors have to approve and are asking why) and will be rebooting production servers into the night tonight in order resolve.

There needs to be some sort of logic to state. Hey I can't upgrade this automatically because it will break backups until a reboot is done. Even put a note (not recommended) and remove it from default setup for a protection group.

[PTide]

Hi,

Thank you for bringing this up, your point has been fully understood. We will discuss that with the team internally and the thread will be updated accordingly with the resolution.

Thanks

[Dima P.]

Aaron,

Thank you for your post! To clarify, were these servers running agent with CBT driver installed?

[dschuler]

We are seeing the same issue here. We upgraded our Veeam B&R to 9.5u3a last week and our two SQL clusters using the Veeam agents started throwing the "Host Reboot required" error. We have rebooted the hosts of one of the clusters and are still getting the same error.

We have not opened a ticket with Veeam yet as we are still troubleshooting internally.

[dschuler]

It looks like you need to perform a rescan on the protection group after the reboot before trying the jobs again. Ran a rescan and the Status reads Installed now and the job runs without the "Host Reboot required" error.

[pdelacruz]

We also experienced the same issue. Even after I rebooted the servers, the backups were failing with same error. I then later realized that each protection group needed to be manually re-scan for the "required reboot" to clear. I think this needs to be fixed.

[adb98]

So your kinda in a damned if you do , damned if you don't situation with choosing to upgrade automatically. If you upgrade automatically there is a chance that it will stop you from being able to backup again if it upgrades the CBT driver, until you reboot the server being backed up and if choose not to upgrade automatically, it won't let you backup because it states it needs a upgrade.

This needs to be fixed so warns you that there is an available update but doesn't stop you from backing up.

I can understand that the workstation world it doesn't matter but in the server world with only one reboot a month.... This is a huge issue.

[Dima P.]

Thanks for your input. We will discuss with QA how to properly notify about the reboot requirement.

From technical standpoint there is no way to eliminate the reboot requirements when:

- You are upgrading CBT driver as a part of Veeam Agent for Windows upgrade. Drive must be unloaded from the OS thus reboot is a mandatory.
- OS is sitting in waiting for reboot state which is usually caused by Windows updates (such state blocks any product installations or upgrade).
- Core components for Windows agent must be changed. Usually it happens during major agent upgrade.

[holden205]

After applying update 3a, all of my VAW physical servers [with CBT driver installed] were negatively affected. First, they immediately quit processing backups pending an unexpected reboot. Secondly, and most destructive, all restore points for each server were deleted without a trace. There is no longer any record of the restore points in the B&R console under Backups/Disk or in the repositories themselves. All servers having the CBT driver installed deleted existing restore points and created new backup chains after reboot. This was extremely painful when considering this impacted more than 50 servers with the loss of data protection and additional bandwidth costs to recreate fulls across the WAN. I have a case open with support, but it is most likely an unexpected result of update 3a on physical agents with no recovery of previous restore points. I verified this by the simple fact servers which I did not have the CBT driver installed were unaffected. They did not lose restore points and did not require reboot after updating. This is a major setback for us.

Also, to be clear, the release notes did not mention any impact to physical servers with Veeam Agent for Windows {with or without} the CBT driver installed.

[Dima P.]

Hello Jeff.

Thank you for your post. I can confirm that CBT driver upgrade (which comes with agent upgrade in this version) requires reboot. However, backup chain should remain untouched and this behavior is unexpected. May I ask you to submit a case ID and share it with me? Thank you!

[holden205]

The case number is 03107357. I'm out of my depth, but I believe that update 3a altered my restore points by eliminating them from the B&R console. Had I known I may have been able to repopulate them by rescanning the repositories. However, all backup jobs have a 3 day retention on deleted items. Since i discovered the missing restore points on the forth day after update, the repositories deleted all backup files associated with the affected servers. My case is slowly working its way up the tiers. But, it is already to late to recover backup data aside from what may be on GFS tape.

[Frenchyaz]

We also experienced the same issue. Even after I rebooted the servers, the backups were failing with same error. I then later realized that each protection group needed to be manually re-scan for the "required reboot" to clear. I think this needs to be fixed.

Thank you. I was having the same issue and a rescan did the trick. It would be nice to have that automated after the reboot....

[Dima P.]

Hi folks,

We've implemented a special logic to the CBT driver distributed via B&R server in Update 4: once you upgrade your agent with new version of the CBT driver it will await for the reboot, but to make sure backup job will run without any issues will use the previously installed components.

Reboot might be still required by Windows components or other installed application (OS can sit in pending for reboot state and that would require reboot), but that's not related to driver update any longer with agent managed by Veeam B&R. Cheers!

[masonit]

Hi

Plan to upgrade vbr server from U4 to U4a. Connected protection group server are running 3.0.0.748 (Windows) and 3.0.0.865 (Linux). Will U4a require an agent update?

\Masonit

[DGrinev]

Hi Masonit,

We have no such requirement for Agents to work with Veeam B&R U4a. Thanks!

[masonit]

Thanks!

[DGrinev]

Please keep in mind that VBR will propose you to update the Agent to the most recent version which is 3.1, but it's not mandatory for work.

Thanks!

[jdrch]

Hi

Plan to upgrade vbr server from U4 to U4a. Connected protection group server are running 3.0.0.748 (Windows) and 3.0.0.865 (Linux). Will U4a require an agent update?

\Masonit

You should definitely be running the latest Agent builds anyway.

[GDLtechie01]

I just upgraded to VBR Version 9 update 4a, and about half of my agents are requiring a reboot (with failed backups last night!). Agent versions 2.2+ . What could be the issue?

[GDLtechie01]

This is incorrect to some extent...I just updated my VBR server to version 9 update 4a and half of my agents didn't require an update, but they do require a reboot. Is there a reasonable explanation for all of this? Why is it not included in the upgrade checklist in this link https://helpcenter.veeam.com/docs/backu ... u4#upgrade

[DGrinev]

Hi,

Please review post by Dima P. above with explanations on the possible reasons. Thanks!

[jnaude]

Hi, We have upgraded Veeam from 4 to 4a. After update we also noticed that all Veeam Windows agents experienced pending reboot state, host rescan required state, agents aborting but in 99% not completing backup scheduling.

I have opened a case [Veeam Backup & Replication for Veeam agent backups behaves intermittently - Case # 03569591] . haven't received any feedback as to what is happing.

[Dima P.]

Hello and welcome to the community jnaude.

I'll ask support team to review your case details, stay tuned.

[spfs62]

I have upgraded from 9.5 to 10 and my Veeam agents for physical servers are broken. They said needed reboot, I did that and they are still failing.
I tried doing an uninstall through Veeam and re-install and that is failing "backup agent is missing"

I've tried the "Upgrade agent" and that fails

I will be putting in a service call, but any help would be appreciated

thanks

[Mildur]

There is a chance that with agent 4.0 (veeam v10) you will have todo 2 reboots of the same host.
First, veeam need to install .net 4.7.2 if it‘s not installed.
After that, the agent/cbt version will be upgraded.
Each of them need their own reboot process.

[Egor Yakovlev]

Hi Steve,

"Failing" is quite generic description. Can you please specify error from the session log in more details?

/Thanks!