Backup agent for Microsoft Windows servers and workstations (formerly Veeam Endpoint Backup FREE)
Post Reply
adb98
Enthusiast
Posts: 38
Liked: 4 times
Joined: Jul 21, 2016 5:03 pm
Full Name: Aaron B
Contact:

Agent Upgrade Process - Need to be fixed!!!

Post by adb98 » Jul 12, 2018 3:23 pm

The backup agent upgrade process has a nasty issue and not sure if it is a bug or if it is as designed. Either way they need to put wordage in the setup at the least when having the check mark for auto update client in a protection group. Here is the nasty issue I ran into.

I upgraded our main VBR server and proxies to the latest version yesterday morning. All was great and running ok. I come in this morning and half of my physical servers didn't backup last night and each of them state "Host Reboot required". WTH!!!!

I started going thru it and apparently the default for the protection groups is to check box " install agent automatically" and to "upgrade the agent automatically". Well that is fine if its not going to break my backups but it did. The protection groups did the once daily scan last night and saw hey I have an upgrade and upgraded but it requires a reboot in order to go thru and will no longer take backups until that is done. Then Veeam is just failing.

This is a very very very bad situation to be in. The worst part is we have one physical file server that is the main file share for 4 hospitals and is backed up every 2 hours. Now we are not getting any backups at all. Now we are scrambling and doing an emergency change (which directors have to approve and are asking why) and will be rebooting production servers into the night tonight in order resolve.

There needs to be some sort of logic to state. Hey I can't upgrade this automatically because it will break backups until a reboot is done. Even put a note (not recommended) and remove it from default setup for a protection group.

PTide
Veeam Software
Posts: 4431
Liked: 365 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Agent Upgrade Process - Need to be fixed!!!

Post by PTide » Jul 13, 2018 4:10 pm

Hi,

Thank you for bringing this up, your point has been fully understood. We will discuss that with the team internally and the thread will be updated accordingly with the resolution.

Thanks

Dima P.
Veeam Software
Posts: 8783
Liked: 647 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Agent Upgrade Process - Need to be fixed!!!

Post by Dima P. » Jul 15, 2018 10:55 pm

Aaron,

Thank you for your post! To clarify, were these servers running agent with CBT driver installed?

dschuler
Novice
Posts: 6
Liked: never
Joined: Aug 29, 2016 4:35 pm
Full Name: Dann Schuler
Contact:

Re: Agent Upgrade Process - Need to be fixed!!!

Post by dschuler » Jul 19, 2018 3:49 pm

We are seeing the same issue here. We upgraded our Veeam B&R to 9.5u3a last week and our two SQL clusters using the Veeam agents started throwing the "Host Reboot required" error. We have rebooted the hosts of one of the clusters and are still getting the same error.

We have not opened a ticket with Veeam yet as we are still troubleshooting internally.

dschuler
Novice
Posts: 6
Liked: never
Joined: Aug 29, 2016 4:35 pm
Full Name: Dann Schuler
Contact:

Re: Agent Upgrade Process - Need to be fixed!!!

Post by dschuler » Jul 19, 2018 4:16 pm

It looks like you need to perform a rescan on the protection group after the reboot before trying the jobs again. Ran a rescan and the Status reads Installed now and the job runs without the "Host Reboot required" error.

pdelacruz
Novice
Posts: 6
Liked: never
Joined: Jun 06, 2017 7:17 pm
Full Name: Pablo De La Cruz
Contact:

Re: Agent Upgrade Process - Need to be fixed!!!

Post by pdelacruz » Jul 19, 2018 5:16 pm

We also experienced the same issue. Even after I rebooted the servers, the backups were failing with same error. I then later realized that each protection group needed to be manually re-scan for the "required reboot" to clear. I think this needs to be fixed.

adb98
Enthusiast
Posts: 38
Liked: 4 times
Joined: Jul 21, 2016 5:03 pm
Full Name: Aaron B
Contact:

Re: Agent Upgrade Process - Need to be fixed!!!

Post by adb98 » Jul 23, 2018 7:58 pm

So your kinda in a damned if you do , damned if you don't situation with choosing to upgrade automatically. If you upgrade automatically there is a chance that it will stop you from being able to backup again if it upgrades the CBT driver, until you reboot the server being backed up and if choose not to upgrade automatically, it won't let you backup because it states it needs a upgrade.

This needs to be fixed so warns you that there is an available update but doesn't stop you from backing up.

I can understand that the workstation world it doesn't matter but in the server world with only one reboot a month.... This is a huge issue.

Dima P.
Veeam Software
Posts: 8783
Liked: 647 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Agent Upgrade Process - Need to be fixed!!!

Post by Dima P. » Jul 24, 2018 9:45 am

Thanks for your input. We will discuss with QA how to properly notify about the reboot requirement.

From technical standpoint there is no way to eliminate the reboot requirements when:

- You are upgrading CBT driver as a part of Veeam Agent for Windows upgrade. Drive must be unloaded from the OS thus reboot is a mandatory.
- OS is sitting in waiting for reboot state which is usually caused by Windows updates (such state blocks any product installations or upgrade).
- Core components for Windows agent must be changed. Usually it happens during major agent upgrade.

holden205
Novice
Posts: 7
Liked: never
Joined: Oct 27, 2017 4:18 pm
Full Name: Jeff Holden
Contact:

[MERGED] Update 3a impact on VAW agents with CBT driver

Post by holden205 » Jul 24, 2018 3:37 pm

After applying update 3a, all of my VAW physical servers [with CBT driver installed] were negatively affected. First, they immediately quit processing backups pending an unexpected reboot. Secondly, and most destructive, all restore points for each server were deleted without a trace. There is no longer any record of the restore points in the B&R console under Backups/Disk or in the repositories themselves. All servers having the CBT driver installed deleted existing restore points and created new backup chains after reboot. This was extremely painful when considering this impacted more than 50 servers with the loss of data protection and additional bandwidth costs to recreate fulls across the WAN. I have a case open with support, but it is most likely an unexpected result of update 3a on physical agents with no recovery of previous restore points. I verified this by the simple fact servers which I did not have the CBT driver installed were unaffected. They did not lose restore points and did not require reboot after updating. This is a major setback for us.

Also, to be clear, the release notes did not mention any impact to physical servers with Veeam Agent for Windows {with or without} the CBT driver installed.

Dima P.
Veeam Software
Posts: 8783
Liked: 647 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

[MERGED] Update 3a impact on VAW agents with CBT driver

Post by Dima P. » Jul 24, 2018 4:35 pm

Hello Jeff.

Thank you for your post. I can confirm that CBT driver upgrade (which comes with agent upgrade in this version) requires reboot. However, backup chain should remain untouched and this behavior is unexpected. May I ask you to submit a case ID and share it with me? Thank you!

holden205
Novice
Posts: 7
Liked: never
Joined: Oct 27, 2017 4:18 pm
Full Name: Jeff Holden
Contact:

Re: Agent Upgrade Process - Need to be fixed!!!

Post by holden205 » Jul 24, 2018 8:43 pm

The case number is 03107357. I'm out of my depth, but I believe that update 3a altered my restore points by eliminating them from the B&R console. Had I known I may have been able to repopulate them by rescanning the repositories. However, all backup jobs have a 3 day retention on deleted items. Since i discovered the missing restore points on the forth day after update, the repositories deleted all backup files associated with the affected servers. My case is slowly working its way up the tiers. But, it is already to late to recover backup data aside from what may be on GFS tape.

Post Reply

Who is online

Users browsing this forum: Baidu [Spider] and 10 guests