Standalone backup agent for Microsoft Windows servers and workstations (formerly Veeam Endpoint Backup FREE)
Post Reply
adb98
Enthusiast
Posts: 42
Liked: 4 times
Joined: Jul 21, 2016 5:03 pm
Full Name: Aaron B
Contact:

Agent Upgrade Process - Need to be fixed!!!

Post by adb98 » Jul 12, 2018 3:23 pm

The backup agent upgrade process has a nasty issue and not sure if it is a bug or if it is as designed. Either way they need to put wordage in the setup at the least when having the check mark for auto update client in a protection group. Here is the nasty issue I ran into.

I upgraded our main VBR server and proxies to the latest version yesterday morning. All was great and running ok. I come in this morning and half of my physical servers didn't backup last night and each of them state "Host Reboot required". WTH!!!!

I started going thru it and apparently the default for the protection groups is to check box " install agent automatically" and to "upgrade the agent automatically". Well that is fine if its not going to break my backups but it did. The protection groups did the once daily scan last night and saw hey I have an upgrade and upgraded but it requires a reboot in order to go thru and will no longer take backups until that is done. Then Veeam is just failing.

This is a very very very bad situation to be in. The worst part is we have one physical file server that is the main file share for 4 hospitals and is backed up every 2 hours. Now we are not getting any backups at all. Now we are scrambling and doing an emergency change (which directors have to approve and are asking why) and will be rebooting production servers into the night tonight in order resolve.

There needs to be some sort of logic to state. Hey I can't upgrade this automatically because it will break backups until a reboot is done. Even put a note (not recommended) and remove it from default setup for a protection group.

P.Tide
Product Manager
Posts: 5148
Liked: 444 times
Joined: May 19, 2015 1:46 pm
Contact:

Re: Agent Upgrade Process - Need to be fixed!!!

Post by P.Tide » Jul 13, 2018 4:10 pm

Hi,

Thank you for bringing this up, your point has been fully understood. We will discuss that with the team internally and the thread will be updated accordingly with the resolution.

Thanks

Dima P.
Product Manager
Posts: 10148
Liked: 817 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Agent Upgrade Process - Need to be fixed!!!

Post by Dima P. » Jul 15, 2018 10:55 pm

Aaron,

Thank you for your post! To clarify, were these servers running agent with CBT driver installed?

dschuler
Novice
Posts: 6
Liked: never
Joined: Aug 29, 2016 4:35 pm
Full Name: Dann Schuler
Contact:

Re: Agent Upgrade Process - Need to be fixed!!!

Post by dschuler » Jul 19, 2018 3:49 pm

We are seeing the same issue here. We upgraded our Veeam B&R to 9.5u3a last week and our two SQL clusters using the Veeam agents started throwing the "Host Reboot required" error. We have rebooted the hosts of one of the clusters and are still getting the same error.

We have not opened a ticket with Veeam yet as we are still troubleshooting internally.

dschuler
Novice
Posts: 6
Liked: never
Joined: Aug 29, 2016 4:35 pm
Full Name: Dann Schuler
Contact:

Re: Agent Upgrade Process - Need to be fixed!!!

Post by dschuler » Jul 19, 2018 4:16 pm

It looks like you need to perform a rescan on the protection group after the reboot before trying the jobs again. Ran a rescan and the Status reads Installed now and the job runs without the "Host Reboot required" error.

pdelacruz
Novice
Posts: 6
Liked: 1 time
Joined: Jun 06, 2017 7:17 pm
Full Name: Pablo De La Cruz
Contact:

Re: Agent Upgrade Process - Need to be fixed!!!

Post by pdelacruz » Jul 19, 2018 5:16 pm 1 person likes this post

We also experienced the same issue. Even after I rebooted the servers, the backups were failing with same error. I then later realized that each protection group needed to be manually re-scan for the "required reboot" to clear. I think this needs to be fixed.

adb98
Enthusiast
Posts: 42
Liked: 4 times
Joined: Jul 21, 2016 5:03 pm
Full Name: Aaron B
Contact:

Re: Agent Upgrade Process - Need to be fixed!!!

Post by adb98 » Jul 23, 2018 7:58 pm

So your kinda in a damned if you do , damned if you don't situation with choosing to upgrade automatically. If you upgrade automatically there is a chance that it will stop you from being able to backup again if it upgrades the CBT driver, until you reboot the server being backed up and if choose not to upgrade automatically, it won't let you backup because it states it needs a upgrade.

This needs to be fixed so warns you that there is an available update but doesn't stop you from backing up.

I can understand that the workstation world it doesn't matter but in the server world with only one reboot a month.... This is a huge issue.

Dima P.
Product Manager
Posts: 10148
Liked: 817 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Agent Upgrade Process - Need to be fixed!!!

Post by Dima P. » Jul 24, 2018 9:45 am

Thanks for your input. We will discuss with QA how to properly notify about the reboot requirement.

From technical standpoint there is no way to eliminate the reboot requirements when:

- You are upgrading CBT driver as a part of Veeam Agent for Windows upgrade. Drive must be unloaded from the OS thus reboot is a mandatory.
- OS is sitting in waiting for reboot state which is usually caused by Windows updates (such state blocks any product installations or upgrade).
- Core components for Windows agent must be changed. Usually it happens during major agent upgrade.

holden205
Novice
Posts: 7
Liked: never
Joined: Oct 27, 2017 4:18 pm
Full Name: Jeff Holden
Contact:

[MERGED] Update 3a impact on VAW agents with CBT driver

Post by holden205 » Jul 24, 2018 3:37 pm

After applying update 3a, all of my VAW physical servers [with CBT driver installed] were negatively affected. First, they immediately quit processing backups pending an unexpected reboot. Secondly, and most destructive, all restore points for each server were deleted without a trace. There is no longer any record of the restore points in the B&R console under Backups/Disk or in the repositories themselves. All servers having the CBT driver installed deleted existing restore points and created new backup chains after reboot. This was extremely painful when considering this impacted more than 50 servers with the loss of data protection and additional bandwidth costs to recreate fulls across the WAN. I have a case open with support, but it is most likely an unexpected result of update 3a on physical agents with no recovery of previous restore points. I verified this by the simple fact servers which I did not have the CBT driver installed were unaffected. They did not lose restore points and did not require reboot after updating. This is a major setback for us.

Also, to be clear, the release notes did not mention any impact to physical servers with Veeam Agent for Windows {with or without} the CBT driver installed.

Dima P.
Product Manager
Posts: 10148
Liked: 817 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

[MERGED] Update 3a impact on VAW agents with CBT driver

Post by Dima P. » Jul 24, 2018 4:35 pm

Hello Jeff.

Thank you for your post. I can confirm that CBT driver upgrade (which comes with agent upgrade in this version) requires reboot. However, backup chain should remain untouched and this behavior is unexpected. May I ask you to submit a case ID and share it with me? Thank you!

holden205
Novice
Posts: 7
Liked: never
Joined: Oct 27, 2017 4:18 pm
Full Name: Jeff Holden
Contact:

Re: Agent Upgrade Process - Need to be fixed!!!

Post by holden205 » Jul 24, 2018 8:43 pm

The case number is 03107357. I'm out of my depth, but I believe that update 3a altered my restore points by eliminating them from the B&R console. Had I known I may have been able to repopulate them by rescanning the repositories. However, all backup jobs have a 3 day retention on deleted items. Since i discovered the missing restore points on the forth day after update, the repositories deleted all backup files associated with the affected servers. My case is slowly working its way up the tiers. But, it is already to late to recover backup data aside from what may be on GFS tape.

Frenchyaz
Influencer
Posts: 12
Liked: 4 times
Joined: Nov 01, 2018 8:32 pm
Contact:

Re: Agent Upgrade Process - Need to be fixed!!!

Post by Frenchyaz » Dec 06, 2018 5:45 pm

pdelacruz wrote:
Jul 19, 2018 5:16 pm
We also experienced the same issue. Even after I rebooted the servers, the backups were failing with same error. I then later realized that each protection group needed to be manually re-scan for the "required reboot" to clear. I think this needs to be fixed.
Thank you. I was having the same issue and a rescan did the trick. It would be nice to have that automated after the reboot....

Dima P.
Product Manager
Posts: 10148
Liked: 817 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Agent Upgrade Process - Need to be fixed!!!

Post by Dima P. » Dec 07, 2018 11:55 am

Hi folks,

We've implemented a special logic to the CBT driver distributed via B&R server in Update 4: once you upgrade your agent with new version of the CBT driver it will await for the reboot, but to make sure backup job will run without any issues will use the previously installed components.

Reboot might be still required by Windows components or other installed application (OS can sit in pending for reboot state and that would require reboot), but that's not related to driver update any longer with agent managed by Veeam B&R. Cheers!

masonit
Service Provider
Posts: 212
Liked: 14 times
Joined: Oct 09, 2012 2:30 pm
Full Name: Magnus
Contact:

[MERGED] 9.5 Update 4a

Post by masonit » May 15, 2019 7:00 am

Hi

Plan to upgrade vbr server from U4 to U4a. Connected protection group server are running 3.0.0.748 (Windows) and 3.0.0.865 (Linux). Will U4a require an agent update?

\Masonit

DGrinev
Veeam Software
Posts: 1650
Liked: 201 times
Joined: Dec 01, 2016 3:49 pm
Full Name: Dmitry Grinev
Location: St.Petersburg
Contact:

Re: 9.5 Update 4a

Post by DGrinev » May 15, 2019 10:25 am

Hi Masonit,

We have no such requirement for Agents to work with Veeam B&R U4a. Thanks!

masonit
Service Provider
Posts: 212
Liked: 14 times
Joined: Oct 09, 2012 2:30 pm
Full Name: Magnus
Contact:

Re: 9.5 Update 4a

Post by masonit » May 15, 2019 11:28 am

Thanks!

DGrinev
Veeam Software
Posts: 1650
Liked: 201 times
Joined: Dec 01, 2016 3:49 pm
Full Name: Dmitry Grinev
Location: St.Petersburg
Contact:

Re: 9.5 Update 4a

Post by DGrinev » May 15, 2019 11:45 am

Please keep in mind that VBR will propose you to update the Agent to the most recent version which is 3.1, but it's not mandatory for work.

Thanks!

jdrch
Influencer
Posts: 20
Liked: 8 times
Joined: Jun 18, 2018 2:41 am
Contact:

Re: 9.5 Update 4a

Post by jdrch » May 15, 2019 10:05 pm 1 person likes this post

masonit wrote:
May 15, 2019 7:00 am
Hi

Plan to upgrade vbr server from U4 to U4a. Connected protection group server are running 3.0.0.748 (Windows) and 3.0.0.865 (Linux). Will U4a require an agent update?

\Masonit
You should definitely be running the latest Agent builds anyway.

GDLtechie01
Service Provider
Posts: 3
Liked: never
Joined: Oct 04, 2018 3:52 pm
Full Name: GDL Tech Team
Contact:

Re: Agent Upgrade Process - Need to be fixed!!!

Post by GDLtechie01 » May 17, 2019 1:29 pm

I just upgraded to VBR Version 9 update 4a, and about half of my agents are requiring a reboot (with failed backups last night!). Agent versions 2.2+ . What could be the issue?

GDLtechie01
Service Provider
Posts: 3
Liked: never
Joined: Oct 04, 2018 3:52 pm
Full Name: GDL Tech Team
Contact:

Re: 9.5 Update 4a

Post by GDLtechie01 » May 17, 2019 1:32 pm

This is incorrect to some extent...I just updated my VBR server to version 9 update 4a and half of my agents didn't require an update, but they do require a reboot. Is there a reasonable explanation for all of this? Why is it not included in the upgrade checklist in this link https://helpcenter.veeam.com/docs/backu ... u4#upgrade

DGrinev
Veeam Software
Posts: 1650
Liked: 201 times
Joined: Dec 01, 2016 3:49 pm
Full Name: Dmitry Grinev
Location: St.Petersburg
Contact:

Re: 9.5 Update 4a

Post by DGrinev » May 17, 2019 2:09 pm

Hi,

Please review post by Dima P. above with explanations on the possible reasons. Thanks!

jnaude
Lurker
Posts: 1
Liked: never
Joined: Mar 26, 2019 8:39 am
Full Name: Jaco Naude
Contact:

Re: Agent Upgrade Process - Need to be fixed!!!

Post by jnaude » May 18, 2019 5:21 am

Hi, We have upgraded Veeam from 4 to 4a. After update we also noticed that all Veeam Windows agents experienced pending reboot state, host rescan required state, agents aborting but in 99% not completing backup scheduling.

I have opened a case [Veeam Backup & Replication for Veeam agent backups behaves intermittently - Case # 03569591] . haven't received any feedback as to what is happing.

Dima P.
Product Manager
Posts: 10148
Liked: 817 times
Joined: Feb 04, 2013 2:07 pm
Full Name: Dmitry Popov
Location: Prague
Contact:

Re: Agent Upgrade Process - Need to be fixed!!!

Post by Dima P. » May 20, 2019 2:10 pm

Hello and welcome to the community jnaude.

I'll ask support team to review your case details, stay tuned.

Post Reply

Who is online

Users browsing this forum: Exabot [Bot] and 17 guests