-
- Enthusiast
- Posts: 63
- Liked: 13 times
- Joined: Jul 21, 2016 5:03 pm
- Full Name: Aaron B
- Contact:
Agent Upgrade Process - Need to be fixed!!!
The backup agent upgrade process has a nasty issue and not sure if it is a bug or if it is as designed. Either way they need to put wordage in the setup at the least when having the check mark for auto update client in a protection group. Here is the nasty issue I ran into.
I upgraded our main VBR server and proxies to the latest version yesterday morning. All was great and running ok. I come in this morning and half of my physical servers didn't backup last night and each of them state "Host Reboot required". WTH!!!!
I started going thru it and apparently the default for the protection groups is to check box " install agent automatically" and to "upgrade the agent automatically". Well that is fine if its not going to break my backups but it did. The protection groups did the once daily scan last night and saw hey I have an upgrade and upgraded but it requires a reboot in order to go thru and will no longer take backups until that is done. Then Veeam is just failing.
This is a very very very bad situation to be in. The worst part is we have one physical file server that is the main file share for 4 hospitals and is backed up every 2 hours. Now we are not getting any backups at all. Now we are scrambling and doing an emergency change (which directors have to approve and are asking why) and will be rebooting production servers into the night tonight in order resolve.
There needs to be some sort of logic to state. Hey I can't upgrade this automatically because it will break backups until a reboot is done. Even put a note (not recommended) and remove it from default setup for a protection group.
I upgraded our main VBR server and proxies to the latest version yesterday morning. All was great and running ok. I come in this morning and half of my physical servers didn't backup last night and each of them state "Host Reboot required". WTH!!!!
I started going thru it and apparently the default for the protection groups is to check box " install agent automatically" and to "upgrade the agent automatically". Well that is fine if its not going to break my backups but it did. The protection groups did the once daily scan last night and saw hey I have an upgrade and upgraded but it requires a reboot in order to go thru and will no longer take backups until that is done. Then Veeam is just failing.
This is a very very very bad situation to be in. The worst part is we have one physical file server that is the main file share for 4 hospitals and is backed up every 2 hours. Now we are not getting any backups at all. Now we are scrambling and doing an emergency change (which directors have to approve and are asking why) and will be rebooting production servers into the night tonight in order resolve.
There needs to be some sort of logic to state. Hey I can't upgrade this automatically because it will break backups until a reboot is done. Even put a note (not recommended) and remove it from default setup for a protection group.
-
- Product Manager
- Posts: 6551
- Liked: 765 times
- Joined: May 19, 2015 1:46 pm
- Contact:
Re: Agent Upgrade Process - Need to be fixed!!!
Hi,
Thank you for bringing this up, your point has been fully understood. We will discuss that with the team internally and the thread will be updated accordingly with the resolution.
Thanks
Thank you for bringing this up, your point has been fully understood. We will discuss that with the team internally and the thread will be updated accordingly with the resolution.
Thanks
-
- Product Manager
- Posts: 14726
- Liked: 1707 times
- Joined: Feb 04, 2013 2:07 pm
- Full Name: Dmitry Popov
- Location: Prague
- Contact:
Re: Agent Upgrade Process - Need to be fixed!!!
Aaron,
Thank you for your post! To clarify, were these servers running agent with CBT driver installed?
Thank you for your post! To clarify, were these servers running agent with CBT driver installed?
-
- Novice
- Posts: 8
- Liked: never
- Joined: Aug 29, 2016 4:35 pm
- Full Name: Dann Schuler
- Contact:
Re: Agent Upgrade Process - Need to be fixed!!!
We are seeing the same issue here. We upgraded our Veeam B&R to 9.5u3a last week and our two SQL clusters using the Veeam agents started throwing the "Host Reboot required" error. We have rebooted the hosts of one of the clusters and are still getting the same error.
We have not opened a ticket with Veeam yet as we are still troubleshooting internally.
We have not opened a ticket with Veeam yet as we are still troubleshooting internally.
-
- Novice
- Posts: 8
- Liked: never
- Joined: Aug 29, 2016 4:35 pm
- Full Name: Dann Schuler
- Contact:
Re: Agent Upgrade Process - Need to be fixed!!!
It looks like you need to perform a rescan on the protection group after the reboot before trying the jobs again. Ran a rescan and the Status reads Installed now and the job runs without the "Host Reboot required" error.
-
- Novice
- Posts: 7
- Liked: 1 time
- Joined: Jun 06, 2017 7:17 pm
- Full Name: Pablo De La Cruz
- Contact:
Re: Agent Upgrade Process - Need to be fixed!!!
We also experienced the same issue. Even after I rebooted the servers, the backups were failing with same error. I then later realized that each protection group needed to be manually re-scan for the "required reboot" to clear. I think this needs to be fixed.
-
- Enthusiast
- Posts: 63
- Liked: 13 times
- Joined: Jul 21, 2016 5:03 pm
- Full Name: Aaron B
- Contact:
Re: Agent Upgrade Process - Need to be fixed!!!
So your kinda in a damned if you do , damned if you don't situation with choosing to upgrade automatically. If you upgrade automatically there is a chance that it will stop you from being able to backup again if it upgrades the CBT driver, until you reboot the server being backed up and if choose not to upgrade automatically, it won't let you backup because it states it needs a upgrade.
This needs to be fixed so warns you that there is an available update but doesn't stop you from backing up.
I can understand that the workstation world it doesn't matter but in the server world with only one reboot a month.... This is a huge issue.
This needs to be fixed so warns you that there is an available update but doesn't stop you from backing up.
I can understand that the workstation world it doesn't matter but in the server world with only one reboot a month.... This is a huge issue.
-
- Product Manager
- Posts: 14726
- Liked: 1707 times
- Joined: Feb 04, 2013 2:07 pm
- Full Name: Dmitry Popov
- Location: Prague
- Contact:
Re: Agent Upgrade Process - Need to be fixed!!!
Thanks for your input. We will discuss with QA how to properly notify about the reboot requirement.
From technical standpoint there is no way to eliminate the reboot requirements when:
- You are upgrading CBT driver as a part of Veeam Agent for Windows upgrade. Drive must be unloaded from the OS thus reboot is a mandatory.
- OS is sitting in waiting for reboot state which is usually caused by Windows updates (such state blocks any product installations or upgrade).
- Core components for Windows agent must be changed. Usually it happens during major agent upgrade.
From technical standpoint there is no way to eliminate the reboot requirements when:
- You are upgrading CBT driver as a part of Veeam Agent for Windows upgrade. Drive must be unloaded from the OS thus reboot is a mandatory.
- OS is sitting in waiting for reboot state which is usually caused by Windows updates (such state blocks any product installations or upgrade).
- Core components for Windows agent must be changed. Usually it happens during major agent upgrade.
-
- Novice
- Posts: 7
- Liked: 1 time
- Joined: Oct 27, 2017 4:18 pm
- Full Name: Jeff Holden
- Contact:
[MERGED] Update 3a impact on VAW agents with CBT driver
After applying update 3a, all of my VAW physical servers [with CBT driver installed] were negatively affected. First, they immediately quit processing backups pending an unexpected reboot. Secondly, and most destructive, all restore points for each server were deleted without a trace. There is no longer any record of the restore points in the B&R console under Backups/Disk or in the repositories themselves. All servers having the CBT driver installed deleted existing restore points and created new backup chains after reboot. This was extremely painful when considering this impacted more than 50 servers with the loss of data protection and additional bandwidth costs to recreate fulls across the WAN. I have a case open with support, but it is most likely an unexpected result of update 3a on physical agents with no recovery of previous restore points. I verified this by the simple fact servers which I did not have the CBT driver installed were unaffected. They did not lose restore points and did not require reboot after updating. This is a major setback for us.
Also, to be clear, the release notes did not mention any impact to physical servers with Veeam Agent for Windows {with or without} the CBT driver installed.
Also, to be clear, the release notes did not mention any impact to physical servers with Veeam Agent for Windows {with or without} the CBT driver installed.
-
- Product Manager
- Posts: 14726
- Liked: 1707 times
- Joined: Feb 04, 2013 2:07 pm
- Full Name: Dmitry Popov
- Location: Prague
- Contact:
[MERGED] Update 3a impact on VAW agents with CBT driver
Hello Jeff.
Thank you for your post. I can confirm that CBT driver upgrade (which comes with agent upgrade in this version) requires reboot. However, backup chain should remain untouched and this behavior is unexpected. May I ask you to submit a case ID and share it with me? Thank you!
Thank you for your post. I can confirm that CBT driver upgrade (which comes with agent upgrade in this version) requires reboot. However, backup chain should remain untouched and this behavior is unexpected. May I ask you to submit a case ID and share it with me? Thank you!
-
- Novice
- Posts: 7
- Liked: 1 time
- Joined: Oct 27, 2017 4:18 pm
- Full Name: Jeff Holden
- Contact:
Re: Agent Upgrade Process - Need to be fixed!!!
The case number is 03107357. I'm out of my depth, but I believe that update 3a altered my restore points by eliminating them from the B&R console. Had I known I may have been able to repopulate them by rescanning the repositories. However, all backup jobs have a 3 day retention on deleted items. Since i discovered the missing restore points on the forth day after update, the repositories deleted all backup files associated with the affected servers. My case is slowly working its way up the tiers. But, it is already to late to recover backup data aside from what may be on GFS tape.
-
- Influencer
- Posts: 13
- Liked: 4 times
- Joined: Nov 01, 2018 8:32 pm
- Contact:
Re: Agent Upgrade Process - Need to be fixed!!!
Thank you. I was having the same issue and a rescan did the trick. It would be nice to have that automated after the reboot....
-
- Product Manager
- Posts: 14726
- Liked: 1707 times
- Joined: Feb 04, 2013 2:07 pm
- Full Name: Dmitry Popov
- Location: Prague
- Contact:
Re: Agent Upgrade Process - Need to be fixed!!!
Hi folks,
We've implemented a special logic to the CBT driver distributed via B&R server in Update 4: once you upgrade your agent with new version of the CBT driver it will await for the reboot, but to make sure backup job will run without any issues will use the previously installed components.
Reboot might be still required by Windows components or other installed application (OS can sit in pending for reboot state and that would require reboot), but that's not related to driver update any longer with agent managed by Veeam B&R. Cheers!
We've implemented a special logic to the CBT driver distributed via B&R server in Update 4: once you upgrade your agent with new version of the CBT driver it will await for the reboot, but to make sure backup job will run without any issues will use the previously installed components.
Reboot might be still required by Windows components or other installed application (OS can sit in pending for reboot state and that would require reboot), but that's not related to driver update any longer with agent managed by Veeam B&R. Cheers!
-
- Service Provider
- Posts: 327
- Liked: 23 times
- Joined: Oct 09, 2012 2:30 pm
- Full Name: Maso
- Contact:
[MERGED] 9.5 Update 4a
Hi
Plan to upgrade vbr server from U4 to U4a. Connected protection group server are running 3.0.0.748 (Windows) and 3.0.0.865 (Linux). Will U4a require an agent update?
\Masonit
Plan to upgrade vbr server from U4 to U4a. Connected protection group server are running 3.0.0.748 (Windows) and 3.0.0.865 (Linux). Will U4a require an agent update?
\Masonit
-
- Veteran
- Posts: 1943
- Liked: 247 times
- Joined: Dec 01, 2016 3:49 pm
- Full Name: Dmitry Grinev
- Location: St.Petersburg
- Contact:
Re: 9.5 Update 4a
Hi Masonit,
We have no such requirement for Agents to work with Veeam B&R U4a. Thanks!
We have no such requirement for Agents to work with Veeam B&R U4a. Thanks!
-
- Service Provider
- Posts: 327
- Liked: 23 times
- Joined: Oct 09, 2012 2:30 pm
- Full Name: Maso
- Contact:
Re: 9.5 Update 4a
Thanks!
-
- Veteran
- Posts: 1943
- Liked: 247 times
- Joined: Dec 01, 2016 3:49 pm
- Full Name: Dmitry Grinev
- Location: St.Petersburg
- Contact:
Re: 9.5 Update 4a
Please keep in mind that VBR will propose you to update the Agent to the most recent version which is 3.1, but it's not mandatory for work.
Thanks!
Thanks!
-
- Enthusiast
- Posts: 28
- Liked: 10 times
- Joined: Jun 18, 2018 2:41 am
- Contact:
-
- Novice
- Posts: 3
- Liked: never
- Joined: Oct 04, 2018 3:52 pm
- Full Name: GDL Tech Team
- Contact:
Re: Agent Upgrade Process - Need to be fixed!!!
I just upgraded to VBR Version 9 update 4a, and about half of my agents are requiring a reboot (with failed backups last night!). Agent versions 2.2+ . What could be the issue?
-
- Novice
- Posts: 3
- Liked: never
- Joined: Oct 04, 2018 3:52 pm
- Full Name: GDL Tech Team
- Contact:
Re: 9.5 Update 4a
This is incorrect to some extent...I just updated my VBR server to version 9 update 4a and half of my agents didn't require an update, but they do require a reboot. Is there a reasonable explanation for all of this? Why is it not included in the upgrade checklist in this link https://helpcenter.veeam.com/docs/backu ... u4#upgrade
-
- Veteran
- Posts: 1943
- Liked: 247 times
- Joined: Dec 01, 2016 3:49 pm
- Full Name: Dmitry Grinev
- Location: St.Petersburg
- Contact:
-
- Lurker
- Posts: 2
- Liked: never
- Joined: Mar 26, 2019 8:39 am
- Full Name: Jaco Naude
- Contact:
Re: Agent Upgrade Process - Need to be fixed!!!
Hi, We have upgraded Veeam from 4 to 4a. After update we also noticed that all Veeam Windows agents experienced pending reboot state, host rescan required state, agents aborting but in 99% not completing backup scheduling.
I have opened a case [Veeam Backup & Replication for Veeam agent backups behaves intermittently - Case # 03569591] . haven't received any feedback as to what is happing.
I have opened a case [Veeam Backup & Replication for Veeam agent backups behaves intermittently - Case # 03569591] . haven't received any feedback as to what is happing.
-
- Product Manager
- Posts: 14726
- Liked: 1707 times
- Joined: Feb 04, 2013 2:07 pm
- Full Name: Dmitry Popov
- Location: Prague
- Contact:
Re: Agent Upgrade Process - Need to be fixed!!!
Hello and welcome to the community jnaude.
I'll ask support team to review your case details, stay tuned.
I'll ask support team to review your case details, stay tuned.
-
- Influencer
- Posts: 11
- Liked: never
- Joined: Sep 16, 2020 6:14 pm
- Full Name: Steve Fritsch
- Contact:
Re: Agent Upgrade Process - Need to be fixed!!!
I have upgraded from 9.5 to 10 and my Veeam agents for physical servers are broken. They said needed reboot, I did that and they are still failing.
I tried doing an uninstall through Veeam and re-install and that is failing "backup agent is missing"
I've tried the "Upgrade agent" and that fails
I will be putting in a service call, but any help would be appreciated
thanks
I tried doing an uninstall through Veeam and re-install and that is failing "backup agent is missing"
I've tried the "Upgrade agent" and that fails
I will be putting in a service call, but any help would be appreciated
thanks
-
- Product Manager
- Posts: 9848
- Liked: 2607 times
- Joined: May 13, 2017 4:51 pm
- Full Name: Fabian K.
- Location: Switzerland
- Contact:
Re: Agent Upgrade Process - Need to be fixed!!!
There is a chance that with agent 4.0 (veeam v10) you will have todo 2 reboots of the same host.
First, veeam need to install .net 4.7.2 if it‘s not installed.
After that, the agent/cbt version will be upgraded.
Each of them need their own reboot process.
First, veeam need to install .net 4.7.2 if it‘s not installed.
After that, the agent/cbt version will be upgraded.
Each of them need their own reboot process.
Product Management Analyst @ Veeam Software
-
- Product Manager
- Posts: 2581
- Liked: 708 times
- Joined: Jun 14, 2013 9:30 am
- Full Name: Egor Yakovlev
- Location: Prague, Czech Republic
- Contact:
Re: Agent Upgrade Process - Need to be fixed!!!
Hi Steve,
"Failing" is quite generic description. Can you please specify error from the session log in more details?
/Thanks!
"Failing" is quite generic description. Can you please specify error from the session log in more details?
/Thanks!
Who is online
Users browsing this forum: Baidu [Spider], Majestic-12 [Bot] and 17 guests