Comprehensive data protection for all workloads
Post Reply
sonobob
Lurker
Posts: 1
Liked: never
Joined: Jul 08, 2025 2:13 pm
Full Name: Chris
Contact:

[VHRISO] Veeam Hardened Repos unavailable after automatic update

Post by sonobob »

Hi,

we are currently using 30 hardened Veeam repos that were created with the official Veeam Hardened Repository ISO. After yesterdays automatic Monday patching, that they seem to do, all of them became unavailable in VBR and we had to reboot each of them to get them up and running again. Four of which we had to do a repair install and one we even had to run xfs_repair towards the backup storage.

Would it be possible to make this patching optional? We Plan to add another 20 and thinking this can potentially occur every Monday is a big concern.

Thanks!
HannesK
Product Manager
Posts: 15339
Liked: 3321 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: [VHRISO] Veeam Hardened Repos unavailable after automatic update

Post by HannesK »

Hello,
and welcome to the forums. That sounds like a (big) technical issue. Can you please provide a support case ID for this issue, as requested when you click New Topic.

In Hardened Repository ISO 2.0 there is no option to direct option control the updates. There are unsupported options, but let's first see what support finds out. The symptoms do not match any known issues.

Best regards
Hannes
EviLin
Influencer
Posts: 16
Liked: 2 times
Joined: Apr 28, 2025 7:37 pm
Contact:

Re: [VHRISO] Veeam Hardened Repos unavailable after automatic update

Post by EviLin » 2 people like this post

We have 1 VHR, and experienced the same thing yesterday, it was unavailable in VBR, but after a reboot of VHR it came it was OK.

I think I read it some where, that you have to reboot the VHR server once in a while, Even after succesfully installing latest updates it's missing a reboot feature.
HannesK
Product Manager
Posts: 15339
Liked: 3321 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: [VHRISO] Veeam Hardened Repos unavailable after automatic update

Post by HannesK » 2 people like this post

yes, update management, reboot, defining a maintenance window is something for the V13 ISO which includes a component called "Veeam Updater" which manages all these things.
johnnytan
Service Provider
Posts: 9
Liked: 3 times
Joined: Oct 09, 2023 3:52 pm
Full Name: Johnny Tan
Contact:

[MERGED] [RELEASE] Managed Hardened Repository ISO by Veeam

Post by johnnytan » 1 person likes this post

Hi guys, we have an ongoing case with Veeam Support now.

The issue was that the hardened repository suddenly disconnected last Sunday/Saturday. This happened to two deployment on a different client and it happen at the same time.
1 of the hardened repository was deployed recently only (1 month++) and we performed a reboot, and re-configure the hardened repository and it managed to remove both veeamdeployment and veeamtransport services and reinstall it back. This was solved.

The other client was deployed more than 3 months ago. We did the same thing and it just stuck at testing veeam transport. This is what we observed.
1. During re-installation, the veeamdeployment service is running and listening to port 6160. Meanwhile the veeamtransport service is running (based on systemctl status veeamtransport) but it is not listening to port 6162.
2. There's no network/firewall blockage. Port 6160 are 'telnet-able' from VBR to the Hardened Repository.

We have a lots other hardened repository running, but not from this recent ISO. It was prebuilt back then with installaton of bare Ubuntu and running the script to hardened back then which is downloaded from github community and it's still running like a beast.

For now we are still waiting for Veeam Support to reach back to us.
HannesK
Product Manager
Posts: 15339
Liked: 3321 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: [VHRISO] Veeam Hardened Repos unavailable after automatic update

Post by HannesK »

Hello,
I moved your post to another thread that mentions similar issues. Can you please share the support case number so that we can look into it?

Thanks,
Hannes
johnnytan
Service Provider
Posts: 9
Liked: 3 times
Joined: Oct 09, 2023 3:52 pm
Full Name: Johnny Tan
Contact:

Re: [VHRISO] Veeam Hardened Repos unavailable after automatic update

Post by johnnytan »

Hi Hannes,

This are my Case #07757523

Thanks in looking in. Appreciate the effort
ASG
Enthusiast
Posts: 79
Liked: 7 times
Joined: Aug 08, 2018 10:19 am
Contact:

Re: [VHRISO] Veeam Hardened Repos unavailable after automatic update

Post by ASG » 2 people like this post

We had the same problem (machine was up but unavailable to VBR) - reboot, problem solved. We did not create a ticket because hiccups happen, however I wanted to +1 that we experienced the same problem at the same time like some others here.
HannesK
Product Manager
Posts: 15339
Liked: 3321 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: [VHRISO] Veeam Hardened Repos unavailable after automatic update

Post by HannesK »

Hello,
thanks for the ticket number. I contacted support and hope that we can find the root cause from the logs. I also emphasized to support again that it's about Hardened Repository ISO and that they cannot simply request "root" commands from you.

In the other thread there was the suspicion that a networkmanager update caused the connection outage and I see that there was a new network manager package.

In any case, an update should not cause such big problems that repair / re-install is necessary or that XFS needs a repair (well, I don't know anything about the underlying hardware, maybe that caused XFS issues)

Best regards
Hannes
Dynamic
Veeam Vanguard
Posts: 40
Liked: 9 times
Joined: Mar 02, 2017 2:43 pm
Full Name: Markus Hartmann
Contact:

Re: [VHRISO] Veeam Hardened Repos unavailable after automatic update

Post by Dynamic »

Hi there,
we had the same issue, multiple and different environments.
Just found this thread.
After a reboot and rescan, it’s working again.
Markus Hartmann | Veeam Vanguard | Veeam Legend 2024 | VMCA 2024 & VMCE 2024 | VMware Certified Implementation Expert - Data Center Virtualization 2024 | https://markushartmann.blog/
johnnytan
Service Provider
Posts: 9
Liked: 3 times
Joined: Oct 09, 2023 3:52 pm
Full Name: Johnny Tan
Contact:

Re: [VHRISO] Veeam Hardened Repos unavailable after automatic update

Post by johnnytan »

HannesK wrote: Jul 09, 2025 6:54 am Hello,
thanks for the ticket number. I contacted support and hope that we can find the root cause from the logs. I also emphasized to support again that it's about Hardened Repository ISO and that they cannot simply request "root" commands from you.

In the other thread there was the suspicion that a networkmanager update caused the connection outage and I see that there was a new network manager package.

In any case, an update should not cause such big problems that repair / re-install is necessary or that XFS needs a repair (well, I don't know anything about the underlying hardware, maybe that caused XFS issues)

Best regards
Hannes
Thanks for the heads up.
Being more than a decade's old system engineer, them keep asking for "root" password and adding user into "sudo" really frustrate me a lot.

Being that, I've taken my own ways and zip up the VeeamBackup logs to be attached in the ticket as well.

One thing to note, VeeamTransportSvc.log seems like grow unexpectedly large(12++ of Gigs) and not being rotated properly.
1dna
Influencer
Posts: 12
Liked: never
Joined: Jan 28, 2025 6:29 am
Full Name: Lars Knakkergaard
Contact:

Re: [VHRISO] Veeam Hardened Repos unavailable after automatic update

Post by 1dna »

Hi

we are experincing the exact same Case #07756166
1dna
Influencer
Posts: 12
Liked: never
Joined: Jan 28, 2025 6:29 am
Full Name: Lars Knakkergaard
Contact:

Re: [VHRISO] Veeam Hardened Repos unavailable after automatic update

Post by 1dna »

Anyone has found a fix or workaround to keep them online ?
Backup.Operator
Enthusiast
Posts: 96
Liked: 6 times
Joined: Oct 31, 2022 11:39 pm
Full Name: Backup Administrator
Contact:

Re: [VHRISO] Veeam Hardened Repos unavailable after automatic update

Post by Backup.Operator » 1 person likes this post

HannesK wrote: Jul 08, 2025 4:44 pm yes, update management, reboot, defining a maintenance window is something for the V13 ISO which includes a component called "Veeam Updater" which manages all these things.
OK, so in this case, for the Ubuntu Server Linux VHR, can we still perform the manual update and upgrade ?

Code: Select all

apt update
apt full-upgrade
do-release-upgrade
:arrow: :mrgreen:
HannesK
Product Manager
Posts: 15339
Liked: 3321 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: [VHRISO] Veeam Hardened Repos unavailable after automatic update

Post by HannesK » 1 person likes this post

Hello,
the current state of investigations is, that the update of network manager caused the firewall rules to reload. The Hardened Repository role (the software component that is installed on any Linux machine / on the Hardened Repository ISO) creates firewall rules dynamically during service start. That's why a reboot solves the problem. The bug was fixed in future versions.

I don't see a way to keep the repositories online because to restart the services one needs root and the current Hardened Repository ISO does not have an option to directly get a root shell to execute such tasks.

@Backup.Operator : this thread is about Hardened Repository ISO so let's please not confuse future readers by talking about self-managed Ubuntu servers where you can do whatever you like because they are self-managed.

Best regards
Hannes
1dna
Influencer
Posts: 12
Liked: never
Joined: Jan 28, 2025 6:29 am
Full Name: Lars Knakkergaard
Contact:

Re: [VHRISO] Veeam Hardened Repos unavailable after automatic update

Post by 1dna »

@HannesK

Is there a fix yet to avoid it breaks? or still work in progress?
mdippold
Novice
Posts: 7
Liked: 3 times
Joined: Jun 23, 2023 9:09 pm
Contact:

Re: [VHRISO] Veeam Hardened Repos unavailable after automatic update

Post by mdippold »

1dna
Influencer
Posts: 12
Liked: never
Joined: Jan 28, 2025 6:29 am
Full Name: Lars Knakkergaard
Contact:

Re: [VHRISO] Veeam Hardened Repos unavailable after automatic update

Post by 1dna »

@mdippold - I get this much, but it dosent state if it is fixed again (or and requires an update) - just hoping for a update instruction on what the next steps for this is.
fabbar
Influencer
Posts: 10
Liked: 5 times
Joined: Jan 02, 2024 10:42 am
Contact:

Re: [VHRISO] Veeam Hardened Repos unavailable after automatic update

Post by fabbar »

Hi,
also our VHRs (9+9 hosts) experienced this.

Since we cannot of course login in repository and reboot all is a time-consuming tasks I hope in the future updates will be notifiled. It would be enough also an email or a feed that notifies "veeam has just updated vhr repos, you might have to reboot your vhr in the next few days). Of course it would be pretty better if all would be done from VBR console.
HannesK
Product Manager
Posts: 15339
Liked: 3321 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: [VHRISO] Veeam Hardened Repos unavailable after automatic update

Post by HannesK »

@1dna Conversations about what to do with the next potential network manager upgrade are still ongoing. We only ship security updates for Hardened Repository ISO for foreseeable future and that means there is a good chance not to see a network manager update for longer time and we might have another cumulative update for 12.3 long before the next network manager update.

The tests we make before making updates available to customers were also expanded to cover more scenarios. If we see anything in the staging environment, then we can delay updates as long as needed.

@fabbar: update notifications exist in V13, yes. And one can also centrally control the updates, yes.
1dna
Influencer
Posts: 12
Liked: never
Joined: Jan 28, 2025 6:29 am
Full Name: Lars Knakkergaard
Contact:

Re: [VHRISO] Veeam Hardened Repos unavailable after automatic update

Post by 1dna »

@HannesK

Thaank you for an update - is there an official recommendation if we should continue using this iso? - or an update will be released to fix it?

I mean, will there be an updated iso we could use for reinstalled with the fixed Network Manager? so we can regain a lovely stable repository.
HannesK
Product Manager
Posts: 15339
Liked: 3321 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: [VHRISO] Veeam Hardened Repos unavailable after automatic update

Post by HannesK » 1 person likes this post

the bug is not in the ISO. The bug is in the "core software". Updating the ISO won't really help if a new Network Manager update becomes available.

If you want to block all updates, then you could set a non-existing HTTP proxy and it will never receive operating system updates again.

I'm not sure how one could give an "official recommendation". The Hardened Repository ISO has been working without any issues for around half a year now and we are working on improving operating system update user experience. The updates are the main reason, why Hardened Repository stays with "experimental support" because we did not want to delay the Hardened Repository ISO for many months until V13 ships with a better update management system. If you are fine doing Linux management yourself, then there is no real reason to use the ISO. I work with Linux myself since 2002 and would run Linux myself if I was a customer, but that's my personal background.
1dna
Influencer
Posts: 12
Liked: never
Joined: Jan 28, 2025 6:29 am
Full Name: Lars Knakkergaard
Contact:

Re: [VHRISO] Veeam Hardened Repos unavailable after automatic update

Post by 1dna »

Hi Hannesk,

okay sorry had to get it in my head hehe,

I get it - but as we are in progress of migrating to a linux repo - i would also like to get an idea of we should do a linux ourself also considering migration to v13 - we wouldent miss that train.

As i see it - cannot do much about this as understand, or could we add a defekt proxy to avoid further crashes? (if i understand you for now) until path of migration is then selected
johnnytan
Service Provider
Posts: 9
Liked: 3 times
Joined: Oct 09, 2023 3:52 pm
Full Name: Johnny Tan
Contact:

Re: [VHRISO] Veeam Hardened Repos unavailable after automatic update

Post by johnnytan » 1 person likes this post

johnnytan wrote: Jul 09, 2025 8:24 am Thanks for the heads up.
Being more than a decade's old system engineer, them keep asking for "root" password and adding user into "sudo" really frustrate me a lot.

Being that, I've taken my own ways and zip up the VeeamBackup logs to be attached in the ticket as well.

One thing to note, VeeamTransportSvc.log seems like grow unexpectedly large(12++ of Gigs) and not being rotated properly.
Finally I managed to fix my issue.
Multiple reboot or Update All doesn't solve my issue at all.

The main thing is veeamtransport service seems to be running from systemctl status but infact the service doesn't run at all.
I booted into Live System, add veeamsvc into sudoer files and begin my troubleshooting journey.

When I tried to redeploy the VHR via VBR, I stopped the veeamtransport service (after gaining root access via sudo su) and re-run it manually using
veeamtransport --run-service --port 6162
The command just hang there and I started to tail /var/log/VeeamBackup/VeeamTransportSvc.log
Little did I know, there's nothing to see at all from the log. NOTHING being written at all. I was expecting to see "service start... etc" but only realize the log stopped at 7th July (the time when everything went haywire globally)

As I've mentioned previously I noticed the VeeamTransportSvc.log being 12++Gigs, I checked /opt/veeam/transport/VeeamTransportConfig and there's a parameters there mentioned "MaxLogSize=10485760"

I truncated the log file, re-run the discovery and deployment via VBR and Voila! the Hardened Repo reconnected and came back Online.

I cannot be very sure about this if the whole culprit are the log file size issue, but the team will need to look seriously into this as un-rotated log file can be very dangerous
HannesK
Product Manager
Posts: 15339
Liked: 3321 times
Joined: Sep 01, 2014 11:46 am
Full Name: Hannes Kasparick
Location: Austria
Contact:

Re: [VHRISO] Veeam Hardened Repos unavailable after automatic update

Post by HannesK »

@johnnytan: The MaxLogSize is in bytes. That means 10MB. I can only recommend to ask support in the case to investigate why it grew to that size.

@1dna: Migrating to V13 will be easier from Hardened Repository ISO 2.0.0.8 as we are working on a migration path. I'm not sure what you mean with "further crashes" as this network manager update only happened once and there have been no other similar issues reported to us since we released Hardened Repository ISO. Configuring a non-existing HTTP proxy will stop the machine from downloading security updates for the operating system. You can also block the connection outgoing to repository.veeam.com outgoing on your firewall to achieve the same.
1dna
Influencer
Posts: 12
Liked: never
Joined: Jan 28, 2025 6:29 am
Full Name: Lars Knakkergaard
Contact:

Re: [VHRISO] Veeam Hardened Repos unavailable after automatic update

Post by 1dna »

@HannsK

So just for my brainfart head (requireing vaction soon) - as i understand it (please correct me if im wrong in my asssumption )

i get the network manager part (we have experinced it twice (course of 14 days every monday) - but entering invalid proxy data (http&https) will prevent this "break down" to occur.

it was a path a frustration - but i guess crashes and strange updates can never really be avoided 100%.

As we are moving our data out - due to some disk we have to change aswell - we should be okay to just either block the update (wouldent that prevent the pgp keys to update?)
ITP-Stan
Expert
Posts: 230
Liked: 68 times
Joined: Feb 18, 2013 10:45 am
Full Name: Stan G
Contact:

Re: [VHRISO] Veeam Hardened Repos unavailable after automatic update

Post by ITP-Stan » 1 person likes this post

Had this issue as well last week.
It would be nice if we could do an OS restart on the VHR from within VBR, but maybe that's already coming in V13.
Keep up the nice work!
dloseke
Veeam Vanguard
Posts: 80
Liked: 37 times
Joined: Jul 13, 2018 3:33 pm
Full Name: Derek M. Loseke
Location: Omaha, NE, US
Contact:

Re: [VHRISO] Veeam Hardened Repos unavailable after automatic update

Post by dloseke »

@ITP-Stan, I agree, but in this case, it looks like the VHR wasn't communicating with VBR, so I'm not sure if it would have been possible to restart the VHR from the VBR console if that function existed. But I agree that the functionality would be welcomed.
Derek M. Loseke, Senior Systems Engineer | Veeam Vanguard 2025 | Veeam Legend 2022-2024 | VMSP/VMTSP | VCP6-DCV | VSP/VTSP | CCNA | https://technotesanddadjokes.com | @dloseke
Post Reply

Who is online

Users browsing this forum: Amazon [Bot], Baidu [Spider] and 5 guests