[VHRISO] Veeam Hardened Repos unavailable after automatic update

[sonobob]

Hi,

we are currently using 30 hardened Veeam repos that were created with the official Veeam Hardened Repository ISO. After yesterdays automatic Monday patching, that they seem to do, all of them became unavailable in VBR and we had to reboot each of them to get them up and running again. Four of which we had to do a repair install and one we even had to run xfs_repair towards the backup storage.

Would it be possible to make this patching optional? We Plan to add another 20 and thinking this can potentially occur every Monday is a big concern.

Thanks!

[HannesK]

Hello,
and welcome to the forums. That sounds like a (big) technical issue. Can you please provide a support case ID for this issue, as requested when you click New Topic.

In Hardened Repository ISO 2.0 there is no option to direct option control the updates. There are unsupported options, but let's first see what support finds out. The symptoms do not match any known issues.

Best regards
Hannes

[EviLin]

We have 1 VHR, and experienced the same thing yesterday, it was unavailable in VBR, but after a reboot of VHR it came it was OK.

I think I read it some where, that you have to reboot the VHR server once in a while, Even after succesfully installing latest updates it's missing a reboot feature.

[HannesK]

yes, update management, reboot, defining a maintenance window is something for the V13 ISO which includes a component called "Veeam Updater" which manages all these things.

[johnnytan]

Hi guys, we have an ongoing case with Veeam Support now.

The issue was that the hardened repository suddenly disconnected last Sunday/Saturday. This happened to two deployment on a different client and it happen at the same time.
1 of the hardened repository was deployed recently only (1 month++) and we performed a reboot, and re-configure the hardened repository and it managed to remove both veeamdeployment and veeamtransport services and reinstall it back. This was solved.

The other client was deployed more than 3 months ago. We did the same thing and it just stuck at testing veeam transport. This is what we observed.
1. During re-installation, the veeamdeployment service is running and listening to port 6160. Meanwhile the veeamtransport service is running (based on systemctl status veeamtransport) but it is not listening to port 6162.
2. There's no network/firewall blockage. Port 6160 are 'telnet-able' from VBR to the Hardened Repository.

We have a lots other hardened repository running, but not from this recent ISO. It was prebuilt back then with installaton of bare Ubuntu and running the script to hardened back then which is downloaded from github community and it's still running like a beast.

For now we are still waiting for Veeam Support to reach back to us.

[HannesK]

Hello,
I moved your post to another thread that mentions similar issues. Can you please share the support case number so that we can look into it?

Thanks,
Hannes

[johnnytan]

Hi Hannes,

This are my Case #07757523

Thanks in looking in. Appreciate the effort

[ASG]

We had the same problem (machine was up but unavailable to VBR) - reboot, problem solved. We did not create a ticket because hiccups happen, however I wanted to +1 that we experienced the same problem at the same time like some others here.

[HannesK]

Hello,
thanks for the ticket number. I contacted support and hope that we can find the root cause from the logs. I also emphasized to support again that it's about Hardened Repository ISO and that they cannot simply request "root" commands from you.

In the other thread there was the suspicion that a networkmanager update caused the connection outage and I see that there was a new network manager package.

In any case, an update should not cause such big problems that repair / re-install is necessary or that XFS needs a repair (well, I don't know anything about the underlying hardware, maybe that caused XFS issues)

Best regards
Hannes

[Dynamic]

Hi there,
we had the same issue, multiple and different environments.
Just found this thread.
After a reboot and rescan, it’s working again.

[johnnytan]

Hello,
thanks for the ticket number. I contacted support and hope that we can find the root cause from the logs. I also emphasized to support again that it's about Hardened Repository ISO and that they cannot simply request "root" commands from you.

In the other thread there was the suspicion that a networkmanager update caused the connection outage and I see that there was a new network manager package.

In any case, an update should not cause such big problems that repair / re-install is necessary or that XFS needs a repair (well, I don't know anything about the underlying hardware, maybe that caused XFS issues)

Best regards
Hannes

Thanks for the heads up.
Being more than a decade's old system engineer, them keep asking for "root" password and adding user into "sudo" really frustrate me a lot.

Being that, I've taken my own ways and zip up the VeeamBackup logs to be attached in the ticket as well.

One thing to note, VeeamTransportSvc.log seems like grow unexpectedly large(12++ of Gigs) and not being rotated properly.

[1dna]

Hi

we are experincing the exact same Case #07756166

[1dna]

Anyone has found a fix or workaround to keep them online ?

[Backup.Operator]

yes, update management, reboot, defining a maintenance window is something for the V13 ISO which includes a component called "Veeam Updater" which manages all these things.

OK, so in this case, for the Ubuntu Server Linux VHR, can we still perform the manual update and upgrade ?

Code: Select all

apt update
apt full-upgrade
do-release-upgrade

[HannesK]

Hello,
the current state of investigations is, that the update of network manager caused the firewall rules to reload. The Hardened Repository role (the software component that is installed on any Linux machine / on the Hardened Repository ISO) creates firewall rules dynamically during service start. That's why a reboot solves the problem. The bug was fixed in future versions.

I don't see a way to keep the repositories online because to restart the services one needs root and the current Hardened Repository ISO does not have an option to directly get a root shell to execute such tasks.

@Backup.Operator : this thread is about Hardened Repository ISO so let's please not confuse future readers by talking about self-managed Ubuntu servers where you can do whatever you like because they are self-managed.

Best regards
Hannes

[1dna]

@HannesK

Is there a fix yet to avoid it breaks? or still work in progress?

[mdippold]

See post549533.html#p549533

[1dna]

@mdippold - I get this much, but it dosent state if it is fixed again (or and requires an update) - just hoping for a update instruction on what the next steps for this is.

[fabbar]

Hi,
also our VHRs (9+9 hosts) experienced this.

Since we cannot of course login in repository and reboot all is a time-consuming tasks I hope in the future updates will be notifiled. It would be enough also an email or a feed that notifies "veeam has just updated vhr repos, you might have to reboot your vhr in the next few days). Of course it would be pretty better if all would be done from VBR console.

[HannesK]

@1dna Conversations about what to do with the next potential network manager upgrade are still ongoing. We only ship security updates for Hardened Repository ISO for foreseeable future and that means there is a good chance not to see a network manager update for longer time and we might have another cumulative update for 12.3 long before the next network manager update.

The tests we make before making updates available to customers were also expanded to cover more scenarios. If we see anything in the staging environment, then we can delay updates as long as needed.

@fabbar: update notifications exist in V13, yes. And one can also centrally control the updates, yes.

[1dna]

@HannesK

Thaank you for an update - is there an official recommendation if we should continue using this iso? - or an update will be released to fix it?

I mean, will there be an updated iso we could use for reinstalled with the fixed Network Manager? so we can regain a lovely stable repository.

[HannesK]

the bug is not in the ISO. The bug is in the "core software". Updating the ISO won't really help if a new Network Manager update becomes available.

If you want to block all updates, then you could set a non-existing HTTP proxy and it will never receive operating system updates again.

I'm not sure how one could give an "official recommendation". The Hardened Repository ISO has been working without any issues for around half a year now and we are working on improving operating system update user experience. The updates are the main reason, why Hardened Repository stays with "experimental support" because we did not want to delay the Hardened Repository ISO for many months until V13 ships with a better update management system. If you are fine doing Linux management yourself, then there is no real reason to use the ISO. I work with Linux myself since 2002 and would run Linux myself if I was a customer, but that's my personal background.

[1dna]

Hi Hannesk,

okay sorry had to get it in my head hehe,

I get it - but as we are in progress of migrating to a linux repo - i would also like to get an idea of we should do a linux ourself also considering migration to v13 - we wouldent miss that train.

As i see it - cannot do much about this as understand, or could we add a defekt proxy to avoid further crashes? (if i understand you for now) until path of migration is then selected

[johnnytan]

Thanks for the heads up.
Being more than a decade's old system engineer, them keep asking for "root" password and adding user into "sudo" really frustrate me a lot.

Being that, I've taken my own ways and zip up the VeeamBackup logs to be attached in the ticket as well.

One thing to note, VeeamTransportSvc.log seems like grow unexpectedly large(12++ of Gigs) and not being rotated properly.

Finally I managed to fix my issue.
Multiple reboot or Update All doesn't solve my issue at all.

The main thing is veeamtransport service seems to be running from systemctl status but infact the service doesn't run at all.
I booted into Live System, add veeamsvc into sudoer files and begin my troubleshooting journey.

When I tried to redeploy the VHR via VBR, I stopped the veeamtransport service (after gaining root access via sudo su) and re-run it manually using

veeamtransport --run-service --port 6162

The command just hang there and I started to tail /var/log/VeeamBackup/VeeamTransportSvc.log
Little did I know, there's nothing to see at all from the log. NOTHING being written at all. I was expecting to see "service start... etc" but only realize the log stopped at 7th July (the time when everything went haywire globally)

As I've mentioned previously I noticed the VeeamTransportSvc.log being 12++Gigs, I checked /opt/veeam/transport/VeeamTransportConfig and there's a parameters there mentioned "MaxLogSize=10485760"

I truncated the log file, re-run the discovery and deployment via VBR and Voila! the Hardened Repo reconnected and came back Online.

I cannot be very sure about this if the whole culprit are the log file size issue, but the team will need to look seriously into this as un-rotated log file can be very dangerous

[HannesK]

@johnnytan: The MaxLogSize is in bytes. That means 10MB. I can only recommend to ask support in the case to investigate why it grew to that size.

@1dna: Migrating to V13 will be easier from Hardened Repository ISO 2.0.0.8 as we are working on a migration path. I'm not sure what you mean with "further crashes" as this network manager update only happened once and there have been no other similar issues reported to us since we released Hardened Repository ISO. Configuring a non-existing HTTP proxy will stop the machine from downloading security updates for the operating system. You can also block the connection outgoing to repository.veeam.com outgoing on your firewall to achieve the same.

[1dna]

@HannsK

So just for my brainfart head (requireing vaction soon) - as i understand it (please correct me if im wrong in my asssumption )

i get the network manager part (we have experinced it twice (course of 14 days every monday) - but entering invalid proxy data (http&https) will prevent this "break down" to occur.

it was a path a frustration - but i guess crashes and strange updates can never really be avoided 100%.

As we are moving our data out - due to some disk we have to change aswell - we should be okay to just either block the update (wouldent that prevent the pgp keys to update?)

[ITP-Stan]

Had this issue as well last week.
It would be nice if we could do an OS restart on the VHR from within VBR, but maybe that's already coming in V13.
Keep up the nice work!

[dloseke]

@ITP-Stan, I agree, but in this case, it looks like the VHR wasn't communicating with VBR, so I'm not sure if it would have been possible to restart the VHR from the VBR console if that function existed. But I agree that the functionality would be welcomed.