VBSF backup fails across all our deployments

[sumeet]

Hello,

All old cases# 08073203, 07676393 and another older case too, but not able to find the number.

Backups fails with this error - Cannot run program "/usr/bin/java" (in directory "/var/log/vbsf/vbsf-backup/2026-07-24_10-00-00_vbsf_backup_428_6131_bizcover-backup-policy-daily_metadata"): error=2, No such file or directory

Strangely, happens across all our VBSF deployments - which are different RHEL versions, different VBSF versions, different VMs, different networks, different storage volumes/datastores, rebooted at different times (as last week I upgraded few setups to v3.2 on different days)

The only way to resolve this is to either restart vbsf service or ofcourse reboot the server.

Happend on the night of 24th April 2026 and then 24th July 2026. Has happend last year several times.
I have checked no cron jobs or anything that updates java.

Has anyone else observed this issue in their environment?

[tomsinner]

No, I haven't seen this in our environment. We've been running VBSF across multiple deployments without `/usr/bin/java` disappearing, so this doesn't seem to be a common issue from my experience. FNF

[sumeet]

Thanks Tom for your confirmation.
Strange that this happens in our environment. Very strange that it happens on the very same day across different servers.
We have checked that this path and file exists /usr/bin/java
Restarting VBSF backend service fixes this.
Is your deployments on RHEL too? And do you use native LVM?

[sumeet]

I may have found the cause for this issue.

Whenever the package/rpm java-17-openjdk-17.0.20.1.1-1.2.el9.x86_64 gets updated, the backup fails with error /usr/bin/java is not found.

If I search for vbsf process - it has /usr/bin/java in its path
And if I do ls -lh for /usr/bin/java, it points to /etc/alternatives/java, which then links to /usr/lib/jvm/java-17-openjdk-17.0.20.1.1-1.2.el9.x86_64/bin/java
And the rpm java-17-openjdk-17.0.20.1.1-1.2.el9.x86_64 was updated yesterday on of our setups and strangely did not update on rest of our env. So only this server where this rpm got updated, the backups failed last night.
rpm -qi java-17-openjdk-17.0.20.1.1-1.2.el9.x86_64 - shows install date as yesterday. If I search for when java-17-openjdk-17.0.20.1.1-1.2.el9.x86_64 was released, it is Aug 18 (this may be US TZ and our is almost a day apart)

The rest of my servers still have java-17-openjdk-17.0.20.0.8-1.1.el8.x86_64 and they did not fail.
On rest of my servers, if I check the install date of java-17-openjdk-17.0.20.0.8-1.1.el8.x86_64, it is Jul 24 - when all failed.
And if I search for, when was this java-17-openjdk-17.0.20.0.8-1.1.el8.x86_64 released, it shows Jul 23.

Java doco says that the reason it has links to /etc/alternatives/java is so that the jdk can be updated, without breaking the applications.
But in this case, vbsf fails.

Why is this specific to just my env? Does rest of the deployments have a lock on jdk or are the servers isolated to not allow auto updates or does the application have to handle such jdk updates?
Since this is salesforce backup, we have to enable restricted external access, and so these rpms get updated.