vmware unable to quiesce VM oracle database

[pat_ren]

hey guys,

we had a weird issue come up i've not seen before, I have a VM that fails to take snapshots/quiesce - this happens with either a manual test snapshot in vmware or a veeam job.
looking into the logs, i believe the issue is due to the oracle database vss writer failing constantly.

the oracle db error is this:
https://www.veeam.com/kb2686

database in NOARCHIVELOG mode must be in a consistent state for backup

we do not manage or maintain this database so I have sent an email to the third party team who do, my guess is the database is not in a good state after some kind of unclean shutdown. I don't know much about oracle so I haven't tried to investigate it any further yet.
the backups were working fine up until a few days ago (seemed to line up with a server reboot) so I am guessing that might be the cause.

has anyone seen this before? do I need to get third party vendor to check the database? maybe give it a clean stop/start?

I'll wait until I hear back from them, but I'm just curious about this error and trying to learn a bit more about it and why it suddenly occurred. I will not be making any changes to the DB.
let me know if you guys have any ideas on any other possible explanations? thanks.

[tj818]

If the DB is in an Open state and in no archive log mode, then the snapshot will fail as there is no way to keep the DB consistent in Read/Write mode without archive logging. If no archive log mode is enabled prior to the snap, the DB will need to be in a Mounted or Offline State prior. The DB could be put into the correct state prior to the snapshot, but this will take it out of read/write. Is it possible to enable archive logs?

[pat_ren]

If the DB is in an Open state and in no archive log mode, then the snapshot will fail as there is no way to keep the DB consistent in Read/Write mode without archive logging. If no archive log mode is enabled prior to the snap, the DB will need to be in a Mounted or Offline State prior. The DB could be put into the correct state prior to the snapshot, but this will take it out of read/write. Is it possible to enable archive logs?

thanks, we don't manage or have anything to do with this database, and it's been backing up fine for ages until recently, so I suspect it's some kind of temporary issue. We've asked the third party to check it out and will see what they say, hopefully it can just be remounted in a normal state and will work normally again.

[Andreas Neufert]

If you can not take a regular manual VMware Snapshot, then I would look into this independently of Oracle as the processing has nothing to do with each other. (Starting VMware Snapshot without quiescense).

Our Veeam Guest Processing halts a Oracle Database if it is not in Archive Log mode to make it consistent on disk.

[pat_ren]

If you can not take a regular manual VMware Snapshot, then I would look into this independently of Oracle as the processing has nothing to do with each other. (Starting VMware Snapshot without quiescense).

Our Veeam Guest Processing halts a Oracle Database if it is not in Archive Log mode to make it consistent on disk.

Hi Andreas,

Thanks for the info, we can take a regular snapshot, but not a snapshot with quiescence (which relies on VSS) so I think this oracle VSS error is the root cause, but I'm waiting to get this database issue sorted first so that we know if it's the cause of the problem or not.

[Andreas Neufert]

VMware Tools based quiescense in Veeam or natively in VMware, does trigger a regular VSS processing on Windows if enabled.
This VMware processing just triggers regular VSS without any "intelligence" or fallback.

When Veeam Guest Processing is used, we are detecting possible misconfiguration or other VSS issues and fallback to other consistency methods like "Alter Database Begin Backup" (which leads in non archive log mode to a downtime as the database is halted shortly for the snapshot).

For the actual situation, you can issue the command
vssadmin list writers
to check for writers in a non clear state. Go to the Windows Services Console and restart the relevant VSS writer services. Run the command again to check if the service is back and in a clean state.

Or maybe plan a reboot of the whole server.

[pat_ren]

VMware Tools based quiescense in Veeam or natively in VMware, does trigger a regular VSS processing on Windows if enabled.
This VMware processing just triggers regular VSS without any "intelligence" or fallback.

When Veeam Guest Processing is used, we are detecting possible misconfiguration or other VSS issues and fallback to other consistency methods like "Alter Database Begin Backup" (which leads in non archive log mode to a downtime as the database is halted shortly for the snapshot).

For the actual situation, you can issue the command
vssadmin list writers
to check for writers in a non clear state. Go to the Windows Services Console and restart the relevant VSS writer services. Run the command again to check if the service is back and in a clean state.

Or maybe plan a reboot of the whole server.

Thanks Andreas, I've checked the vsswriters and other typical troubleshooting, and it's the Oracle VSS writer which is failing every time, the windows event logs confirmed that. No matter if we snapshot (app aware) in VMware or trigger a Veeam backup, we get the same Oracle VSS issue, so it's the Oracle VSS writer which is causing the problem here. We've rebooted the server already but that did not help. Backups were working fine for many months before this issue, so this makes me think the problem is something new within the database, not a Veeam or a VMware problem. I don't know enough about Oracle databases (not do I have any credentials) so I'm not planning to touch the database myself, one of our guys has been trying to get this client to engage their third party who looks after this specific software/database to take a look before we do anything further.

[Andreas Neufert]

Maybe helpful:
https://www.veeam.com/kb3164

There is a tracing option available for the Oracle writer:
https://docs.oracle.com/en/database/ora ... rvice.html

[pat_ren]

thank you, will use that to investigate further if needed, will be sure to report back once it's fixed too