Backup for ORACLE RAC VMs

[ibarizz]

I’m currently facing a technical challenge that has recently emerged. One of my clients operates a 3-node virtual Oracle RAC cluster with over 40 shared disks, which they back up using the native RMAN tool.

In addition to these 40 disks, each node has between 3 and 4 persistent, non-shared disks that they wish to back up. The issue arises when attempting to back up these disks in the conventional manner (by creating a backup job targeting these VMs and excluding the other disks). This process freezes the disks, and during this freeze, the engine perceives the node as having failed due to the lack of response during those seconds. This causes the node’s database to go down and failover to another active node. However, some applications do not support this failover and cease to function until the original node is restored. Even performing a resource vMotion to move the VM to another node causes the engine to fail.

Given this situation, I need to validate a method of backing up these disks that avoids any type of freeze, whether caused by the VMware snapshot or a script (as these are Linux machines). I found an alternative solution.

Through Linux agents, there is a feature called “Snapshot-Less File-Level Backup” offered by Veeam Agent for Linux. This feature allows you to perform file-level backups without needing to create a snapshot of the volume being backed up.

While this alternative avoids disk freezing, the Veeam agent does not track if files and directories have changed in their original location since the backup process began. Therefore, it’s crucial to ensure the volume state remains consistent, meaning no write operations should be performed on the file system containing the backed-up data until the backup process is complete.

The client is not fond of this alternative, as the requirement to halt write operations on the file system seems unfeasible. These are system disks that are constantly being written to. Is this alternative viable, or should we consider another option? Thank you!

[david.domask]

Hi ibarizz,

Just to confirm a few things to make sure I got you right:

1. By shared disks, this is with SCSI-bus sharing in VMware or how are the disks shared between the VMs in the cluster?
2. I know it's a production machine, but since this might just be a impact of having VMware snapshots, if you take a manual snapshot and leave it running for the same time does it reproduce?
3. With Veeam Agent for Linux, did the normal volume level backup for the desired devices work or you also had issues with disk freezing?

I'm not confident the File level backup is the best solution for you, but if you can share the information above can advise a bit more, but if the nodes are struggling with just normal snapshot creation/removal within VMware, backup operations will likely also result in similar issues.

[ibarizz]

Hello David!

1. They are shared through an SCSI-bus sharing in VMware.
2. The VMware snapshot made the application to start the failover when trying the standard Virtual Machine Backup, so I believe a manual VMware snapshot would reproduce the same. I'm asking my customer relating this.
3. We haven't tried this yet, but I assume that the freeze created by the script would reproduce the same scenario.

I will try to come at you with the right answer regarding the manual snap from VMware and the pre-freeze script from the linux Agent.

[PetrM]

Hello,

I'd recommend our plug-in for Oracle RMAN, it can work in standalone and managed modes and designed especially to support Oracle RAC. I don't think that image-level backup could be an option to protect Oracle RAC workloads, even if we make snapshots work somehow, this scenario was not included in our testing plans, thus it would not be supported.

Thanks!

[ibarizz]

Hello PetrM,

I comprehend that the requirement is to generate a backup of the Virtual Machine (VM), ensuring at the very least, the revival of the Operating System and the configurations of the Oracle RAC. The independent multiwriter disks are ASM disks and it is not necessary to take them into account when performing the backup as they are already taken with the Oracle RMAN (the customer has the plugin integrated with Veeam).
So the thing is that we have to recreate the virtual machine in case of a huge failure that involves the whole server. So far we only have the databases backed up with RMAN, but freezing those non-shared disks gave the customer many issues.

Thank you!

[tgietz]

Hi @ibarizz maybe Veeam CDP could be an alternative way for the OS-Part. CDP don't use Snapshots and so you don't have a freeze.

Here you can see a Webinar about it: https://www.youtube.com/watch?v=P4KhMV6 ... _R&index=1

Regards
Tobias

[ibarizz]

Hello @tgietz, I heard about Veeam CDP, but is there any experience from the technical view if this solution fits good for Oracle RAC VMs?

[PetrM]

Hello,

Well, I'd break down this topic into two separate questions:
1. How to back up RAC? - My recommendation is to use RMAN plug-in.
2. How to perform image-level backups of VMs which run this very RAC? - I would try simple backup with AAIP disabled to avoid any interaction with Oracle. It should be possible to restore VMs from image-level backups and recover databases afterward using RMAN. Anyway, it's better to test this approach first.

Thanks!