Backup of 'Real-Time' servers

[TerryNoriegaDurrant]

Is there a white paper or technical note on best practice on using Veeam to backup mission critical real-time servers ? In this case we are talking about a server that is communicating in real-time to PLCs which is running critical factory equipment ( and runs 24/7 ). Can't afford even a a second of dropped connectivity with the PLC as that causes the factory production lines to freeze taking at least 30 mins to get going again.

The server is a VM with application running in foreground. There is also an SQL Server database installed on this machine.

[aceit]

Interesting set up - I too have some machines connected to PLCs albeit connected with serial and not with strict realtime requirements (and not running over hypervisor), but not backupped with veeam (some are old XPs).

What connectivity do you have to the PLCs? Ethernet?

Anyway I suspect that your requirements are not "real" realtime too, and maybe you can tolerate a fraction of second of freeze where the packet are enqueued, because you can't have realtime guarantees with windows or standard linux especially running over an hypervisor (that has a non realtime scheduler).
Maybe what you mean is that you can tolerate delay but not micro-packetloss (so this isn't strict realtime, you just need strict reliability with tolerable delay).
The vsphere snapshot machinery can create micro interruptions but it should be greatly improved with > 6.0 versions.

[jmmarton]

Are you able to take regular VMware snapshots of these VMs? As long as that is tolerable, then I would think the best thing would be to keep the length of time these snapshots are open to a minimum. That means making sure the underlying storage for the datastore is a platform we have specific integrations with and leveraging Backup from Storage Snapshots.

Joe

[TerryNoriegaDurrant]

Thanks for the replies,

The PLCs are either Ethernet connected or are serial connected via a ethernet/serial servers ( moxa nports ).

We have tried to use the VM snapshot functionality but when we have attempted this we have had missed PLC communications during the fraction of a second the Veeam snapshot is occurring. If used in anger this would have caused the mill to not close a gate causing a 30 minute shutdown as product is manually dug out of a bin.

We are using Veeam Backup & Replication 9.5 - How do we configure / allow micro interruptions ?

[jmmarton]

This doesn't sound like a Veeam issue. If you're unable to even manually take VMware snpshots, then that's the real concern. We're simply triggering VMware snapshots. If you find the same problem when manually creating a snapshot then you can't use a snapshot-based solution for backing up these VMs. In that case you may want to consider using the Veeam agent inside these VMs and basically treat them as if they're physical servers.

Joe

[albertwt]

Is there a white paper or technical note on best practice on using Veeam to backup mission critical real-time servers ? In this case we are talking about a server that is communicating in real-time to PLCs which is running critical factory equipment ( and runs 24/7 ). Can't afford even a a second of dropped connectivity with the PLC as that causes the factory production lines to freeze taking at least 30 mins to get going again.

The server is a VM with application running in foreground. There is also an SQL Server database installed on this machine.

Try to utilize the Veeam Backup enterprise plus feature that can use Storage Array Snapshot.
In this way, the VMware snapshot is shortly opened.

[aceit]

Storage Array Snapshot I think it just reduces the time window in the open-close cycle, but it doesn't avoid the micro interruption that can happen so shouldn't solve the problem.
The hypervisor supported by Veeam just don't have realtime requirements and can have packet loss and delays during such operations and so Veeam B&R is indeed out of scope here, due to the reliance on hypervisor snapshotting.

In your case (without specific details given...), if you really want to drive this kind of interface via VM, I would go for a storage array specific solution (snapshot/closing provided by the storage array... also without particular integrations at least can give you a crash consistent copy).
Or you can try to use veeam endpoint if it is a supported OS (treat it as physical as suggested by prev. post...)

Anyway, backup for this kind of system it is really kind of specific and interesting in every case (in every company I see each time a different approach).

bye

[albertwt]

Or better yet, use Veeam Backup v10
https://go.veeam.com/v10

Veeam CDP (Continuous Data Protection)

Preserve critical data during a disaster and ensure the Availability of tier-1 VMs with VMware-certified continuous data protection to reduce RPOs from minutes to seconds, as well as leverage VMware’s VAIO framework to reliably intercept and redirect VM I/O to the replica without a need to create standard VM snapshots.