Veeam Server DR

[epruess]

We are evaluating VEEAM as a replication solution. I would like to replicate our virtual machines to our DR datacenter. If I understand correctly the VEEAM server needs to be at the source site for performance. In the event of a disaster where the source site is destroyed I would no longer be able to recover my replicas at the target site since the VEEAM database would also be destroyed. I have 3 questions.

1. What is the recommended solution to provide redundancy for VEEAM in this scenario?
2. If the VEEAM database is lost and I manually power up the target side virtual machine will its state be the latest replica taken?

3. Currently VEEAM is running in a virtual machine. Could I use VEEAM to replicate it's own VM to the target DR site?

Thanks,
Eric

[Gostev]

Hello Eric,
1. You can backup the SQL server database using standard tools (SQL Management Studio), specialized products or scripts.
2. Correct.
3. Veeam can only replicate itself as crash consistent, because at the moment of transactionally consistent snapshot creation it would freeze itself causing deadlock. You can do crash-consistent, or you can use separate Veeam Backup install to replicate your primary Veeam server in the transactionally consistent manner.

[epruess]

Thanks for the fast reply! Since a manually power up state is the latest replica that should be satisfactory in the event of a disaster since we would want to recover with the least amount of data loss. That did raise one other question. If a disaster occurred during a replication would the replica VM be left in a corrupted state? I think I read that this is not the case but wanted to confirm changes are only applied after a successfully completed replication.

Thanks,
Eric

[Gostev]

Eric, that is correct, in this scenario latest replica state will be inconsistent. And it does not really matter when those changes are commited into VMDK, as the outage can happen while those changes are being commited into VMDK, which can take long depending on the amount of changes.

So indeed, when this scenario you have described happens, you would need Veeam Backup console to restore to earlier point in time, instead of inconsistent VMDK.

[epruess]

Let me make sure I understand you. If we lose network connectivity during a replication, the replica VMDK will be left in an inconsistent state? (Changes are applied on the fly?)

Manually starting the VM would not be an option due to an inconsistent replica VMDK?

Would we be able to continue replication once connectivity is restored or would we have to re-seed the replica?

VEEAM with intact database would be required to failover to a previous successful replica.

If this scenario is true, then in a disaster, to assure the RPOs achieved by replicating often (30 min) I would also need to backup the SQL database offsite as often as our replicas are taken.

This seems odd. A better method would be to only commit the replica changes to the replica VMDK once the network transfer is successful. That way during a disaster you could manually start the VMDK at the last time of successful replication with the understanding that all prior replicas would then be unusable. (OK in a disaster because I likely want most recent data). Basically your replica VMDK would always be left in a consistent state of the last good replica received.

I apologize if I'm not understanding correctly. We would want to use this primarily to create DR replica's and I need to understand how it would work in that scenario before we can make a purchase decision.

Thanks,
Eric

[Gostev]

Eric, you understanding is correct in all points you have made. We also have a white paper by one of our system engineers describing various methods of achieving highly available Veeam Backup, where we documented existing customer's experiences - you can request it through your sales rep.

You idea of commiting changes through staging area (after all the data is transferred) is generally valid for "fat" ESX and replication solutions which do not provide ability to failover to earlier point in time. But because our solution keeps number of previous states, we do not need to care about staging the data before applying it to VMDK. First, this would affect performance; but more importantly, the concept will not work with ESXi targets where all operations are remote and are done from Veeam backup server over the network (including final data commit from staging file to VMDK). So reliability would not be increased, and VMDK would still have the same chance to be corrupted if disaster happens during changed data commit into VMDK.

You concern that VMDK may end up in inconsistent state if the outage happens during the replication cycle is also valid, and you are right that in this case you will not be able to power it on manually (only through Veeam UI to the previous good state). On the other hand, as you will see from documentation, we do not actually recommend powering on replica VMs manually through VIC as doing this invalidates all replica rollback points. So in case when you power on the VM manually and find that there is an issue with the latest state, you will no longer be able to repeat failover to earlier states until you find good and working one (even through UI). This is exactly why we recommend using Veeam Backup UI to perform failover vs. just starting replica from VIC.

Thank you!

[darrellarbaugh]

Let me make sure I have a clear head on this:

If Veeam exists at the production datacenter and that production datacenter is unavailable, thus making the production Veeam B&R database unavailable, even with an installation of Veeam B&R at the DR datacetner, you cannot power on VM back to a replica because you do not have the most current SQL database?

Is that accurate, or can the Veeam UI scan the replication folder at the remote location and rebuild the replica database, so you can restore a VM to the most recent replica that produces a consistent VMDK?

[Vitaliy S.]

Darrell,

No, that's not corect. You will be able to power on replicated VMs to the most recent state using VIC, you just won't have an ability to choose earlier restore points. Choosing restore point is only available with the Veeam console.

Veeam UI cannot rebuild the configuration based on replication folder, you should also backup your SQL configuration DB located at the production datacenter as Anton has correctly noted above.

[NightBird]

So if I understand, if you have a replicating period of 30mn, you must backup your SQL database every 30mn too... smaller is the replicating period, bigger is the risk of inconsistant replicat on the DR site if the primary site fail...

Very annoying

Regards,
Boris

[NightBird]

Is it possible to have the Veeam Server that do replications on the DR site (with fat ESX on both side), so if the primary site fail during replication, we could start replicated VM to rollback n-1 ???

arghh no way... no vstorage API available in this scenario, suxx !!!

Boris

[Gostev]

Many customers choose to deploy secondary Veeam Backup server in the DR location, and replicate Veeam Backup configuration database between production site and DR site SQL server. This helps to address all concerns mentioned above.

[darrellarbaugh]

Vitaliy,

As long as the current state is not inconsistent (i.e. was in the middle of a replication), correct?

[Gostev]

Correct.

[NightBird]

Many customers choose to deploy secondary Veeam Backup server in the DR location, and replicate Veeam Backup configuration database between production site and DR site SQL server. This helps to address all concerns mentioned above.

Replicate Veeam Backup configuration database as often as replication intervall of VM (example 30mn) ? how ?

[Gostev]

Microsoft SQL Server has built-in replication functionality, this can be scheduled as often as you like, up to real-time.

Alternatively, you can configure your Veeam Backup installation to use configuration database on remote SQL VM, and replicate that VM with Veeam Backup to DR site (when needed, you will be able to connect DR site's Veeam Backup to replica DB).

Alternatively, in case of local SQL, you can replicate Veeam Backup with both Veeam VSS, this will produce crash consistent SQL replicas (same as if you snapshot running VM through VIC, this also does not quiesce VM).

[NightBird]

SQL Express 2005 ship with Veeam Backup doesn't have publisher fonctionnality for replication !

Could we program post job script after replication job finish, as we could for backup job ?

So we can launch a SQL backup script asap when replication finish.

[Gostev]

Yes, I actually referenced the actual script which does that earlier in this thread

[epruess]

I'm not much of a SQL expert but would it be possible to install VEEAM at the production site and connect it to a database at the DR site?

Thanks,
Eric

[Gostev]

Sure, but then in case if DR site dies, you will not be able to perform production backups. So it is best to have some redundance.

[romanos]

Hello all,
I am working on PoC and I'd want use design which you mentioned early.
SQL configuration DB placed on external SQL server and this DB mirrored to DR SQL server.
I came to issue with DR Veeam server. I'd want only start this server or service and user DR SQL DB in case on primary site failure.
But when I stop SQL mirroring and start service in DR site I will get message during start of the client that this DB is used by Veeam server in primary site. I have created request on support and they told me, that I have to reinstall Veeam on DR server and point it to DR SQL server (DB). Is it right? This is very unusefull and time consuming. In case of primary site failure I have to fastly restore servers in DR not reinstall Veeam.

[Gostev]

Roman, actually Veeam Backup installation takes less than 2 mins when you have an existing SQL database, so it should not really be a problem to reinstall.

But you can also try to copy the content of SqlLockInfo value (under Veeam Backup & FastSCP registry key) from production Veeam server to DR Veeam server, this should prevent this error from appearing.

Thanks!

[romanos]

Hi Gostev, thanks for your suggestion. I don't want to reinstall SW during primary site failure. I want only switch do DR. Yesterday evening I got the same information from Veeam support.

SqlLockInfo resolves issue during switching do DR site.

Thank you again.
Roman.

[jeremystw]

If you are doing backups of the SQL database to your DR site at what point in time is it best to perform the backup, if you want to be able to roll back to the previous backup? Just before a replication job runs? Right after it starts?
Thanks,

[Gostev]

After the job finishes.

[glennsantacruz]

Specific to the earlier point regarding network failure during a replication cycle: if we have a network issue during a replication, you indicated that this issue would indeed introduce corruption into the replicant VM. Three questions:

1) Is this corruption detected by subsequent replication attempts ( or by any other process ), so that we can proactively address this issue? It would be very bad indeed if we were unaware of a corruption problem.

2) If the replicant becomes corrupt ( due to the above ), do subsequent replication attempts "fix" the corruption? In other words are *all* changes applied to the replicant since the original seed, or only changes incremental between the last replication and current replication? Simply put, are replications considered "incremental" (requires last full plus all interim backups) or "differential" (based on last "full" plus last backup) ?

3) Supposing that # 2 above is *not* automatically resolved: how do we "re-seed" the replicant? Is the process to delete all replication jobs / delete from disk / start from square one, or do you have a means by which we can keep all settings of the job and re-seed (to removable storage)?

[Gostev]

1. Yes, by subsequent replication attempts.

2. Yes, this is resolved by our product in fully automated fashion. In case if replica inconsistency is detected in the beginning of the next job cycle, VMDK is rolled back to the last known good state, before the next incremental run is performed.

Replication is forever-incremental with synthetic fulls produced each cycle, see here for more information: Veeam Synthetic Backup Explained (replication explained towards the end of this article).

[glennsantacruz]

Considering that the Veeam SQL database maintains all metadata regarding replication / rollback points / etc,
it appears that a successful replication/recovery strategy hinges on having a replicated copy of this database,
up-to-date as of the most recent replication itself.

* To ensure we have latest Veeam SQL database at the DR site, we could:
a) run "Post Job Activity" custom task after each replication job; the custom task will backup the SQL database
and push it to the DR site (via some external replication mechanism)
b) setup two SQL instances (one at each site) and run "native" SQL replication between them (transaction log
shipping, etc.)
c) use a "remote" sql instance that is itself backed up (and replicated) via Veeam to the DR site. Presumably this
VM would need to be the "last" VM in the job sequence, to ensure all data are properly replicated after the actual
replication jobs themselves are done.
d) use a separate backup agent / toolset to backup the Veeam server completely


* Network interruption during replication cycle will cause corruption to replicant. Issues:
a) How to know when the corruption occurs? ( So we can proactively address )
b) How to return to "normal" operations with a "valid" replicant? ( Full reseed vs. automatic? )
c) How to identify "last known good state"?

* Site1 (Veeam) --> Site2 replication, followed by Site1 failure. Procedure:
a) Replicants are ok?
i) yes
a) manual restart of each VM. DR site is online & ready, requiring no Veeam intervention/installation
b) restore Veeam environment & "failover" using native tools
ii) no
a) restore Veeam environment & "failover" using native tools


* To perform Veeam restoration:
a) build VM (or physical server) for Veeam ; let it install local SQLExpress
b) restore SQLExpress backup (should have a copy at DR site) & restart Veeam
c) Veeam should "know" state of all VM's / replicants (based on latest database);
use Veeam interface to roll-back replicants / failover /etc.


The above procedure is dependent upon knowing the following:
a) Whether replicants are in good condition
b) "last good version" of replicant in case we need to rollback


Based on your explanation, a replication attempt will resolve itself from corruption ; that is certainly a relief, but we
still need to know the state of each replication attempt to properly plan for a successful failover to the DR site. It is
important to understand where the failover event occurs in respect to the replication, so we know which action to take for
each VM (and the overall infrastructure.)

For instance, the failure could occur during the window of replication, leaving some VMs in a properly replicated state
and others that are either corrupt or haven't yet been attempted (based on the replication window). Each of these scenarios
involves a different recovery strategy (unless you are recommending a single approach to solve all of the above.)

Do you have a link to the whitepaper you referred to (regarding high availability for Veeam)? Since we're in evaluation mode,
we've not identified any sales rep and therefore cannot ask them to give us product literature.

[Gostev]

Glenn, technical white papers are not shared on our web-site, but I can put you in contact with sales rep if you let me know your country. You can also register for live product demo with our system engineer on the web-site, this may be beneficial for your evaluation as well.

Single and universal approach is to simply go ahead and use Veeam Backup to failover to the latest state when the disaster happens, and if the latest state does boot because disaster broke replication for this specific VM, perform "Undo Failover", and then failover again to the previous state.

You should realize here that even if disaster happens while replication job is running, it only has chance to affect a single VM, as all VMs are processed sequentially. So worst case scenario, in case your replication job is interrupted by disaster (which is not so likely by itself), you will only have 1 VM not booting - and for that VM you just need to cancel and repeat failover to the previous restore point.

[glennsantacruz]

Yes, please do share your resources. I am in the USA and would love to talk with someone about best practice whitepapers.

I am somewhat confused by your response, though -- if a replication job is running for *multiple* VM's (let n=VM count), we could have at most one "damaged" VMs, and zero to n VMs that are "good". For instance, if the replication job were to process 50 VMs, we could have 5 "good", 1 "bad" and 44 "unchanged" since the job processes sequentially. As long as the restored Veeam server had an understanding of these states ( by way of SQL replication ), we could use a secondary Veeam server to start all the replicants, correct? The secondary Veeam server would "know" that the 6th machine (in this case) was a bad replica, and auto-failover to the last "known good" machine?

Simply put, it looks like you're recommending that we (as a rule) ensure that the Veeam database is kept "king" across the sites -- that it is replicated more frequently than the replication job for VMs, and we just use native tools to handle the restoration/failover.

[epruess]

Is it normal when installing VEEAM and connecting to an existing database that I am required to re-input all login credentials for the VCenter server and hosts?
The existing credentials appear to be there but don't work.

Thanks,
Eric