Recovery procedure when Veeam server lost?

[evander]

Hi all - is there a documented recovery procedure for what steps to follow when you loose your Veeam Server?

Assume the following situation:

You have a full Virtual Environement with two sites, a Primary and a DR site

Veeam is an existing Virtual machine sitting on your primary site, running backups and replications locally and remotely to DR site. If the SAN at your primary site dies, what procedure should you follow to either recover your backups or your replicas? Bearing in mind your Veeam server was just hosed with the rest of your Primary Site servers.

So questions running through my head:

What is the best way to start the Replicas on DR site with no Veeam Server?
What is the best way to recover backups on DR site with no Veeam Server?
Assuming you re-install Veeam at DR site how do you import backups and replication jobs for recovery?

I am pretty sure this is a real world situation that I'm probably not the only one wondering about so I was really hoping there is an offcial documented Veeam procudure to follow??

[singy2002]

Hi,
In my case when we run through a full DR scenario we pretend that Veeam server has also gone.

The Replicas can just be powered on, (manually) although if you are only testing then a snapshot should be taken and fired up, this way once complete the snapshot can be reverted without needing to do another full replica.

The backups can be restored to a DR veeam box and the file(s) imported.

As for importing replica jobs, I am not aware of this being possible (sorry fi this is incorrect) as the replicas belong to the Veeam server they are created on.

[evander]

Thanks Singy - I'm going to test that backup import function to make sure there are no unexpected issues that might arise so if ever the fan does get dirty its not too late already.

As a side note, I am also replicating my Veeam box (without Application Aware processing) which will, I assume allow me to get that box up first at the DR site and simply carry on from there as per normal, so technically I wouldnt need to import backups and replications jobs etc etc?

[Ontrial]

an added complication is Vcentre.

If the replication to the DR site uses Vcentre in the primary site what do you do ? Bring up the Vcentre server dirty in the DR site ?

If the Vcentre server has its SQL on a SQL DB in the primary site what do you do? Bring up DB server dirty in the DR site ?

Messy!

[foggy]

Actually, there's a good topic on protecting the Veeam server. Please take a look, this approach allows you to quickly recover the Veeam server itself and start the required VMs using the Veeam native UI with the least effort.

Besides, there's always an ability to recover even without Veeam server in the DR site. As Andrew already mentioned, in v6, replicas are just common VM snapshots and can be started up using vSphere Client just fine (note, however, that in this case, re-IP rules will not be applied and you will not be able to perform advanced replica operations such as failback or permanent replica failover). For backups, there's a standalone utility (extract.exe) coming with every Veeam B&R installation that allows to restore VMs from any VBK file created by Veeam B&R, in case you do not have the product installed.

[evander]

Thanks Foggy - gonna have a look at that thread right now...

[Ontrial]

Besides, there's always an ability to recover even without Veeam server in the DR site. As Andrew already mentioned, in v6, replicas are just common VM snapshots and can be started up using vSphere Client just fine.

This is messy because

1. If you want to test on a Saturday morning you have ruined your replicas that you brought up dirty, and therefore spend the following week cleaning up.
2. In a real DR you may have to go to version-1 replica .. how do you do you do that with the vsphere client ?

[foggy]

1. What do you mean by saying ruined? Each restore point is a VM snapshot, and it will remain intact if you revert to it, as all disk changes from the running VM will be redirected into a separate snapshot file. And replication job also is not touched.

2. You can obviously revert to any of the stored restore points (any of the VM snapshots) with vSphere client.

[ststrasser]

[merged]

I've been thinking: Our backup target is our main backup server as well... so if it goes down, how can I access and start my replicated machines, which are stored on our Synology NAS?

How do you all ensure you can access your backups?
Is it possible to manage backups from two seperate servers?

[dellock6]

I usually install Veeam Server in a VM, and I create a replication job of the Veeam Server itself, so if the primary fails I can start the replica VM.
If I need to deal with FC or other "physical" elements, I add to the virtual Veeam Server a physical proxy/repository, and I use Veeam Server only as a central console.

[maxfontana]

[merged]

Hi there.
I have 2 hosts esxi 4.1 (A e B ) both with local storage and veeam installed as virtual machine on esxi B (vsphere essential) . Everyday it is backed up to a nas and in addition replicated to HOST A. Here comes the question:
if my host B dies (and with it my veeam vm), how do I start my replicated vms on host A ?
On the manual I've read that it is better to avoid starting a replicated vm directly from Vcenter and it would be better to use the veeam wizard .....this fact is confusing me as how can I use wizard if I have no available Veeam vm at all?
Many thanks in advance.
Max
Italy

[Jfmoots]

I always replicate my Veeam server to the DR side and I'd power that one on manually so that I could properly failover my other VMs.

[JasonAF]

[merged]

I need some input on v6 replication architecture...

I'm looking at a scenario where the basic need is to replicate from one building to another building on the same campus. They have a high-speed fiber link between the buildings, with vSphere hosts and FC SAN in each building. That being the case, my understanding from the user guide and other reading is that a single Veeam replication server would suffice, similar to an onsite replication scenario. I understand that the one server would act as both the source and target proxy, and thus would not perform any compression, which is fine since this is not over a slow WAN link.

However, the user guide also states "If possible, avoid powering on a replica manually as it may disrupt further replication operations or cause loss of important data. It is strongly recommended to use Veeam Backup & Replication functionality to perform failover operations."

So, first off, is it acceptable practice to use a single Veeam server in this scenario?

If so, should it reside at the source or target site? My first inclination would be to put it in the source site, but what if that site has a disaster? With there being no Veeam server at the target site in that scenario, my concern is about proper practice to perform a failover operation? I'd imagine the replica VMs in the target site could simply be powered on from the vSphere Client, but the warning above recommends against that.

What is best practice for this scenario? One server or two? If one, where should it reside and what is the acceptable/supported "failover" method to bring VMs in the target site online if source site is down?

Thanks

[dellock6]

Hi Jason,
I usually design any replica operation, no matter is local, campus or wan, with at least two proxies, one at the source site and the other at the target site, and then I configure the replica jobs accordingly.
One "trick" for saving also the Veeam server is simply to replicate also this server (if it's installed on virtual...). In this way:
- if is have to failover any VM, you goes on using the original Veeam server for this operation
- if you have a complete site fail, you first power on Veeam server and from here you can go on managing all backup and replica jobs

If you are having a site failure, the least problem is loosing consistency in the replica operations, first goal is to power up all the failed VM...

[foggy]

I usually design any replica operation, no matter is local, campus or wan, with at least two proxies, one at the source site and the other at the target site, and then I configure the replica jobs accordingly.

Another reason to have proxies in both locations is the ability to use hotadd mode both for reading and writing data (unless your single proxy has access to both datastores).

Jason, also please review this discussion above for manual failover considerations. Also, here is another good topic on this matter.

[pcrebe]

Hi Jason,
I backup the veeam sql db at the end of every jobs with a post activity cmd so, if i need start the veeam_replica vm, i can restore the db at the nearest point in a few seconds. Naturally i put the db backup on the DR site

Carlo

[applc]

Ok if your VCenter and Veeam server are both located at the production site, how about this...(just thinking out loud).
1. Set up a second Veeam server at the DR site, just to control all replication jobs.
2. Replicate all VM's, including VCenter, from production site to DR site.
Then in a disaster, can you not just fire up the replicated VCenter, and then use the Veeam server at the DR site to control all the advanced B&R replication features?
All comments are welcome, as I haven't actually tried this. As I said, only thinking out loud.

[Vitaliy S.]

Yes, this would also work!

[RHDG]

Interesting topic, but I'm missing something.

I'm only seeing scenario's where there is the need to go to the DR site.
An important part of disaster recovery planning is also how to roll-back to the primary site.
In worst case scenario, we'll need to configure a new (physical) Veeam B&R server when we roll-back to the primary site.

I suppose this would be possible with a Bare Metal Recovery (BMR)
I only found this blogpost about Veeam and BMR. Unfortunately, this only applies to Veeam Endpoint Backup.

So my question is: Whats the best practice of BMR for a Veeam B&R server? Is there a dedicated manual for this?

Thanks

[foggy]

Why BMR? There's a built-in Veeam B&R configuration backup functionality that allows you to fire up a fresh instance with all your previous jobs/settings on any Windows machine.

[Vitaliy S.]

...and as to the BMR question, then you can definitely use Veeam Endpoint Backup for this purpose. There is no dedicated manual, cause this procedure is pretty straight forward:

1. Select backups you want to restore from via Recovery Media
2. Do the restore

P.S. here is a user guide for VEB talking about various ways to do that > http://helpcenter.veeam.com/endpoint/11 ... overy.html

[OlafGuenther]

Hello,

this is my first post and maybe a stupid question.

We have some VMs on Hyper-V Server running.
The Veeam Server runs also as VM.
Our SQL Server (with some Databases of our ERP Systems) runs also as VM.
The Veeam Database is on this SQL Server.

What if the SQL Server crashes completly?
How can I restore it? Is it possible to start Veeam without Database Access?

How looks the restore process?
Install a new SQL Server. Change Database Location to the new SQL Server. Import Veeam Seetings.
And start with Backup of the old SQL Server?

Thanks a lot!!

[PTide]

Hi,

It would be better if you kept Veeam database separated from other dbs.

Install a new SQL Server. Change Database Location to the new SQL Server. Import Veeam Seetings.

That will work, however an easier way would be to install Veeam Server anew and simply import the configuration. Please also see the thread for other users' experience.

Thanks

[BibieR]

Hi Guys,

we are currently having recovery issue. Veeam server went down and we lost all administrative tools and all programs wiped out. How can we restore VIB files stored on an external drive?

[Vitaliy S.]

You need to spin up a new backup server (Veeam), import backup files from the external drive and then do the restore.