Host-based backup of VMware vSphere VMs.
Post Reply
rfn
Expert
Posts: 141
Liked: 5 times
Joined: Jan 27, 2010 9:43 am
Full Name: René Frej Nielsen
Contact:

Problems with Replica Failback to Production

Post by rfn »

Hi,

I'm testing failover til replica and failback to production and I'm having problems when failing back. I have created a support ticket, ID#5209842, but with a low severity because it's not critical, so it might be quicker to get a solution here. Support directed me to this article http://www.veeam.com/kb_articles.html/KB1218 and from that I can see that I might have problems because I'm using to separate vCenters for production and test/DR.

Productions hosts are running vSphere 5 and have a FC connection to a HDS HUS 110. I have a backup proxy (Windows Server 2008 R2) running on production hosts and I'm using hot add for backup and also for testing replication. That works beautifully.

Test/DR hosts are running vSphere 5 and have an EqualLogic PS4000e as a replication target.

Veeam Backup and Replication is running on a physical server with no SAN access at the moment, only LAN. Backup storage is directly connected to this server (two EONstor FC DAS and one LaCie eSATA box).

When performing backup and replication from production I can see that hot add is working just fine and when replication then target is the VBR server itself using NBD. I can perform a replica failover with no problems, but when I try to do a failback to production then I get an error. In the logs it's something about permissions and read/write acces. It looks like it's trying to use hot add but it also looks like it's the vCenter on the test/DR side that's trying to get access on productions hosts and that's not working.

Is that my problem, that I have to separate vCenter servers? I can't really do it any other way since we're using vSphere Essentials and I also like to have my production and test/DR totally separate.

If I switch the backup proxy on production hosts to use NBD then it works, but I don't want that since it would really slow down our backups. One solution is to use the VBR server as source and target proxy for replications, but that would also slow replications since it would use NBD both ways and the VBR server itself isn't exactly the fastest server in the world (two older Xeon Dual Core processors and 10 GB RAM), but it's a possibility.

If I have to use the VBR server as replication proxy would it then help to give it SAN access to both productions SAN (HDS HUS 110) and replication target (PS4000e) through iSCSI? It's possible to do so, but would it speed things up?
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Problems with Replica Failback to Production

Post by foggy »

What version of Veeam B&R are you running? If it is not the latest v6.1, I would suggest to upgrade and verify whether this issue could be reproduced (there were some fixes regarding hotadd during failback according to our QC). If the problem still persists, it is worth deeper investigation (I'm not sure whether the given support KB article is still applicable to v6.x as it states v5.x in it).
rfn wrote:If I have to use the VBR server as replication proxy would it then help to give it SAN access to both productions SAN (HDS HUS 110) and replication target (PS4000e) through iSCSI? It's possible to do so, but would it speed things up?
Direct SAN mode cannot be used to populate replica VMs on target host. Only hotadd and NBD modes are supported for writing data to the target destination.
rfn
Expert
Posts: 141
Liked: 5 times
Joined: Jan 27, 2010 9:43 am
Full Name: René Frej Nielsen
Contact:

Re: Problems with Replica Failback to Production

Post by rfn »

We're using VBR 6.1.0.181 64 bit. When trying to failback I can see that it's using the VBR server as souce and the proxy on production host as target. That look ok, but the logs says this:

Code: Select all

srv| ERR |Failed to open VDDK disk [[Replica_Tier2] CGPC-Win7_replica/CGPC-Win7.vmdk] ( is read-only mode - [true] )
srv| >>  |Logon attempt with parameters [VC/ESX: [vcenterdev.tonsbakken.local];Port: 443;Login: [****];VMX Spec: [moref=vm-828];Snapshot mor: [snapshot-834];Transports: [hotadd];Read Only: [true]] failed because of the following errors:
srv| >>  |--tr:VDDK error: 3.One of the parameters was invalid
srv| >>  |--tr:Failed to create VM connection.
vcenterdev.tonsbakken.local is the vCenter on the test/DR side of things.

About using Direct SAN then I guess that it would still speed up the replication if I'm forced to use the VBR server for replication jobs?
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Problems with Replica Failback to Production

Post by foggy »

Seems that indeed the wrong vCenter is selected during failback resulting in inability to find VM by its ID. Please continue working with support to find the reasons.
rfn wrote:About using Direct SAN then I guess that it would still speed up the replication if I'm forced to use the VBR server for replication jobs?
Not necessarily, but worth checking. What could really speed the things up is installing a dedicated target proxy on a VM with access to the target datastores to allow Veaam B&R to use hotadd for replica population.
rfn
Expert
Posts: 141
Liked: 5 times
Joined: Jan 27, 2010 9:43 am
Full Name: René Frej Nielsen
Contact:

Re: Problems with Replica Failback to Production

Post by rfn »

Actually I also created a post where I asked if it's possible to use a domain controller I have running on test/DR hosts as a backup proxy so that I could use hot add for writes. For some mysterious reason that post got deleted! I can't see how that could have violated forum rules in any way!
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Problems with Replica Failback to Production

Post by foggy »

It wasn't deleted, just merged into existing discussion.
rfn
Expert
Posts: 141
Liked: 5 times
Joined: Jan 27, 2010 9:43 am
Full Name: René Frej Nielsen
Contact:

Re: Problems with Replica Failback to Production

Post by rfn »

Ahh... that wasn't so obvious... But it seems it should work. I think that I will try that once this issue has been solved by support. I hope they'll return soon.
rfn
Expert
Posts: 141
Liked: 5 times
Joined: Jan 27, 2010 9:43 am
Full Name: René Frej Nielsen
Contact:

Re: Problems with Replica Failback to Production

Post by rfn »

The solution was to enable "Failover to network mode" on the backup proxy that runs on production hosts. Then it was able to write the changes back to the original (source) VM.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Problems with Replica Failback to Production

Post by foggy »

That's not a solution, but rather an obvious workaround (as you've stated that NBD mode did work successfully on the source proxy). Was support able to track down the reason of hotadd failure?
rfn
Expert
Posts: 141
Liked: 5 times
Joined: Jan 27, 2010 9:43 am
Full Name: René Frej Nielsen
Contact:

Re: Problems with Replica Failback to Production

Post by rfn »

I actually think that it's a solution since VBR selected to use the backup proxy on production host as both source and target when doing the failback. This means that it fetches the data from the replica with NBD and writes the data with hot add. That's a better solution than using NBD in both directions as I was doing with the backup proxy on the VBR server.

I'm guessing that if I install backup proxy on the test/DR servers then I would be able to use hot add for both fetching data from the replica and to write data back to production hosts. Then VBR should select two different backup proxies during failback. Without that proxy I will always be forced to NBD on the replica side because I don't have a proxy with access to the replica storage.

It was support who suggested this solution.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Problems with Replica Failback to Production

Post by foggy »

Then I'm confused by this:
rfn wrote:The solution was to enable "Failover to network mode" on the backup proxy that runs on production hosts. Then it was able to write the changes back to the original (source) VM.
How enabling failover to network mode allowed source proxy to use hotadd?
rfn wrote:I'm guessing that if I install backup proxy on the test/DR servers then I would be able to use hot add for both fetching data from the replica and to write data back to production hosts. Then VBR should select two different backup proxies during failback. Without that proxy I will always be forced to NBD on the replica side because I don't have a proxy with access to the replica storage.
Yes, this is the right architecture for offsite replication scenario, which is thoroughly described in the product user guide.
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Problems with Replica Failback to Production

Post by foggy »

I guess the real issue was not with the source proxy failing to use hotadd to write to the production host (as I originally thought and as follows from your previous posts), but rather with this proxy failing to use hotadd to read from DR host.
rfn
Expert
Posts: 141
Liked: 5 times
Joined: Jan 27, 2010 9:43 am
Full Name: René Frej Nielsen
Contact:

Re: Problems with Replica Failback to Production

Post by rfn »

foggy wrote:Then I'm confused by this: How enabling failover to network mode allowed source proxy to use hotadd?
Before it used the VBR server as source, using NBD, and the proxy as target, using hot add. That failed since it appeared that it was the vCenter on the test/DR site that tried to instruct the proxy to use hot add, which it refused since it belongs to another vCenter on production.

After allowing the proxy to use NBD then it's able to be the source as well, using NBD, and for some reason that also moved the role of "director" to the vCenter that is on production and the the proxy could use hot add to write successfully. At least that's how it appears. Why the shift was made between the two vCenters, just be enabling NBD, is something that I can't explain.
foggy wrote: Yes, this is the right architecture for offsite replication scenario, which is thoroughly described in the product user guide.
I know, but this isn't offsite but just replication between server rooms, but I definately try it out to see the difference in performance.
rfn
Expert
Posts: 141
Liked: 5 times
Joined: Jan 27, 2010 9:43 am
Full Name: René Frej Nielsen
Contact:

Re: Problems with Replica Failback to Production

Post by rfn »

foggy wrote:I guess the real issue was not with the source proxy failing to use hotadd to write to the production host (as I originally thought and as follows from your previous posts), but rather with this proxy failing to use hotadd to read from DR host.
Maybe.. But on the log screen when the failback was running it listed the VBR server as source using NBD and the proxy as target, using hot add. That failed.
Post Reply

Who is online

Users browsing this forum: Amazon [Bot] and 101 guests