Discussions specific to the VMware vSphere hypervisor
Post Reply
STGdb
Enthusiast
Posts: 33
Liked: 2 times
Joined: Sep 06, 2013 5:17 pm
Full Name: STGdb
Contact:

Backing up Exchange 2010 CAS array on Windows 2008 R2 w/NLB

Post by STGdb » Sep 25, 2013 6:20 pm

I am using Windows 2008 R2 w/SP1 for my Exchange 2010 environment and using VEEAM B&R for backups (v7). Whenever I back up the Exchange 2010 CAS array VM’s, the backup fails, usually with RPC errors like:

Code: Select all

“Unable to truncate transaction logs. Details: RPC function call failed. Function name: [BlobCall]. Target machine: [<CAS array IP address>]. RPC error:There are no more endpoints available from the endpoint mapper. Code: 1753)” or “Error: RPC function call failed. Function name: [GetLastFreezeError]. Target machine: [<CAS array IP address>]. RPC error:There are no more endpoints available from the endpoint mapper. Code: 1753”.
This doesn’t seem to be an issue backing up Exchange but rather an issue backing up Windows with NLB’s (the error always points to the NLB address). My backups are designed one VM per job (so I can have better control over the backups). These errors only happen when these backup jobs overlap each other (because I allow multiple connections to my repository). If I run the backups separately or set the repository to only allow a single connection, I encounter no issues or errors. If they overlap at all, I get these types of errors. Each Windows Server has two vNIC’s (one on the core network and one for the CAS array NLB – heartbeat). Simple solution is to make sure that the jobs run separately and don’t overlap but that causes other issues for me.

Has anyone encountered this issue before and if so anyone have a fix for it? Thanks

soylent
Enthusiast
Posts: 56
Liked: 5 times
Joined: Aug 01, 2012 8:33 pm
Full Name: Max
Location: Fort Lauderdale, Florida
Contact:

Re: Backing up Exchange 2010 CAS array on Windows 2008 R2 w/

Post by soylent » Sep 26, 2013 2:17 pm

We have the same setup, two CAS and two DB servers, with Windows NLB. I don't have NLB on a separate server, I just use the two CAS servers for NLB.

I set it up for both off-site replication and backup. As far as Veeam is concerned, it's just backing up snapshots of four virtual servers. The NLB shouldn't affect it at all.

If you are having issues truncating logs, look into your Exchange environment, and look at the DAG health. It's possible something has corrupted the logs.

STGdb
Enthusiast
Posts: 33
Liked: 2 times
Joined: Sep 06, 2013 5:17 pm
Full Name: STGdb
Contact:

Re: Backing up Exchange 2010 CAS array on Windows 2008 R2 w/

Post by STGdb » Sep 27, 2013 2:12 pm

Thanks for the reply.

I'm still looking into the t-logs issue but I'm wondering (when backing up your CAS servers w/NLB), do the CAS server backups overlap each other? For me, the backups work fine when run separately (success w/no errors) unless they backups happen to overlap (if they overlap I get errors). Other VEEAM backups (w/o NLB on the Windows hosts) can overlap with no issues at all.

STGdb
Enthusiast
Posts: 33
Liked: 2 times
Joined: Sep 06, 2013 5:17 pm
Full Name: STGdb
Contact:

Re: Backing up Exchange 2010 CAS array on Windows 2008 R2 w/

Post by STGdb » Oct 04, 2013 7:00 pm

I've spoken with VEEAM support about this and it seems that this behavior can probably be expected. So I have to make sure not to run the CAS server backups at the same time so that they don't overlap.

Andreas Neufert
Veeam Software
Posts: 3812
Liked: 683 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: Backing up Exchange 2010 CAS array on Windows 2008 R2 w/

Post by Andreas Neufert » Oct 06, 2013 9:39 pm

I´m very interessted in more details of your setup, because I see the NLB Setup very often in the filed.
Did you set
netsh interface ipv4 set int “NLB LAN” forwarding=enabled
in your environment?

NLB has it´s limitations (NLB Cluster is only network aware, not service aware and can end up with user Connection Problems). Is there a special requirement (beside costs) not to use a load balancer?

STGdb
Enthusiast
Posts: 33
Liked: 2 times
Joined: Sep 06, 2013 5:17 pm
Full Name: STGdb
Contact:

Re: Backing up Exchange 2010 CAS array on Windows 2008 R2 w/

Post by STGdb » Oct 07, 2013 2:30 pm

Yeah, it's enabled on both CAS members. Thanks for asking.

This is a brand new Exchange 2010 vSphere 5.1 cluster with 2 CAS/HT nodes, 3 DAG MBX nodes and 2 EDGE roles. All are 2010 SP3 UR2, Windows 2008 R2 SP1, etc. Standard stuff - nothing unusual to cause the issue.

For now, it is just easier (and cheaper) for us to use Windows NLB until this process has moved further along. We were looking at implementing Barracuda's probably next year (have to have multiple appliances for HA to remove the single-point-of-failure issue). We are also looking to implement a more aggressive deployment of VEEAM at the time as well (so I need things to be working properly now before moving forward - although this isn't a show stopper).

Andreas Neufert
Veeam Software
Posts: 3812
Liked: 683 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: Backing up Exchange 2010 CAS array on Windows 2008 R2 w/

Post by Andreas Neufert » Oct 07, 2013 7:45 pm

I´m just guessing. But can you please check if normal Network Interface is higher than NLB Interface at Network-Advaced Settings - Adapters and Bridge.

In my pervious working life I implemented some big Exchange Environments. When customer wanted NLB I used the follwing great article to configure everything.
http://www.msexchange.org/articles-tuto ... part1.html
Maybe you can find there an Setting/Option/tweak that fix also the Problem described in your first mail. Worth a try, but maybe different Jobs are the only Option.
To coordinate backup Job schedule, I think it`s a good idea to chain the Jobs one after the other at Job schedule page.

b.vanhaastrecht
Service Provider
Posts: 562
Liked: 97 times
Joined: Aug 26, 2013 7:46 am
Full Name: Bastiaan van Haastrecht
Location: The Netherlands
Contact:

[MERGED] Windows NLB IP address used for RPC

Post by b.vanhaastrecht » Jan 27, 2014 8:38 am

[no technical case created (yet), first lets see if my setup is ok or not]

We have two CAS servers configured as a CAS array with Windows Load Balancing.
SRV-MAIL04 / LAN 10.10.31.10 / NLB 10.10.31.11
SRV-MAIL05 / LAN 10.10.31.20 / NLB 10.10.31.21
NLB IP 10.10.31.1

We have a problem with backing up the CAS servers. It gives an error regarding RPC, but the thing is, both CAS server are being contacted to the NLB IP address 10.10.31.1. When I analyze the logs of both CAS servers, I only see entries to the same NLB IP address, not their individual IP.

LAN Adapters are set first priority in Windows, in VMware the first network adapter is the LAN adapter.

Why is Veeam using the NLB IP in stead of the individual LAN IP?
========================================
Veeam ProPartner and Cloud Connect Provider

Vitaliy S.
Product Manager
Posts: 22971
Liked: 1555 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Backing up Exchange 2010 CAS array on Windows 2008 R2 w/

Post by Vitaliy S. » Jan 27, 2014 8:56 am

Hello Bastiaan,

Do you see the same errors as the OP above? Can you please tell me if you backup these VMs at the same time?

Thanks!

b.vanhaastrecht
Service Provider
Posts: 562
Liked: 97 times
Joined: Aug 26, 2013 7:46 am
Full Name: Bastiaan van Haastrecht
Location: The Netherlands
Contact:

Re: Backing up Exchange 2010 CAS array on Windows 2008 R2 w/

Post by b.vanhaastrecht » Jan 27, 2014 9:13 am

Vitaliy S. wrote:Hello Bastiaan,

Do you see the same errors as the OP above? Can you please tell me if you backup these VMs at the same time?

Thanks!
Hi, yes the same RPC error. And yes, the servers run at the same time. But dispite of this, Veeam should be contacting the VM thru the primairy IP.
Thanks!
========================================
Veeam ProPartner and Cloud Connect Provider

Vitaliy S.
Product Manager
Posts: 22971
Liked: 1555 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Backing up Exchange 2010 CAS array on Windows 2008 R2 w/

Post by Vitaliy S. » Jan 27, 2014 9:19 am

Yes, I agree, however Veeam retrieves VM information (DNS name, IP address etc.) via VMware Tools, so we don't have a control over it. I would still suggest reaching out to our support team to make sure you don't have any misconfigurations.

Andreas Neufert
Veeam Software
Posts: 3812
Liked: 683 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: Backing up Exchange 2010 CAS array on Windows 2008 R2 w/

Post by Andreas Neufert » Jan 27, 2014 9:24 am 1 person likes this post

Just guessing because I´m traveling and have no access to vcenter.... vcenter list the IP´s from low to high numbers, so the NLB address where contacted first, and because server is answering it will use this IP for further communication. Maybe increase the IP Address Number something higher then normal LAN adapter will solve it?
Beside this, check your DNS with nslookup at Veeam B&R Server so that the NLB Address isn´t used as name forward lookup address.

ta_jb
Service Provider
Posts: 20
Liked: 3 times
Joined: Sep 29, 2009 1:40 pm
Contact:

Re: Backing up Exchange 2010 CAS array on Windows 2008 R2 w/

Post by ta_jb » May 10, 2014 10:58 am

Just to let you know, Andreas' recommendation to change the order of LAN and virtual IP solved the problem for me. Thanks!
We're (at our customer) running three 2008 R2 RDS Host in such a NLB Cluster and had the same issues with parallel processing of the three VMs.

Andreas Neufert
Veeam Software
Posts: 3812
Liked: 683 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: Backing up Exchange 2010 CAS array on Windows 2008 R2 w/

Post by Andreas Neufert » May 10, 2014 12:07 pm

Thank you very much for the feedback.

I will ask Veeam Support to create an KB Article for this tip.

Ben Milligan
Expert
Posts: 173
Liked: 40 times
Joined: Jan 01, 2006 1:01 am
Contact:

Re: Backing up Exchange 2010 CAS array on Windows 2008 R2 w/

Post by Ben Milligan » May 13, 2014 1:55 pm 1 person likes this post

[Disclaimer: I'm from Veeam Support :) ]

Thank you Andy!

We are not looking to create a KB for this currently, however we do have a hotfix that corrects this by inversing the IP order for guest processing. I encourage all customers experiencing this to open a case with our support team so we can help you with this hotfix.

Andreas Neufert
Veeam Software
Posts: 3812
Liked: 683 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: Backing up Exchange 2010 CAS array on Windows 2008 R2 w/

Post by Andreas Neufert » May 13, 2014 2:01 pm

To have an solution (patch) is way better than changing IPs in production. Great stuff, thank you Ben and Support/Dev/QA

albertwt
Expert
Posts: 640
Liked: 20 times
Joined: Nov 05, 2009 12:24 pm
Location: Sydney, NSW
Contact:

Re: Backing up Exchange 2010 CAS array on Windows 2008 R2 w/

Post by albertwt » Jan 29, 2015 3:48 am 1 person likes this post

FYI: After you've updated the Veeam Backup & Replication to version 8, the correct registry entry is HKLM\Software\VeeaM\Veeam Backup and Replication NLBClusterPrimaryIps
--
/* Veeam software enthusiast user & supporter ! */

Andreas Neufert
Veeam Software
Posts: 3812
Liked: 683 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: Backing up Exchange 2010 CAS array on Windows 2008 R2 w/

Post by Andreas Neufert » Jan 30, 2015 5:35 am

Thanks for sharing AlbertWT

NightBird
Service Provider
Posts: 178
Liked: 32 times
Joined: Apr 28, 2009 8:33 am
Location: Strasbourg, FRANCE
Contact:

Re: Backing up Exchange 2010 CAS array on Windows 2008 R2 w/

Post by NightBird » Mar 04, 2015 8:46 am

Hi, the NLBClusterPrimaryIps is a multi string value ?

NightBird
Service Provider
Posts: 178
Liked: 32 times
Joined: Apr 28, 2009 8:33 am
Location: Strasbourg, FRANCE
Contact:

Re: Backing up Exchange 2010 CAS array on Windows 2008 R2 w/

Post by NightBird » Mar 04, 2015 9:39 am

Ok after trying it's multi string

Post Reply

Who is online

Users browsing this forum: No registered users and 14 guests