Backing up Exchange 2010 CAS array on Windows 2008 R2 w/NLB

[STGdb]

I am using Windows 2008 R2 w/SP1 for my Exchange 2010 environment and using VEEAM B&R for backups (v7). Whenever I back up the Exchange 2010 CAS array VM’s, the backup fails, usually with RPC errors like:

Code: Select all

“Unable to truncate transaction logs. Details: RPC function call failed. Function name: [BlobCall]. Target machine: [<CAS array IP address>]. RPC error:There are no more endpoints available from the endpoint mapper. Code: 1753)” or “Error: RPC function call failed. Function name: [GetLastFreezeError]. Target machine: [<CAS array IP address>]. RPC error:There are no more endpoints available from the endpoint mapper. Code: 1753”.

This doesn’t seem to be an issue backing up Exchange but rather an issue backing up Windows with NLB’s (the error always points to the NLB address). My backups are designed one VM per job (so I can have better control over the backups). These errors only happen when these backup jobs overlap each other (because I allow multiple connections to my repository). If I run the backups separately or set the repository to only allow a single connection, I encounter no issues or errors. If they overlap at all, I get these types of errors. Each Windows Server has two vNIC’s (one on the core network and one for the CAS array NLB – heartbeat). Simple solution is to make sure that the jobs run separately and don’t overlap but that causes other issues for me.

Has anyone encountered this issue before and if so anyone have a fix for it? Thanks

[soylent]

We have the same setup, two CAS and two DB servers, with Windows NLB. I don't have NLB on a separate server, I just use the two CAS servers for NLB.

I set it up for both off-site replication and backup. As far as Veeam is concerned, it's just backing up snapshots of four virtual servers. The NLB shouldn't affect it at all.

If you are having issues truncating logs, look into your Exchange environment, and look at the DAG health. It's possible something has corrupted the logs.

[STGdb]

Thanks for the reply.

I'm still looking into the t-logs issue but I'm wondering (when backing up your CAS servers w/NLB), do the CAS server backups overlap each other? For me, the backups work fine when run separately (success w/no errors) unless they backups happen to overlap (if they overlap I get errors). Other VEEAM backups (w/o NLB on the Windows hosts) can overlap with no issues at all.

[STGdb]

I've spoken with VEEAM support about this and it seems that this behavior can probably be expected. So I have to make sure not to run the CAS server backups at the same time so that they don't overlap.

[Andreas Neufert]

I´m very interessted in more details of your setup, because I see the NLB Setup very often in the filed.
Did you set
netsh interface ipv4 set int “NLB LAN” forwarding=enabled
in your environment?

NLB has it´s limitations (NLB Cluster is only network aware, not service aware and can end up with user Connection Problems). Is there a special requirement (beside costs) not to use a load balancer?

[STGdb]

Yeah, it's enabled on both CAS members. Thanks for asking.

This is a brand new Exchange 2010 vSphere 5.1 cluster with 2 CAS/HT nodes, 3 DAG MBX nodes and 2 EDGE roles. All are 2010 SP3 UR2, Windows 2008 R2 SP1, etc. Standard stuff - nothing unusual to cause the issue.

For now, it is just easier (and cheaper) for us to use Windows NLB until this process has moved further along. We were looking at implementing Barracuda's probably next year (have to have multiple appliances for HA to remove the single-point-of-failure issue). We are also looking to implement a more aggressive deployment of VEEAM at the time as well (so I need things to be working properly now before moving forward - although this isn't a show stopper).

[Andreas Neufert]

I´m just guessing. But can you please check if normal Network Interface is higher than NLB Interface at Network-Advaced Settings - Adapters and Bridge.

In my pervious working life I implemented some big Exchange Environments. When customer wanted NLB I used the follwing great article to configure everything.
http://www.msexchange.org/articles-tuto ... part1.html
Maybe you can find there an Setting/Option/tweak that fix also the Problem described in your first mail. Worth a try, but maybe different Jobs are the only Option.
To coordinate backup Job schedule, I think it`s a good idea to chain the Jobs one after the other at Job schedule page.

[b.vanhaastrecht]

[no technical case created (yet), first lets see if my setup is ok or not]

We have two CAS servers configured as a CAS array with Windows Load Balancing.
SRV-MAIL04 / LAN 10.10.31.10 / NLB 10.10.31.11
SRV-MAIL05 / LAN 10.10.31.20 / NLB 10.10.31.21
NLB IP 10.10.31.1

We have a problem with backing up the CAS servers. It gives an error regarding RPC, but the thing is, both CAS server are being contacted to the NLB IP address 10.10.31.1. When I analyze the logs of both CAS servers, I only see entries to the same NLB IP address, not their individual IP.

LAN Adapters are set first priority in Windows, in VMware the first network adapter is the LAN adapter.

Why is Veeam using the NLB IP in stead of the individual LAN IP?

[Vitaliy S.]

Hello Bastiaan,

Do you see the same errors as the OP above? Can you please tell me if you backup these VMs at the same time?

Thanks!

[b.vanhaastrecht]

Hello Bastiaan,

Do you see the same errors as the OP above? Can you please tell me if you backup these VMs at the same time?

Thanks!

Hi, yes the same RPC error. And yes, the servers run at the same time. But dispite of this, Veeam should be contacting the VM thru the primairy IP.
Thanks!

[Vitaliy S.]

Yes, I agree, however Veeam retrieves VM information (DNS name, IP address etc.) via VMware Tools, so we don't have a control over it. I would still suggest reaching out to our support team to make sure you don't have any misconfigurations.

[Andreas Neufert]

Just guessing because I´m traveling and have no access to vcenter.... vcenter list the IP´s from low to high numbers, so the NLB address where contacted first, and because server is answering it will use this IP for further communication. Maybe increase the IP Address Number something higher then normal LAN adapter will solve it?
Beside this, check your DNS with nslookup at Veeam B&R Server so that the NLB Address isn´t used as name forward lookup address.

[ta_jb]

Just to let you know, Andreas' recommendation to change the order of LAN and virtual IP solved the problem for me. Thanks!
We're (at our customer) running three 2008 R2 RDS Host in such a NLB Cluster and had the same issues with parallel processing of the three VMs.

[Andreas Neufert]

Thank you very much for the feedback.

I will ask Veeam Support to create an KB Article for this tip.

[Ben Milligan]

[Disclaimer: I'm from Veeam Support ]

Thank you Andy!

We are not looking to create a KB for this currently, however we do have a hotfix that corrects this by inversing the IP order for guest processing. I encourage all customers experiencing this to open a case with our support team so we can help you with this hotfix.

[Andreas Neufert]

To have an solution (patch) is way better than changing IPs in production. Great stuff, thank you Ben and Support/Dev/QA

[albertwt]

FYI: After you've updated the Veeam Backup & Replication to version 8, the correct registry entry is HKLM\Software\VeeaM\Veeam Backup and Replication NLBClusterPrimaryIps

[Andreas Neufert]

Thanks for sharing AlbertWT

[NightBird]

Hi, the NLBClusterPrimaryIps is a multi string value ?

[NightBird]

Ok after trying it's multi string