Exchange 2003 Replication nightmare

[rambrosio]

I successfully ran an Exchange Replication job last night.
Replication is to a Backup ESX 3.5 server within the same Domain.

This morning I decide to test the replication so I shut down the original Exchange server 2003 and started the
replicated server on the Backup ESX machine.
The server booted up and all exchange services started, however I was not able to receive/send mail from outlook.
On the Exchange Srv Application Event logs I had about 30 entries with ID 8231.

"Permanent failure reported by policy group provider for 'CN=System Policies,CN=QUANTUM,CN=Microsoft Exchange,CN=Services,CN=Configuration,DC=<Domain>,DC=LAN,DC=local':'MAD.EXE', error=80040103. Taking provider offline. "

I restarted Exchange System Attendant and ensured all services were running but could not send/receive mail. Due to this being a production environment, i stopped my
troubleshooting attempt, and opted to revert back to the original server.

I shut down the Replicated Exch serv and restarted the original. Imagine my surprise when I encountered the same problem and started to get the same Event Error ID 8231.
After a 15 minute interval, messages started to filter through and the original exchange server was fully functional.

Are there any Veeam Caveats for Exchange Server replication?

I'm reluctant to use this software if I cannot make use of a replicated Exchange Server.

Help much appreciated.

Thanks,
Rob

[tsightler]

First question, were you using VSS when you preformed the replication? Is this system also a domain controller?

[rambrosio]

Tom, I did use VSS, and the Exchange Server is not a DC.

[Gostev]

Rob, I will check on this for you with couple of Exchange folks, but I am fairly sure this issue is not connected to our product. From your description, it looks like what happened is our replication picked up Exchange image which already contained some sort of software issue, and replicated this image.

In other words, if you would simply reboot your production Exchange VM (instead of taking it down and starting up replica), you would still experience the same issue as with replica. Replica is nothing but complete mirror of your production VM, so powering on replica VM resulted in observing the very same issue you have later experinced by powering on production VM.

Anyhow, I will update you when I hear something from Exchange experts I know.

But I can assure you there are no known issues with Exchange 2003 backup and replication, I know for sure thousands (literally) of customers are doing this, and most of them have periodic backup/replica testing procedures in place.

[Gostev]

First question they had was whether you were using static or dynamic IP on your production Exchange server VM. What may have happened is replica Exchange server got assigned different IP address than production Exchange server had.

[rambrosio]

Static IP.

[tsightler]

In thinking about this problem overnight, I came up with a possible thought. The only time I've ever seen messages like you describe is when the time sync between the Exchange server and the rest of the domain was off. In this scenario the RUS (Recipient Update Service) will generate messages like the ones you described.

So, is there any chance that, when you powered off your Exchange server, and then powered on the replica, the replica would have booted with it's time out-of-sync with the domain, perhaps because the time on the ESX host itself might be wrong? If that's possible, this could also explain why the problem occurred when you shut off the replica and powered the "real" Exchange server back on. The "real" Exchange server could have also powered on with it's time out-of-sync, but, eventually it would have synced with the domain (I think the default for Windows Time Service is 15 minutes) and things would have worked.

Any chance this is what you saw?

[rambrosio]

Hello Tom, I checked my backup ESX and it has the same time as the production ESX.

However, It is possible that between me stopping the original Exchange and starting the replicated exchange
server, more than 15 minutes could have elapsed, making the time synch a real issue, and frankly I've never seen this happen in real life.

So, when I try this again (first thing tomorrow after a replication job tonight), I will note the any time anomaly and report back.

Thanks,
Rob

[Gostev]

Rob, please remember that you need to perform full replication from scratch tonight. Since you have powered on VM manually during the previous testing (instead of performing Failover command from Veeam Backup UI), VMDK will no longer be in consistent state after incremental replication takes place.

[tsightler]

I'm not too concerned with whether the two ESX servers have the same time, I'm more interested to know if the ESX server times are in sync to the Windows domain time. It's really just a thought because that's the only time I've seen the error you're reporting so I was trying to come up with some explanation for it. When your replicated environment boots, make sure the time reported on the Exchange server is the same as the time of your Windows domain controllers.

Good luck!!

[rambrosio]

Tom, the Exchange server VM is synched to the domain time on the DC.

[tsightler]

Right, but are your ESX servers sync'd with the domain time? That's the important part. If you ESX time is not synchronized with your Domain time, when you power on the VM it will initially have the ESX time, but will, after 10-15 minutes, sync with the Domain time.