Surebackup Exchange issue

[backupquestions]

I've noticed that Exchange server is failing the "mail server" surebackup script test. (which is simply a telnet to port 25 I think). So I started up my surebackup job and set the time limits high so I could investigate what happens during the job.

What it looks like is all the Exchange services simply do not start at all. I started one of the manually and it started right up however.

Some google research showed me someone else with a similar issue and they suspected due to your storage being a little slower for the VM, some services time out and don't start due to this.

Is there any solutions for this? One person said they made a script that tries to start all Exchange services after bootup and keeps running every so often etc. That seems like something I don't really want to use on my production system, but maybe limit it to only the surebackup job. When I look in the "scripts" area of the job, I could make and add a script in here, but how would I ensure this is run first and then a delay before the mail server script? You can put them in order top down, but you can't specify a delay between.

Otherwise, any ideas on how to solve this? Currently I've disabled the mail server test portion of the job but it makes me nervous that there could be database corruption or something in Exchange.

[HannesK]

Hello,
if you work with application groups (I recommend that for exchange), then you can set "keep the application group running after the job completes" . Then you have time to test everything manually.

Do you have one exchange, or is it a DAG? Do you have a domain controller in your application group?

to specify delays: the easiest way it to add hardcoded "sleeps" in your scripts.

Best regards,
Hannes

[Andreas Neufert]

Non DAG
need AD controller running first, you need one to start in an application group.
Potentially you need as well one from you forest master domain as Exchange Groups are stored there.

For DAG, you need 2 members, a CAS Server (or witness disk) plus the above. Without it DAG will not bring up Exchange services automatically.

[backupquestions]

It is only a single stand alone Exchange Server. I'm using application group where it is one DC and this exchange server. The DC is getting fully started up authoritative first so that part is good.

[HannesK]

as you can start everything successfully when you doing it manually, I guess there is no serious issue.

The only idea from my side would look at the Exchange log files. The issues you see in virtual lab, would finally also happen if you do a production restore. So it is a good idea to find the reason why the services don't start automatically. In general, it must work. I have heard from partners that they manage to do SureBackup tests with VirtualLab

I would try with manual reboot first to narrow down the problem.

[Andreas Neufert]

Can you please share Exchange and Operating System Version?

[backupquestions]

Exchange 2016. Server 2016 is the OS for both VMs.

I do have multiple domain controllers and I suppose I could include more or all of them into this job as a test to see if it is any different.

I can tell you that in production my Exchange server is rebooted monthly for patching and it has no issue automatically starting all exchange services.

[Andreas Neufert]

Check if the domain controller that you use at surebackup is a global catalog. You can enable the test within the application group.

[backupquestions]

All 4 of my dcs are global catalogs, and I do always check that option in the testing and it passes. I guess I'll add all the DCs in next instead of just trying one, unless 1 DC so long as it is a DNS/global catalog with authoritative restore *should* work?

[Andreas Neufert]

Yes, it should work... Then maybe something is not correct with the configuration. Let the lab be running (Surebackup job select the setting at the Application groupd at the bottom and disable the email server checks).

Then check Exchange windows event logs. I think SureBackup is just doing it´s job and will show you that something is not correct.

Potentially it is just that someone added another DNS entry and Exchange can not fully communicate with AD?

[firefox15]

We had the exact same issue. Services weren't starting on the Exchange 2013 SureBackup, but they started fine manually, and they started fine automatically when rebooting our production Exchange server. We verified that we had DC connectivity from the SureBackup VM, but for whatever reason, it didn't want to start automatically. I'll say that we opened a support case, and while Veeam didn't really know what to do, they were able to point me to some logging on the script area so that I could work around the issue.

What we ended up doing is writing a small script that accepts the IP as a parameter, then remotely starts the services on Exchange. Luckily, no sleep is necessary because Veeam processes the scripts one at a time and waits for one to complete before continuing, so we just had to make sure it ran before the Mail Server script. We removed the Mail Server script, then selected the custom script, then selected the Mail Server script again, and they were in the correct order.

I can't seem to post it as code to the Veeam forum for whatever reason, so here is a Pastebin link to it. Make sure to pass the "%vm_ip%" argument to it from the test scripts pane. Also, because Veeam is calling PowerShell scripts wrong (they use "-Command" instead of "-File"), you want to make sure there are no spaces in the path or file name, or it will break.

Hopefully this helps out some people who are experiencing this issue.

[backupquestions]

Thank you for posting this!! I will give this a try.

[Andreas Neufert]

Please know that when you use surebackup with this script that you "workaround" the issue instead of fixing it. This basically means that in a disaster recovery environment the exchange server would only come up with the same manual task or your script. SureBackup just do it´s job and point out that you would run into issues at disaster restore.

For all Exchange Servers:
You need your AD Server with Global Catalog role of same domain in a Application Group that you start upfront.
You need your Forest Root AD Server in the same group as well as Exchange places the main security stuff there.
The DNS Server should be placed as well in this group that is configured on your exchange environment (if different from the above).
With these 2 started, a normal Exchange Server will come up and start the services.

For Exchange DAG:
You need to check how your DAG is configured regarding witness. Automatically means that an CAS Server will automatically take the witness role. There is as well an option to use a witness share.
So the CAS Server or the witness disk server need to be in the application group above.
Important point for a DAG is as well that the information store service will come up automatically only if you have at least 2 members of the DAG for each mailbox database.
It is best to place the DAG mailbox servers in the application group. Basically to have a separate SureBackup Job with this Application group and no linked backup job.

[Andreas Neufert]

Please share as well the Ticket number than we can check the case.

[firefox15]

Ticket number: 03801473

We have already tried all the stuff that you have mentioned. The client has an extremely simple VM environment with two hosts and around 6 VMs. We have both AD servers running before Exchange gets powered on,and it does not make a difference. There is no DAG, just a single server.

[Andreas Neufert]

OK please let me explain some additional things based on the support case.

The AD Server has to stay in the Application Group of the SureBackup Job. The reason is that it has to be started before you test the Exchange Server. Select DNS, GC at least as roles for it as it influences how long we wait for boot processing and other things. Then you can place the Exchange Server in a linked backup job that you test within the SureBackup Job.

When you start the SureBackup Job you must not touch the console of this server as it will interrupt the automatic processing. You can not login into the server and see if the services comes up.
It will take a while anyway as the Exchange Server performs VSS restore operations during this time for the database. You can see this within the event log "Exchange VSS Writer (instance GUID) has processed post-restore events successfully.”. The Event IDs are around number 96xx.

If you want to manually check this, add the Exhange Server to the bottom of the application group and enable the "keep the server runnin" checkbox within the SureBackup Job Application Group page. Here as well you need to give the server some time to process the operation above (arround 10 minutes to be save before you touch the console).

Again the DNS Server entry on the exchange server need to be the AD server that you start within the Application Group.

Last thing that I want to mention is that if you use Windows Firewalls, then the Veeam B&R Server need to be allowed to communicate with the system on RPC and other ports. The issue usually here is that the traffic comes from view of windows firewall from an untrusted network (outside of the domain network) and block the traffic by default. So there is no way to check if the service is running or not.

[Andreas Neufert]

Please check as well this: vmware-vsphere-f24/veeam-b-r-10-surebac ... 74045.html