-
- Novice
- Posts: 8
- Liked: never
- Joined: Sep 07, 2020 11:10 pm
- Contact:
Unable to perform application-aware processing on Exchange 2019
I have a small lab environment running a 3 node Hyper-V cluster.
All nodes run Windows Server 2019 Datacenter Core installs.
I have 6 VM-s running on the cluster. 2x Windows Server 2019 Core running as domain controllers. 1 running a linux appliance and 1 running Windows Server 2019 Core as Exchange 2019 server. The remaining 2 are not relevant as they are not backed up.
Outside the cluster I run a VM with Windows 2019 for Veeam B&R v10. All machines are domain joined.
The Veeam backup server backs up the 2 domain controllers, the Exchange server and the Linux appliance.
The Linux does not have application aware processing enabled and the backups are all successful. The Domain Controllers have application aware processing enabled and the processing appears to work as expected successfully. The Exchange server however runs the backup but finishes with warnings and does not perform application aware processing so Exchange logs do not get truncated.
I have opened a case with Veeam - Case #04365848, however this is a community edition Veeam, so the support is best effort and not a guaranteed support, so I was hoping some smart minds here may have encountered this before and can point me in the right direction.
So the warnings the job gives me are:
1. "Unable to perform application-aware processing because connection to the guest could not be established"
2. "Unable to process Microsoft Exchange transaction logs because connection to the guest could not be established"
The OS on all servers is up to latest windows updates. the version of Veeam is 10.0.1.4854
Veeam connects to the servers using a service account that is in the local administrators group for all servers.
As Veeam instructions for testing access and connectivity suggested testing the admin$ share, I logged into the Veeam server using the service account and tested access to the Exchange server admin share using a few options:
\\MAILSERVER\Admin$ - worked
\\MAILSERVER.DOMAIN.COM\Admin$ - worked
\\192.168.0.10\Admin$ - did not work and got Error code: 0x80004005 Unspecified error
When trying the same access methods to domain controllers they also respond to the IP in the share name.
That kind of leads me to believe that Veeam might be trying to access that server using IP address instead of name when it is trying to gain access to the admin share and there is some kind of security setting in the latest versions of Windows 2019 that are not domain controllers or its patches/updates that may be blocking that. I have tested this with firewall on and firewall off with no change in the response.
Any help figuring out how to get around this or how to get Veeam use the server name instead of IP for application aware processing would be very welcome.
All nodes run Windows Server 2019 Datacenter Core installs.
I have 6 VM-s running on the cluster. 2x Windows Server 2019 Core running as domain controllers. 1 running a linux appliance and 1 running Windows Server 2019 Core as Exchange 2019 server. The remaining 2 are not relevant as they are not backed up.
Outside the cluster I run a VM with Windows 2019 for Veeam B&R v10. All machines are domain joined.
The Veeam backup server backs up the 2 domain controllers, the Exchange server and the Linux appliance.
The Linux does not have application aware processing enabled and the backups are all successful. The Domain Controllers have application aware processing enabled and the processing appears to work as expected successfully. The Exchange server however runs the backup but finishes with warnings and does not perform application aware processing so Exchange logs do not get truncated.
I have opened a case with Veeam - Case #04365848, however this is a community edition Veeam, so the support is best effort and not a guaranteed support, so I was hoping some smart minds here may have encountered this before and can point me in the right direction.
So the warnings the job gives me are:
1. "Unable to perform application-aware processing because connection to the guest could not be established"
2. "Unable to process Microsoft Exchange transaction logs because connection to the guest could not be established"
The OS on all servers is up to latest windows updates. the version of Veeam is 10.0.1.4854
Veeam connects to the servers using a service account that is in the local administrators group for all servers.
As Veeam instructions for testing access and connectivity suggested testing the admin$ share, I logged into the Veeam server using the service account and tested access to the Exchange server admin share using a few options:
\\MAILSERVER\Admin$ - worked
\\MAILSERVER.DOMAIN.COM\Admin$ - worked
\\192.168.0.10\Admin$ - did not work and got Error code: 0x80004005 Unspecified error
When trying the same access methods to domain controllers they also respond to the IP in the share name.
That kind of leads me to believe that Veeam might be trying to access that server using IP address instead of name when it is trying to gain access to the admin share and there is some kind of security setting in the latest versions of Windows 2019 that are not domain controllers or its patches/updates that may be blocking that. I have tested this with firewall on and firewall off with no change in the response.
Any help figuring out how to get around this or how to get Veeam use the server name instead of IP for application aware processing would be very welcome.
-
- VP, Product Management
- Posts: 7076
- Liked: 1510 times
- Joined: May 04, 2011 8:36 am
- Full Name: Andreas Neufert
- Location: Germany
- Contact:
Re: Unable to perform application-aware processing on Exchange 2019
It is hard to guess without the logs. Let´s see what the support says. I added some internal notes to the ticket.
-
- Novice
- Posts: 8
- Liked: never
- Joined: Sep 07, 2020 11:10 pm
- Contact:
Re: Unable to perform application-aware processing on Exchange 2019
Hi Andreas,
Any chance you can make those internal notes visible to the ticket, so I can see them? As the system has now closed that ticket for me:
"This is Veeam Support with a system generated courtesy notification. Unfortunately, due to high Support Team load we were unable to process your request and it has been archived automatically."
Any chance you can make those internal notes visible to the ticket, so I can see them? As the system has now closed that ticket for me:
"This is Veeam Support with a system generated courtesy notification. Unfortunately, due to high Support Team load we were unable to process your request and it has been archived automatically."
-
- VeeaMVP
- Posts: 1006
- Liked: 314 times
- Joined: Jan 31, 2011 11:17 am
- Full Name: Max
- Contact:
Re: Unable to perform application-aware processing on Exchange 2019
Is your service account a local account on your exchange?
If so the UAC could prevent access to admin$, although then it also shouldn't work via FQDN.
If so the UAC could prevent access to admin$, although then it also shouldn't work via FQDN.
-
- Novice
- Posts: 8
- Liked: never
- Joined: Sep 07, 2020 11:10 pm
- Contact:
Re: Unable to perform application-aware processing on Exchange 2019
The service account is a domain account that has been added to the local administrators group on the exchange server as well as other servers that get backed up.
-
- Novice
- Posts: 8
- Liked: never
- Joined: Sep 07, 2020 11:10 pm
- Contact:
Re: Unable to perform application-aware processing on Exchange 2019
OK, so I did some more testing on this and I believe I have figured it out and fixed this.
In case anyone else runs into similar issue, this is what I believe the issue was and how to get around it:
The mail culprit for the issue is probably the domain security hardening policy that forces to NOT allow NTLM authentication.
This means that all authentication will be forced to Kerberos and until recently using IP addresses with Kerberos authentication would fail as the SPNs were not registered for IP addresses in AD by default.
I found this Microsoft article that describes a new capability that can help circumvent that:
https://docs.microsoft.com/en-us/window ... os-over-ip
So I added the following to the Backup server and the cluster nodes:
reg add "HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System\Kerberos\Parameters" /v TryIPSPN /t REG_DWORD /d 1 /f
Ran the following on the mail server:
Setspn -s host/192.168.0.10 mailserver
Then tested again that the admin share now was working using \\192.168.0.10\admin$
This appeared to be successful.
Tested the backup again, however it still failed. So restarted all servers in the cluster as well as backup server and the backed up server and retried and this time the backup ran successfully with the application-aware processing.
So my suspicion is that Veeam does not use the name to connect to the server but uses the IP instead and if the NTLM authentication is disabled the IP does not get authenticated correctly as Kerberos ticket fails. To work around that kerberos over IP needs to be configured and servers may need to be restarted for this to start working correctly.
In case anyone else runs into similar issue, this is what I believe the issue was and how to get around it:
The mail culprit for the issue is probably the domain security hardening policy that forces to NOT allow NTLM authentication.
This means that all authentication will be forced to Kerberos and until recently using IP addresses with Kerberos authentication would fail as the SPNs were not registered for IP addresses in AD by default.
I found this Microsoft article that describes a new capability that can help circumvent that:
https://docs.microsoft.com/en-us/window ... os-over-ip
So I added the following to the Backup server and the cluster nodes:
reg add "HKLM\SOFTWARE\Microsoft\Windows\CurrentVersion\Policies\System\Kerberos\Parameters" /v TryIPSPN /t REG_DWORD /d 1 /f
Ran the following on the mail server:
Setspn -s host/192.168.0.10 mailserver
Then tested again that the admin share now was working using \\192.168.0.10\admin$
This appeared to be successful.
Tested the backup again, however it still failed. So restarted all servers in the cluster as well as backup server and the backed up server and retried and this time the backup ran successfully with the application-aware processing.
So my suspicion is that Veeam does not use the name to connect to the server but uses the IP instead and if the NTLM authentication is disabled the IP does not get authenticated correctly as Kerberos ticket fails. To work around that kerberos over IP needs to be configured and servers may need to be restarted for this to start working correctly.
-
- VP, Product Management
- Posts: 7076
- Liked: 1510 times
- Joined: May 04, 2011 8:36 am
- Full Name: Andreas Neufert
- Location: Germany
- Contact:
Re: Unable to perform application-aware processing on Exchange 2019
Thanks for the feedback.
I guess instead of booting the servers a
gpupdate /force
on the systems that are part of this, would help.
Added this information also here:
https://andyandthevms.com/exchange-dag- ... plication/
I guess instead of booting the servers a
gpupdate /force
on the systems that are part of this, would help.
Added this information also here:
https://andyandthevms.com/exchange-dag- ... plication/
-
- Novice
- Posts: 8
- Liked: never
- Joined: Sep 07, 2020 11:10 pm
- Contact:
Re: Unable to perform application-aware processing on Exchange 2019
Andreas, running "gpupdate /force" in my case was not helping because the changes were not done to the GPO-s. The changes were to the local registry and SPN on the domain controllers. The SPN would have worked regardless of reboot or gpupdate, but I suspect the registry change only applied partially and did not fully apply or was still caching some information that was reloaded/cleared during reboot. As the share access started working after the changes without reboot but the backups application processing would not work until I did a reboot.
Who is online
Users browsing this forum: No registered users and 27 guests