Comprehensive data protection for all workloads
Post Reply
m.novelli
Veeam ProPartner
Posts: 504
Liked: 84 times
Joined: Dec 29, 2009 12:48 pm
Full Name: Marco Novelli
Location: Asti - Italy
Contact:

Backup failing with NTLMv2 security policy

Post by m.novelli »

Hi friends, I've set in my AD Domain the security policy "Security Settings > Local Policies > Security Options > Network Security: LAN Manager authentication level" to "Send NTLMv2 response only/refuse LM and NTLM" and then Veeam Backups started to fail with logon errors to all Windows VMs

I've reverted back the policy to the standard "Send NTLM response only" and backups started to work again

Is this a limitation in Veeam Backup or in Windows handshake? With this policy set to maximum security level I can successfully browse administrative shared, but Veeam fails with error "Processing XXXVM Error: Failed to connect to guest agent. Errors: 'Cannot connect to the host's administrative share. Host: [10.0.0.3]. Account: [administrator]. Win32 error:The user name or password is incorrect. Code: 1326

My Veeam Backup is running on Windows 2012 R2, my VM are both Windows 2008 R2 and Windows 2012 R2

Thanks for any advice!

Marco
Gostev
Chief Product Officer
Posts: 31455
Liked: 6646 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Backup failing with NTLMv2 security policy

Post by Gostev »

Hi, Marco. Please, open a support case for investigation, as this cannot be troubleshoot over forum posts. Thanks!
cstemaly
Enthusiast
Posts: 50
Liked: 6 times
Joined: Aug 17, 2012 12:31 pm
Contact:

Re: Backup failing with NTLMv2 security policy

Post by cstemaly »

I sent a PM to the original poster as well in case he doesn't see this, but I would like to know what the outcome was, if any, as I have the same issue. It will save Veeam a support call :)
foggy
Veeam Software
Posts: 21069
Liked: 2115 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Backup failing with NTLMv2 security policy

Post by foggy »

There's no OP's case ID here, so a better way to know the outcome is either contact him or support directly.
cstemaly
Enthusiast
Posts: 50
Liked: 6 times
Joined: Aug 17, 2012 12:31 pm
Contact:

Re: Backup failing with NTLMv2 security policy

Post by cstemaly » 1 person likes this post

OP did not call support. I did, and will report my finding. Case ID is 00721563
cstemaly
Enthusiast
Posts: 50
Liked: 6 times
Joined: Aug 17, 2012 12:31 pm
Contact:

Re: Backup failing with NTLMv2 security policy

Post by cstemaly » 2 people like this post

Turning off NTLM is not supported. See my case communication below.

My support case details:
I turned NTLM off on an unused Windows 2012 SQL server by doing this:
Open gpedit.msc (local group policy) and navigate to:
Computer Configuration\Windows Settings\Security Settings\Local Policies\Security Options

Change the policy "Network Security: Restrict NTLM: Incoming NTLM Traffic" to "Deny all accounts"

My server functioned properly afterwards, but Veeam backups failed (they're application-aware backups) due to not being able to connect to the administrative shares.

The Veeam B&R Server (v8 patch 1 running on Server 2008R2) IS able to browse via UNC path to the admin shares, though.

Is this a known issue? Are there any workarounds?

==============================================================================================================================

Response from support:
The method you use to browse to that path isn't the same method we have to use through an application. We use an RPC to access that share.

If you check your Event Viewer on the Windows 2012 SQL server and browse to the "Operational" Log located under the Applications and Services Log/Microsoft/Windows/NTLM you'll likely see an event like the following;

==Begin Event==
NTLM server blocked: Incoming NTLM traffic to servers that is blocked Calling process PID: 596 Calling process name: C:\Windows\System32\svchost.exe Calling process LUID: 0x3E4 Calling process user identity: RS-PROXYV7$ Calling process domain identity: SSA Mechanism OID: (NULL)

NTLM authentication requests to this server have been blocked.

If you want this server to allow NTLM authentication, set the security policy Network Security: Restrict NTLM: Incoming NTLM Traffic to Allow all.
==End Event==

The option you've described is incompatible with Application Aware Image Processing as we need to access the Administrative Share via RPC commands that use NTLM to authenticate.
KevinK
Enthusiast
Posts: 28
Liked: 10 times
Joined: Apr 24, 2013 9:18 am
Full Name: Kevin Kissack
Contact:

Re: Backup failing with NTLMv2 security policy

Post by KevinK » 1 person likes this post

We have "Send NTLMv2 response only/refuse LM and NTLM" set without backup issues. Application processing and indexing enabled. 2003/2008
jlockie
Novice
Posts: 7
Liked: never
Joined: Dec 23, 2015 9:41 pm
Full Name: John Lockie
Contact:

Re: Backup failing with NTLMv2 security policy

Post by jlockie »

cstemaly wrote:Turning off NTLM is not supported. See my case communication below.

The option you've described is incompatible with Application Aware Image Processing as we need to access the Administrative Share via RPC commands that use NTLM to authenticate.
Seriously? That is it? No alternative option here? No agent path, no nothing? Just...."do away with your domain security, all your domain belong to us"?

Very discouraged to read this.

I hope they understand why we are rejecting LM and NTLM auth requests? Think about it for two seconds. As a backup solution you are asking for administrative access to the systems you are backing up. OK, that's fine and understandable. Given that, you should provide a solution that sticks to best practices when protecting such privileged accounts, right? It's a joke if you are logging in as an administrator (domain or local) and allowing LM auth. Ever hear of wdigest? :roll:
jlockie
Novice
Posts: 7
Liked: never
Joined: Dec 23, 2015 9:41 pm
Full Name: John Lockie
Contact:

Re: Backup failing with NTLMv2 security policy

Post by jlockie »

Here is the guidance from Microsoft regarding LM and NTLM use: https://support.microsoft.com/en-us/kb/2793313
mma
Service Provider
Posts: 111
Liked: 21 times
Joined: Dec 22, 2011 9:12 am
Full Name: Marcel
Location: Lucerne, Switzerland
Contact:

Re: Backup failing with NTLMv2 security policy

Post by mma »

Hello Veeam

Is this still not possible in 2016? :shock:

Regards
Marcel
sandsturm
Veteran
Posts: 278
Liked: 23 times
Joined: Mar 23, 2015 8:30 am
Contact:

Re: Backup failing with NTLMv2 security policy

Post by sandsturm »

It's good to see that I'm not the only one with this problem. We don't allow any NTLM version, because NTLM is a protocol deployed the first time with Windows NT... and what year do we have actually? The hash algorithmus used whithin NTLM Is not considered safe in 2016. (MD4 for NTLM v1 and MD5 for NTLM V2) For this reason we allow Kerberos authentication only in our environment and no more NTLM. But as you said, Veeam Backup does not work with disabled NTLM (v1). So please, enable the possibility for a timely authentication mechanism in Veeam, because we don't have any Windows NT installations any more ;-)
Veeam as a backup software provider must have a serious interest in using timely mechanisms to increase the security level...
mma
Service Provider
Posts: 111
Liked: 21 times
Joined: Dec 22, 2011 9:12 am
Full Name: Marcel
Location: Lucerne, Switzerland
Contact:

Re: Backup failing with NTLMv2 security policy

Post by mma »

It's even worse if you have to use VIX instead of admin$ and have UAC enabled.
In this case you have to use the builtin administrator (.\administrator or domain\administrator)
primeaum
Lurker
Posts: 1
Liked: 1 time
Joined: Oct 21, 2016 11:38 am
Contact:

Re: Backup failing with NTLMv2 security policy

Post by primeaum » 1 person likes this post

Veeam needs to find a way to support application aware processing with NTLM disabled. We have NTLM disabled on our whole domain and cannot use AAP to backup our SQL servers. Hopefully the engineering team can see that this is an issue and figure out a way to make AAP work in the future without NTLM being required.
Andreas Neufert
VP, Product Management
Posts: 6707
Liked: 1401 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: Backup failing with NTLMv2 security policy

Post by Andreas Neufert »

mma wrote:It's even worse if you have to use VIX instead of admin$ and have UAC enabled.
In this case you have to use the builtin administrator (.\administrator or domain\administrator)
Agree on the whole request, but let me add here that the VIX limitation is by design of VMware Tools and the way it is integrated into windows and usage of acconts for their services. So this limtation will stay till vmware changes vmware tools.

As well let me please add that we discuss this whole point as well internally. Changes in the Guest Interaction process potentially affect 2.000.000+ backed up VMs which use Guest Interaction and changes there will be handled with best care from the core team at R&D and QC.
sandsturm
Veteran
Posts: 278
Liked: 23 times
Joined: Mar 23, 2015 8:30 am
Contact:

Re: Backup failing with NTLMv2 security policy

Post by sandsturm »

As well let me please add that we discuss this whole point as well internally. Changes in the Guest Interaction process potentially affect 2.000.000+ backed up VMs which use Guest Interaction and changes there will be handled with best care from the core team at R&D and QC.
Good care from R&D and QC in this topic is very important. It looks like you're on an implementation for it? Can you say a little bit more about it? Progress or approx. release date?
This would be really great and for me absolutely elementary to deploy VEEAM as a backup solution for the whole company (approximately 120 ESXi hosts) and nonetheless with a Kerberos implementation you'll keep your software on an appropriate and timely level of security...
Andreas Neufert
VP, Product Management
Posts: 6707
Liked: 1401 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: Backup failing with NTLMv2 security policy

Post by Andreas Neufert » 1 person likes this post

Based on internal and external feedback NTLMv2 only processing should work with actual versions. Kerberos only processing (disabled NTLM) has some limitations when not everything is in the same domain.

I just guess here at that point that Veeam will leave the authentication as is (extreme good and stable experience over the last 5 years) and will use Kerberos only processing as fallback. But this is only me guessing.
If this feature requests will make it into any update or v10 we will see and we have to wait on official feedback from our product management. As v10 features are not finalized yet it will take some time to say.

If you just read this post and are in the same situation please add your feedback here.
sandsturm
Veteran
Posts: 278
Liked: 23 times
Joined: Mar 23, 2015 8:30 am
Contact:

Re: Backup failing with NTLMv2 security policy

Post by sandsturm »

Based on internal and external feedback NTLMv2 only processing should work with actual versions. Kerberos only processing (disabled NTLM) has some limitations when not everything is in the same domain.
Do you mean 9.5 with actual version? Because Version 9 does not work with disabled NTLM, even everything is in the same domain.
Andreas Neufert
VP, Product Management
Posts: 6707
Liked: 1401 times
Joined: May 04, 2011 8:36 am
Full Name: Andreas Neufert
Location: Germany
Contact:

Re: Backup failing with NTLMv2 security policy

Post by Andreas Neufert » 1 person likes this post

NTLM v1 disabled should work with v9 and 9.5

Kerberos only is something that Veeam discusses for one of the next versions. v10 features are not finalized yet. It is too soon to say.
tsightler
VP, Product Management
Posts: 6009
Liked: 2842 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Backup failing with NTLMv2 security policy

Post by tsightler » 1 person likes this post

sandsturm wrote:Do you mean 9.5 with actual version? Because Version 9 does not work with disabled NTLM, even everything is in the same domain.
I think the confusion comes because this thread started off referring to issues with disabling NTLMv1 and LM (the thread title is about NTLMv2). For example, even you original post on Oct 13th says:
sandsturm wrote:But as you said, Veeam Backup does not work with disabled NTLM (v1).
I'm not aware of any issues with NTLMv2 in current versions, for example, even my lab environment runs with "Send NTLMv2 response only/refuse LM and NTLM" set in the global domain policy.

There are certainly known issues when policies disable all NTLM versions.
signal
Enthusiast
Posts: 65
Liked: 4 times
Joined: Oct 06, 2016 1:19 pm
Contact:

Re: Backup failing with NTLMv2 security policy

Post by signal »

With the domain using NTLMv2 and refusing LM and NTLM I'm having trouble with the console connecting from windows 7 client to VBR 9.5 on Windows Server 2012 R2. Eventlog shows messages about failed logins. Enabling NTLM, and still refusing LM) resolves the issue, but the security manager wants NTLMv2 only. Is there a way to solve this?
signal
Enthusiast
Posts: 65
Liked: 4 times
Joined: Oct 06, 2016 1:19 pm
Contact:

Re: Backup failing with NTLMv2 security policy

Post by signal »

Any feedback here?
tsightler
VP, Product Management
Posts: 6009
Liked: 2842 times
Joined: Jun 05, 2009 12:57 pm
Full Name: Tom Sightler
Contact:

Re: Backup failing with NTLMv2 security policy

Post by tsightler » 1 person likes this post

Assuming you've rebooted everything after the change, and the change is implemented on both the Veeam servers, as well as domain controllers and clients, I see no reason why it should not work. Here's a setup of the local policy enforced on my Veeam server, as well as all servers in my environment:

Image

Pretty much every environment I work in has NTLMv2 enforced and I can't remember having any issues in the last few years. I guess there could always be some corner case, but I'd suggest opening a ticket.
signal
Enthusiast
Posts: 65
Liked: 4 times
Joined: Oct 06, 2016 1:19 pm
Contact:

Re: Backup failing with NTLMv2 security policy

Post by signal »

tsightler wrote:Assuming you've rebooted everything after the change, and the change is implemented on both the Veeam servers, as well as domain controllers and clients, I see no reason why it should not work.
tsightler wrote:Pretty much every environment I work in has NTLMv2 enforced and I can't remember having any issues in the last few years. I guess there could always be some corner case, but I'd suggest opening a ticket.
This was in fact a problem with the client (Windows 7) having the setting lower (level 2, I think), and it needs to be level 3 or higher. No one had changed it, so this looks like it may be the default on Windows 7. This has been changed to force v2, and then it works.

Thanks for the feedback, even though we found the solution before I saw the reply. :roll:
rurouni
Enthusiast
Posts: 68
Liked: 6 times
Joined: Jul 24, 2013 7:21 pm
Contact:

Re: Backup failing with NTLMv2 security policy

Post by rurouni »

Hello,
Corp IT Security Requires that we disable NTLMv1 at Domain Controller Lever to only accept NTLMv2 communications.
Can you provide guidelines to consider Disabling NTLMv1?

At this step we disabled NTLMv1 on Server to set registry key LMCompatibility Level to 5. On AD Domain Controllers, LMCompatibilityLevel is set to 3.
upgrading to 5 triggers a gereal backup failure.

How can we avoid this?
case: 04792812
Regards
Post Reply

Who is online

Users browsing this forum: Bing [Bot], reph and 179 guests