Maintain control of your Microsoft Office 365 email data
Post Reply
obroni
Service Provider
Posts: 103
Liked: 10 times
Joined: Nov 21, 2014 10:50 pm
Full Name: Nick Fisk
Contact:

IPv6 Problems

Post by obroni » Aug 31, 2017 3:58 pm

Has anybody been experiencing problems with the 1.5 Beta and IPv6 in the last few days?

I installed the beta a couple of weeks ago on a server which is dual stacked and ran fine for a number of days. But the job has been failing for the last few days when it tries to connect to O365 at the start. If I disable IPv6 on the server, the job runs through fine.

I'm guessing its some sort of DNS/IPv6 issue as I can see nslookup returns multiple AAAA records and if I run the Organisation wizard, it does finally complete but hangs for several minutes at each step.

foggy
Veeam Software
Posts: 16691
Liked: 1343 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: IPv6 Problems

Post by foggy » Sep 01, 2017 12:45 pm

Hi Nick, were there any changes to network equipment configuration performed recently? If not, please provide the logs for the Organization Wizard and the failed job, guys will take a closer look at possible reasons.

obroni
Service Provider
Posts: 103
Liked: 10 times
Joined: Nov 21, 2014 10:50 pm
Full Name: Nick Fisk
Contact:

Re: IPv6 Problems

Post by obroni » Sep 06, 2017 11:50 am

This is the error from the log

Code: Select all

30/08/2017 17:40:22   24 (5896) Error: Connecting to remote server outlook.office365.com failed with the following error message : WinRM cannot complete the operation. Verify that the specified computer name is valid, that the computer is accessible over the network, and that a firewall exception for the WinRM service is enabled and allows access from this computer. By default, the WinRM firewall exception for public profiles limits access to remote computers within the same local subnet. For more information, see the about_Remote_Troubleshooting Help topic.
30/08/2017 17:40:22   24 (5896) Stack:
30/08/2017 17:40:22   24 (5896)    at System.Management.Automation.Runspaces.AsyncResult.EndInvoke()
   at System.Management.Automation.Runspaces.Internal.RunspacePoolInternal.EndOpen(IAsyncResult asyncResult)
   at Veeam.Archiver.Source.SourcePSOrganization.ConnectionAsync(WSManConnectionInfo connInfo, WaitHandle cancelEvent)
   at Veeam.Archiver.Source.SourcePSOrganization.ConnectInternal(WSManConnectionInfo connInfo, CancellationToken cancel)
   at Veeam.Archiver.Source.SourcePSOrganization.Connect(WSManConnectionInfo connInfo, CancellationToken cancel)
   at Veeam.Archiver.Source.SourceOrganization.ConnectO365(String account, SecureString password, OfficeRegion region, CancellationToken cancel)
   at Veeam.Archiver.Source.SourceOrganization.Connect(ISourceOrganizationConfig config, CancellationToken cancel)
   at Veeam.Archiver.Source.JobActivator.DoWork(CancellationToken cancel)
   at Veeam.Archiver.Source.WorkAction.WorkWrap(CancellationToken cancel)
I've noticed at home over the last few weeks that O365 Skype stops working on my IPv6 enabled home broadband until I reboot my router. I suspect that some sort of incorrect AAAA dns entry is being cached somewhere, but I haven't managed to pinpoint whats going on so far. I'm suspicious that something similar may be happening with this as well.

Mike Resseler
Veeam Software
Posts: 4668
Liked: 498 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: IPv6 Problems

Post by Mike Resseler » Sep 06, 2017 12:02 pm

Nick,

From what I can see, you cannot reach to outlook.office365.com

Can you do a ping -6 outlook.office365.com from that location? (Most likely the request will time out but want to know what it translates into)

Mike

obroni
Service Provider
Posts: 103
Liked: 10 times
Joined: Nov 21, 2014 10:50 pm
Full Name: Nick Fisk
Contact:

Re: IPv6 Problems

Post by obroni » Sep 06, 2017 12:29 pm

Code: Select all

Pinging outlook-namcentral.office365.com [2603:1036:3:17::2] with 32 bytes of data:
Request timed out.
Request timed out.
Request timed out.
Request timed out.

Ping statistics for 2603:1036:3:17::2:
    Packets: Sent = 4, Received = 0, Lost = 4 (100% loss),

Mike Resseler
Veeam Software
Posts: 4668
Liked: 498 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: IPv6 Problems

Post by Mike Resseler » Sep 06, 2017 12:44 pm

Nick,
I am not 100 percent certain but I think that might be a cached one. MSFT has made some changes to their IPv6 addresses 31th of August (a few days ago). I can ping north america central proxy.
Any change you can flush the server dns cache and maybe even other networking gear also?

obroni
Service Provider
Posts: 103
Liked: 10 times
Joined: Nov 21, 2014 10:50 pm
Full Name: Nick Fisk
Contact:

Re: IPv6 Problems

Post by obroni » Sep 06, 2017 1:02 pm

I will give that a go, but I've noticed something very weird. For some reason that address works fine from Linux servers but not windows servers. Am I missing something obvious here?

Tried from multiple windows servers with all the same results:

Code: Select all

Tracing route to outlook-emeacenter4.office365.com [2603:1026:203::2]
over a maximum of 30 hops:

  1     *        *        *     Request timed out.
  2     *        *        *     Request timed out.
  3     *        *        *     Request timed out.
  4  ^C
And this to the same IP from any Linux based server (all on the same L2 subnet as the windows servers, ie same gateway)

Code: Select all

traceroute to 2603:1026:203::2 (2603:1026:203::2) from 2a03:25e0:254:1::40, 30 hops max, 24 byte packets
 1  2a03:25e0:254:1::2  0.473 ms  0.325 ms  0.211 ms
 2  2a03:25e0:254:199::2  0.143 ms  0.102 ms  0.143 ms
 3  2a03:25e0:201::3  0.376 ms  0.297 ms  0.216 ms
 4  fdd6:8544:2820:0:dead:beef:cafe:11  0.432 ms  0.227 ms  0.192 ms
 5  fdd6:8544:2820:0:dead:beef:cafe:3  0.342 ms  0.165 ms  0.181 ms
 6  2a03:25e0:200:0:6273:5cff:fe6b:1f40  0.285 ms  0.261 ms  0.506 ms
 7  2a02:400:1::5:1  0.34 ms  0.29 ms  0.28 ms
 8  2a02:400::1000:31:2  0.354 ms  0.347 ms  0.351 ms
 9  2a02:400::1000:3b:1  6.34 ms  6.267 ms  6.241 ms
10  2001:1900:5:2:2::322d  7.907 ms  7.438 ms  7.656 ms
11  2001:1900::4:7  6.676 ms  6.694 ms  6.684 ms
12  2001:1900:5:3::11a  6.877 ms  6.888 ms  6.861 ms
13  2603:1060:1:12::25  14.989 ms  14.819 ms  15.197 ms
14  2603:1060:1:10::55  14.638 ms  14.985 ms  14.98 ms
15  2603:1060:1:10::2e  15.332 ms  14.748 ms  15.16 ms
16  2603:1060:1:10::79  16.249 ms  14.874 ms  14.995 ms
17  2603:1060:1:12::10a  14.814 ms  14.506 ms  31.356 ms
18  2603:10a0:309:9100::a  13.822 ms  13.737 ms  13.913 ms
19  2603:10a0:309:9000::c6  14.265 ms  14.189 ms  14.18 ms
20  2603:10a0:309:9304::1ae  13.911 ms  13.727 ms  13.73 ms
21  2603:10a0:302:8302::  14.359 ms  14.277 ms  14.296 ms
22  2603:10a0:302:8300::6  13.998 ms  13.919 ms  13.862 ms

ipv6 route table from windows servers:

Code: Select all

IPv6 Route Table
===========================================================================
Active Routes:
 If Metric Network Destination      Gateway
  2    271 ::/0                     fe80::1
  2    271 ::/0                     fe80::250:56ff:fe81:ef51
  1    331 ::1/128                  On-link
  2    271 2a03:25e0:211::/48       On-link
  2    271 2a03:25e0:254:1::/64     On-link
  2    271 2a03:25e0:254:1::122/128 On-link
  2    271 fe80::/64                On-link
  2    271 fe80::2c5e:9c3:452a:311e/128
                                    On-link
  1    331 ff00::/8                 On-link
  2    271 ff00::/8                 On-link
===========================================================================
Persistent Routes:
 If Metric Network Destination      Gateway
  0 4294967295 ::/0                     fe80::1

And just for sanity checks, same windows server to ipv6 http://www.google.com
Tracing route to http://www.google.com [2a00:1450:4009:809::2004]
over a maximum of 30 hops:

1 <1 ms <1 ms <1 ms 2a03:25e0:254:1::2
2 <1 ms <1 ms <1 ms 2a03:25e0:254:199::2
3 <1 ms <1 ms <1 ms 2a03:25e0:201::2
4 <1 ms <1 ms <1 ms 2a03:25e0:200:0:6273:5cff:fe6f:d340
5 <1 ms <1 ms <1 ms 2a02:400:1::6:1
6 * ^C

obroni
Service Provider
Posts: 103
Liked: 10 times
Joined: Nov 21, 2014 10:50 pm
Full Name: Nick Fisk
Contact:

Re: IPv6 Problems

Post by obroni » Sep 06, 2017 2:00 pm

Ok, I think I have solved this now. I found a VM on the same subnet which was sending out router advertisements. Not sure why Windows was randomly preferring this gateway advertisement over the statically set one, that seems a bit of a security flaw to me. Our actual gateway is fe80::1 and the clue was in the route table I posted above. That other default route entry was the rogue VM.

Mike Resseler
Veeam Software
Posts: 4668
Liked: 498 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: IPv6 Problems

Post by Mike Resseler » Sep 06, 2017 2:08 pm

Does this mean you can connect again?

obroni
Service Provider
Posts: 103
Liked: 10 times
Joined: Nov 21, 2014 10:50 pm
Full Name: Nick Fisk
Contact:

Re: IPv6 Problems

Post by obroni » Sep 06, 2017 2:27 pm

Yep, sorry for the false alarm. And just for completeness, it seems that Ubuntu Linux (which form the bulk of our VM's) disables acceptance of RA's when you configure a static ipv6 gateway. Where by default, Windows will accept a RA and give it the same metric as a statically configured gateway.

Mike Resseler
Veeam Software
Posts: 4668
Liked: 498 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: IPv6 Problems

Post by Mike Resseler » Sep 06, 2017 2:30 pm

Good to know! And thanks for letting us know that everything is solved now.

Cheers
Mike

Post Reply

Who is online

Users browsing this forum: No registered users and 1 guest