Time shift on backed up Windows Server...

[captainfeeney]

Hello there, I wonder if anyone would be in a position to assist me, please... I am experiencing a consistent time shift on a backed up Windows 2008 R2 Standard VM Server, notwithstanding that I have been diligent in ensuring that all of the steps within the following Veean KB have been implemented correctly - https://www.veeam.com/kb1202

I have also disabled time synchronisation as per the following VMware KB - https://kb.vmware.com/selfservice/micro ... nalId=1189

Completing the following command to the host indicates that it is correctly synching time from the configured, expected and validated ntp time source rather than the Local CMOS Clock.

C:\Windows\system32>w32tm /query /source
ntp.foo.bar

There is no indication of any failed CMOS battery messages in any of the ESXi hosts - either in their iLO or IML's logs...

Time configuration within the Configuration --> Time Configuration --> NTP Daemon (ntpd) options is correct and in-line with recommended settings on all of the ESXi hosts from the VMware vSphere Client view

I have reviewed the ntpd log entries in the /var/log/messages log files to determine (and confirm) that the NTP daemon has synchronized with the remote NTP server.

Any (and all) assistance that anyone within the community would be so kind as to share with me would be gratefully received to resolve this issue... thanks in advance,

[foggy]

Mark, I'd appreciate if you could post your case ID here, as per forum rules.

[captainfeeney]

Hello - My apologies for not posting my case ID in the first post - the case ID relating to my query is Case #01722861

[GuilleGo]

Hello

Captainfeeney, Have you ever find a solution? I´m having the same issue on a windows server 2008 R2 and followed same Veeam kb1202.

The time source for the windows server 2008 R2 is a domain controller and vmware tools option "synchronize guest time with host" is disabled, but in every veeam snapshot creation/deletion the time shift happens and a new record kernel-general event id 1 is added to the system-event viewer.

We have veeam backup & replication 8.0.0.817

thanks in advance

[captainfeeney]

Hello there GuilleGo...

No, I've still not arrived at a solution to this problem - I can state without reservation that it is not a problem caused by the Veem backup agent engaging or disengaging itself against the host in question. I've worked this through exhaustively with Veeam support so as to arrive at this conclusion.

Enabling debug logging against the Windows Time service shows the following at the point in time when the skew occurs - note that the date enumerates in ANSI so needs to be transliterated forwards to reflect the correct date:

Code: Select all

151716 01:00:12.2783153s - ClockDispln Update: SO:2004579908 KPhO:-731642 *PhO:2005311550 uT:4103 SD:185437 (i) LI:0 S:6 RDl:312500 RDs:286426 TSF:0x0 Unset->Hold
151716 01:03:32.7363061s - ClockDispln Discipline: *SET*TIME* - PhCR:5568 KPho:2004579908
151716 01:03:32.7363061s - ClockDispln Discipline: *SKEW*TIME* - PhCRR:0 CR:156016 PhCR:5568 UI:360000 phcT:4103 KPhO:0
151716 01:03:32.7519076s - TimeProvCommand([NtpClient], TPC_TimeJumped) called.
151716 01:03:32.7519076s - Peer poll: Max:64.0000000s Cur:00.0000000s
151716 01:03:32.7519076s - W32TmServiceMain: waiting 959.987s
151716 01:03:32.7519076s - PeerPollingThread: PeerListUpdated
151716 01:03:32.7519076s - Polling peer ntp.acas.com,0x8 (ntp.m|0x8|0.0.0.0:123->10.25.1.1:123)
151716 01:03:32.7519076s - W32TmServiceMain: ********** Time Slip Notification **********
151716 01:03:32.7519076s - ClockDispln TimeSlip: SetUnsync: LI:0 S:6 RDl:0 RDs:100000000 TSF:0x0 ReleaseClock2CMOS

I've no indication at this point in time exactly why / what has caused the time slip. Once I have this information I will update this thread to reflect the solution that I have diagnosed.

[skrause]

Time-slip on Windows guests in VMware is quite common (thus why the synchronize with host option is popular).

From what I remember, this is caused by the CPU scheduler in VMware delaying processes from a given VM while it waits for the available resources on the host. This slows down the effective CPU clock. Now I don't know the exact inner workings of the Windows clock but I am guessing that it keeps time by counting a number of cycles based on what CPU frequency is reported to it by the hardware, so when the clock is actually slower than what Windows thinks it should be, the time slips behind.

By chance do these guests have high CPU counts (4+)?

[captainfeeney]

Thank you for coming back to me on this skrause... it's appreciated. The hosts indeed do have a high CPU count.

One piece of information that I may not have provided to the community is the fact that the time slip occurs only once a week, and this behaviour has been consistently occurring in the early hours of a Saturday morning.

Bearing the aforementioned in mind, where you able to resolve this issue for yourself on your Windows VMware guests, please?

If this was the case, how was it that you effected a resolution please?

[skrause]

My fix was simple: I synchronize the vm clock with the host which we have synchronizing with NTP.

If your guests have high CPU counts, the way VMware's CPU scheduler works is it has to have a free core for all of the guest vCPU before it will execute a task. Thus, 1-2 core VMs are best if possible.

If this is only happening on the weekends, the question I would ask is: Do you run active fulls on the weekend? The snapshot stun for an active full can be pretty long and I believe that can be a cause of time slippage as well.