Sudden clock drift issues when backup starts

[aeccles]

Has anyone started having clock drift issues recently? About a week or so we started noticing it on a handful of servers. We've been running vmware and veeam for years and have never had any issues. Hosts sync with ntp and the domain syncs with ntp. We can see the time drift by 15 minutes as soon as a backup starts to run and then is corrected when it ends.

[PetrM]

Hello!

Why do you think it's related to backup activity and which operation does backup job perform when you notice a clock drift, for example: building VM list or snapshot creation?

Thanks!

[aeccles]

first, correction to my post. The drift is 15 SECONDS not minutes.
I think it's related because I see event log 7045 appear (VeeamVssSupport service installed in the system) about 1 or 2 minutes before the time drift occurs, then I see the time drift correction at the conclusion of the job. This happens consistently. I have not identified the operation that causes it.

[Gostev]

Please open a support case and let them see the issue live.

[bpellegr]

What this issue resolved? I am running a POC for VDRO, and Veeam Replication of a Domain Controller is causing time sync issue with a 911 service. The customer had to stop replication.

[Mildur]

When a VmWare Snapshot is created, the time gets synced from the esxi host to the vm. The time sync setting from the vmware tools will be ignored when a Snapshot happens. This is by design.

Check the times on the esxi server. I had a similar case on my own. The time on the esxi server was not correct. Each time a backup or replication was running, times get wrong on the guests when the snapshot happened.

VMware https://kb.vmware.com/s/article/1318

These options do not disable one-time synchronizations done by VMware Tools for events such as tools startup, taking a snapshot, resuming from a snapshot, resuming from suspend, or vMotion. These events synchronize time in the guest operating system with time in the host operating system even if VMware Tools periodic time sync is disabled, so it is important to ensure that the host operating system's time is correct

[jlester]

Thank you for this! I've been chasing an intermittent time sync issue for weeks. I finally realized today it only happens after Veeam replication (and not every time). We somehow missed setting up NTP on one of our hosts. I had no idea that vSphere snapshots sync VM time to Host time by default before and after the snap, so I never even thought to check it there.

[Mildur]

Hi Jason
Your welcome