VMs Backup fails due to Trend Micro, Case ID# 02472655

[stack111]

Hi,

We receive the below error in VM backup job statistics if Trend Micro antivirus is loaded in both of guest VM OS and its Hyper-V host OS:

1/18/2018 1:14:09 PM :: Failed to create snapshot (Microsoft CSV Shadow Copy Provider) (mode: Hyper-V child partition snapshot) Details: Writer 'Microsoft Hyper-V VSS Writer' is failed at 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT'. The writer experienced a non-transient error. If the backup process is retried, the error is likely to reoccur. --tr:Failed to verify writers state. --tr:Failed to perform pre-backup tasks.

If we unload Trend Micro from both, backup goes smoothly. We followed this article to configure the AV exclusions: https://www.veeam.com/kb1999 but the same issue persists.

Any suggestion will be much appreciated.

[Mike Resseler]

Hi Ahmad,

First: Welcome to our forums!

Glad to see that you have a case ID, thanks for posting it and keep us informed about the progress.
Did you also excluded Hyper-V specific? (See here: https://support.microsoft.com/en-us/hel ... er-v-hosts)

Cheers
Mike

[stack111]

Hi,

In fact the Trend Micro AV is uninstalled now from all Hyper-V servers, but problem still exists as long as Trend Micro AV is loaded in guest VMs, backup for those VMs fails.

But if the guest VM does not have Trend Micro or Trend Micro is unloaded, backup goes smoothly.

[stack111]

Hi all,

Appreciate your guidance on this issue..

[Mike Resseler]

Does this happen only when guest processing is enabled? Or even without it?

Also, is this only AV or is it the suite with firewall and so on?

[stack111]

It happens either guest processing is enabled or not enabled.

AV enabled features are as below:

1. Behavior Monitoring
2. Device Control
3. Real-Time Scan
4. Suspicious Connection Service
5. Web Reputation

AV disabled features are below:

1. Data Loss Prevention
2. Firewall
3. Outbreak Prevention Policy
4. Smart Scan

[Mike Resseler]

I am assuming that something gets blocked by behavior monitoring. I am surprised though that it doesn't work when you disable guest processing. Make sure you update the case with all that information.

Please keep working with support to figure out what is happening, and can you also check the logs of Trend to see if it is notifying you that it stops something?

Thanks
Mike

[stack111]

Hi,
I updated the case but still waiting support reply since last week, hopefully they get back to me today.
For trend micro logs, I will check with our customer and report back findings here.

[stack111]

Hi,

As per our customer Trend Micro partner, please find the below:

1. They updated the Trend Micro license
2. Firewall already disabled.
3. There is no logs in Antivirus on of the affected VMs that are not getting backed up.
4. Collected the logs (Excel files), please send it to Veeam team to check if there is directory belong to Veeam is blocked or no.

[Mike Resseler]

Ahmad,

Sorry, I am a bit confused at this time. Is that what Trend said? Did you added all that information to the support call?

Thanks
Mike

[stack111]

Yes, this is what Trend Micro partner told our customer.

I added already all these information to the support call.

[Mike Resseler]

OK. Cool. Thanks

[stack111]

This is last message from Veeam support:

From the case notes backup does not work for vms with Trendmicro AV installed. Job fails with following message
"Failed to create snapshot (Microsoft CSV Shadow Copy Provider) (mode: Hyper-V child partition snapshot) Details: Writer 'Microsoft Hyper-V VSS Writer' is failed at 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT'.

According to the logs, veeam component is not uploaded/installed on the guest machine, so AV cannot block any communication from veeam server to guest machine, because connect is not set up.

In the event logs of the guest vm I see
Volume Shadow Copy Service error: Error calling a routine on a Shadow Copy Provider {74600e39-7dc5-4567-a03b-f091d6c7b092}. Routine details PostFinalCommitSnapshots({92203ad0-6e94-424e-9372-58e05258aa30}, 3) [hr = 0x80070005, Access is denied.

When the backup is taken 2 checkpoint files are created on the host, then one of the checkpoints is mounted to the guest vm. AD service opens AD Database on this new volume to finish VSS operation.
It seems that AV blocks the access to this disk.

Usually if the issue is isolated to AV software and there are no veeam components installed on the system where AV prevents the backup we recommend to open a case with AV vendor so that it can help to tune the Antivirus.

I can offer to setup a remote session to help you with collecting additional details before addressing the issue to the AV vendor.

[stack111]

I asked the support to collect needed information to be shared with Trend Micro vendor in order to tune the Antivirus.

In fact, Trend Micro vendor is waiting for us to provide needed exclusions in order for Veeam backup to work

[Mike Resseler]

Ahmad,

Could you look into event viewer (both host and guest) and look for VSS errors? Second. I just see AD. So we are talking about a domain controller? Any change this VM also runs DHCP/ DNS?

And last, now is the time to ask what OS the host and what OS the guest is

[stack111]

This VM is a DC with DHCP and DNS.

However the issue is not for this VM, any VM with trend micro loaded, Veeam cannot do a backup for it, Veeam support know this.

all Windows VMs and Hyper-V hosts are Windows 2012 R2 .

VSS errors in event viewer in guest Vms and hosts wil be shared here today later inshallah.

[stack111]

Hi,

Please find below findings from today remote session with Veeam support:

1. Trend Micro is loaded (services running) into guest VM.
2. When the shadow copy is taken from the host (by Veeam backup), there is a checkpoint file created next to the VM disk; this checkpoint file is then connected to the guest VM and we see it its disk management
3. After the disk is connected, it got to be brought online and mounted so that backup started to take backup from the VM original VHDx file.
4. However, the disk is not brought online, it seems AV prevents disk from being mounted and online, it could be AV is trying to scan the new disk when it tries to come online or trying to scan the boot sector or something. The disk management as a result becomes unresponsive.
5. If we unload Trend Micro (services stopped), the disk created in step 2 above when backup starts is brought online successfully and backup finishes successfully.
6. Per Veeam support, they isolated the issue and found out what can be presented to Trend Micro support.

Maybe there is some options in Trend Micro that you can tune to prevent Trend Micro from accessing the newly created checkpoint (disk) or something..

[Mike Resseler]

Well,

I'm glad that they could find the issue.

I am assuming now that you need to exclude the following directories:

C:\ProgramData\Microsoft\Windows\Hyper-V\Snapshots
"\Device\HardDiskVolumeShadowCopy*"

And basically any location where your original disks reside

(Did you excluded all typical hyper-v exclusions?)

[stack111]

Hi, we did configure all exclusions for Hyper-V and Veeam, also "\Device\HardDiskVolumeShadowCopy*" and
C:\ProgramData\Microsoft\Windows\Hyper-V\Snapshots also excluded.

[Mike Resseler]

Then I would talk to Trend Micro again and explain what our support has seen. Hopefully they can tell you what needs to be excluded

[stack111]

Yes we did already communicated with our customer and ask them to communicate with their Trend Micro partner to look into this issue

[Mike Resseler]

Perfect.

Please keep us informed here also. We are also interested to learn the solution

Many thanks
Mike

[Rumple]

out of curiosity is this trend office scan or is this the new trend Next Gen client that is giving all the grief? I have an enterprise client trying to use NGen on their windows 10 machines and running into all kinds of oddities, like pulse VPN installation failing (but an upgrade of an existing vpn driver works),etc....officescan was always pretty good and stable, but I'm leery of this Ngen client...

[stack111]

This is trend micro office scan.

Today Trend Micro support asked to disable the below options on the antivirus client on one of the guest VMs and test backup, but backup still failed after disabling them as the disk is not coming online in the guest VM

Scan Network Drive
Scan The Boot Sector Of The USB Storage Device After Plugging In
Scan All Files In Removable Storage Devices After Plugging In

[Mike Resseler]

These seem like some options that really don't matter imho.

Was this only done at the guest VM or also at the host VM?

[stack111]

Currently at the host there is no Trend Micro agent, only in guest VMs.

I received the below today from Trend Micro support:

We still need to isolate the issue. OfficeScan have different features and for us to isolate what feature is causing the issue we need to do trial and error by disabling each of the components.

Please provide me your availability so that we can perform this isolation. Once we have identified what feature is causing the issue then we can look for a solution or escalate the case to our Product Developers.

[Mike Resseler]

That's good news in my book. That means they want to investigate what is going on. Keep us informed!

[nmdange]

Check if Trend Micro disabled volume automount. You can check this by opening a command prompt, typing "diskpart", and then "automount". If automount is disabled, you can enable it with diskpart by doing "automount enabled".

[stack111]

I ran below commands

C:\Windows\system32>diskpart

Microsoft DiskPart version 6.3.9600

Copyright (C) 1999-2013 Microsoft Corporation.
On computer: xxxx

DISKPART> automount

Automatic mounting of new volumes enabled.

DISKPART> automount enable

Automatic mounting of new volumes enabled.

DISKPART>

Then retried backup but same issue..

[stack111]

Problem is solved; backup is working now for guest VMs.

The issue was that \Device\HarddiskVolume* was not excluded, once excluded in Trend Micro, backup is working!

I think Veeam should update their needed antivirus exclusion list in this article: https://www.veeam.com/kb1999

We found that \Device\HarddiskVolume* should be excluded from this article: https://kc.mcafee.com/corporate/index?p ... id=KB54457