Discussions specific to the Microsoft Hyper-V hypervisor
stack111
Enthusiast
Posts: 36
Liked: 1 time
Joined: Jan 18, 2018 8:15 am
Full Name: Ahmad Jayyusi
Contact:

VMs Backup fails due to Trend Micro, Case ID# 02472655

Post by stack111 » Jan 25, 2018 6:05 am

Hi,

We receive the below error in VM backup job statistics if Trend Micro antivirus is loaded in both of guest VM OS and its Hyper-V host OS:

1/18/2018 1:14:09 PM :: Failed to create snapshot (Microsoft CSV Shadow Copy Provider) (mode: Hyper-V child partition snapshot) Details: Writer 'Microsoft Hyper-V VSS Writer' is failed at 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT'. The writer experienced a non-transient error. If the backup process is retried, the error is likely to reoccur. --tr:Failed to verify writers state. --tr:Failed to perform pre-backup tasks.

If we unload Trend Micro from both, backup goes smoothly. We followed this article to configure the AV exclusions: https://www.veeam.com/kb1999 but the same issue persists.

Any suggestion will be much appreciated.

Mike Resseler
Veeam Software
Posts: 5000
Liked: 530 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: VMs Backup fails due to Trend Micro, Case ID# 02472655

Post by Mike Resseler » Jan 25, 2018 7:34 am

Hi Ahmad,

First: Welcome to our forums!

Glad to see that you have a case ID, thanks for posting it and keep us informed about the progress.
Did you also excluded Hyper-V specific? (See here: https://support.microsoft.com/en-us/hel ... er-v-hosts)

Cheers
Mike

stack111
Enthusiast
Posts: 36
Liked: 1 time
Joined: Jan 18, 2018 8:15 am
Full Name: Ahmad Jayyusi
Contact:

Re: VMs Backup fails due to Trend Micro, Case ID# 02472655

Post by stack111 » Jan 27, 2018 6:52 pm

Hi,

In fact the Trend Micro AV is uninstalled now from all Hyper-V servers, but problem still exists as long as Trend Micro AV is loaded in guest VMs, backup for those VMs fails.

But if the guest VM does not have Trend Micro or Trend Micro is unloaded, backup goes smoothly.

stack111
Enthusiast
Posts: 36
Liked: 1 time
Joined: Jan 18, 2018 8:15 am
Full Name: Ahmad Jayyusi
Contact:

Re: VMs Backup fails due to Trend Micro, Case ID# 02472655

Post by stack111 » Jan 28, 2018 2:32 pm

Hi all,

Appreciate your guidance on this issue..

Mike Resseler
Veeam Software
Posts: 5000
Liked: 530 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: VMs Backup fails due to Trend Micro, Case ID# 02472655

Post by Mike Resseler » Jan 29, 2018 6:08 am

Does this happen only when guest processing is enabled? Or even without it?

Also, is this only AV or is it the suite with firewall and so on?

stack111
Enthusiast
Posts: 36
Liked: 1 time
Joined: Jan 18, 2018 8:15 am
Full Name: Ahmad Jayyusi
Contact:

Re: VMs Backup fails due to Trend Micro, Case ID# 02472655

Post by stack111 » Jan 29, 2018 6:46 am

It happens either guest processing is enabled or not enabled.

AV enabled features are as below:

1. Behavior Monitoring
2. Device Control
3. Real-Time Scan
4. Suspicious Connection Service
5. Web Reputation

AV disabled features are below:

1. Data Loss Prevention
2. Firewall
3. Outbreak Prevention Policy
4. Smart Scan

Mike Resseler
Veeam Software
Posts: 5000
Liked: 530 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: VMs Backup fails due to Trend Micro, Case ID# 02472655

Post by Mike Resseler » Jan 29, 2018 6:51 am

I am assuming that something gets blocked by behavior monitoring. I am surprised though that it doesn't work when you disable guest processing. Make sure you update the case with all that information.

Please keep working with support to figure out what is happening, and can you also check the logs of Trend to see if it is notifying you that it stops something?

Thanks
Mike

stack111
Enthusiast
Posts: 36
Liked: 1 time
Joined: Jan 18, 2018 8:15 am
Full Name: Ahmad Jayyusi
Contact:

Re: VMs Backup fails due to Trend Micro, Case ID# 02472655

Post by stack111 » Jan 29, 2018 7:03 am

Hi,
I updated the case but still waiting support reply since last week, hopefully they get back to me today.
For trend micro logs, I will check with our customer and report back findings here.

stack111
Enthusiast
Posts: 36
Liked: 1 time
Joined: Jan 18, 2018 8:15 am
Full Name: Ahmad Jayyusi
Contact:

Re: VMs Backup fails due to Trend Micro, Case ID# 02472655

Post by stack111 » Jan 31, 2018 2:52 pm

Hi,

As per our customer Trend Micro partner, please find the below:

1. They updated the Trend Micro license
2. Firewall already disabled.
3. There is no logs in Antivirus on of the affected VMs that are not getting backed up.
4. Collected the logs (Excel files), please send it to Veeam team to check if there is directory belong to Veeam is blocked or no.

Mike Resseler
Veeam Software
Posts: 5000
Liked: 530 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: VMs Backup fails due to Trend Micro, Case ID# 02472655

Post by Mike Resseler » Feb 01, 2018 8:13 am

Ahmad,

Sorry, I am a bit confused at this time. Is that what Trend said? Did you added all that information to the support call?

Thanks
Mike

stack111
Enthusiast
Posts: 36
Liked: 1 time
Joined: Jan 18, 2018 8:15 am
Full Name: Ahmad Jayyusi
Contact:

Re: VMs Backup fails due to Trend Micro, Case ID# 02472655

Post by stack111 » Feb 01, 2018 9:22 am

Yes, this is what Trend Micro partner told our customer.

I added already all these information to the support call.

Mike Resseler
Veeam Software
Posts: 5000
Liked: 530 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: VMs Backup fails due to Trend Micro, Case ID# 02472655

Post by Mike Resseler » Feb 01, 2018 9:23 am

OK. Cool. Thanks

stack111
Enthusiast
Posts: 36
Liked: 1 time
Joined: Jan 18, 2018 8:15 am
Full Name: Ahmad Jayyusi
Contact:

Re: VMs Backup fails due to Trend Micro, Case ID# 02472655

Post by stack111 » Feb 02, 2018 4:24 pm

This is last message from Veeam support:

From the case notes backup does not work for vms with Trendmicro AV installed. Job fails with following message
"Failed to create snapshot (Microsoft CSV Shadow Copy Provider) (mode: Hyper-V child partition snapshot) Details: Writer 'Microsoft Hyper-V VSS Writer' is failed at 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT'.

According to the logs, veeam component is not uploaded/installed on the guest machine, so AV cannot block any communication from veeam server to guest machine, because connect is not set up.

In the event logs of the guest vm I see
Volume Shadow Copy Service error: Error calling a routine on a Shadow Copy Provider {74600e39-7dc5-4567-a03b-f091d6c7b092}. Routine details PostFinalCommitSnapshots({92203ad0-6e94-424e-9372-58e05258aa30}, 3) [hr = 0x80070005, Access is denied.

When the backup is taken 2 checkpoint files are created on the host, then one of the checkpoints is mounted to the guest vm. AD service opens AD Database on this new volume to finish VSS operation.
It seems that AV blocks the access to this disk.

Usually if the issue is isolated to AV software and there are no veeam components installed on the system where AV prevents the backup we recommend to open a case with AV vendor so that it can help to tune the Antivirus.

I can offer to setup a remote session to help you with collecting additional details before addressing the issue to the AV vendor.

stack111
Enthusiast
Posts: 36
Liked: 1 time
Joined: Jan 18, 2018 8:15 am
Full Name: Ahmad Jayyusi
Contact:

Re: VMs Backup fails due to Trend Micro, Case ID# 02472655

Post by stack111 » Feb 02, 2018 4:36 pm

I asked the support to collect needed information to be shared with Trend Micro vendor in order to tune the Antivirus.

In fact, Trend Micro vendor is waiting for us to provide needed exclusions in order for Veeam backup to work

Mike Resseler
Veeam Software
Posts: 5000
Liked: 530 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: VMs Backup fails due to Trend Micro, Case ID# 02472655

Post by Mike Resseler » Feb 05, 2018 6:32 am

Ahmad,

Could you look into event viewer (both host and guest) and look for VSS errors? Second. I just see AD. So we are talking about a domain controller? Any change this VM also runs DHCP/ DNS?

And last, now is the time to ask what OS the host and what OS the guest is :-)

Post Reply

Who is online

Users browsing this forum: PTide and 9 guests