VMs Backup fails due to Trend Micro, Case ID# 02472655

Discussions specific to Microsoft Hyper-V hypervisor

VMs Backup fails due to Trend Micro, Case ID# 02472655

Veeam Logoby stack111 » Thu Jan 25, 2018 6:05 am

Hi,

We receive the below error in VM backup job statistics if Trend Micro antivirus is loaded in both of guest VM OS and its Hyper-V host OS:

1/18/2018 1:14:09 PM :: Failed to create snapshot (Microsoft CSV Shadow Copy Provider) (mode: Hyper-V child partition snapshot) Details: Writer 'Microsoft Hyper-V VSS Writer' is failed at 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT'. The writer experienced a non-transient error. If the backup process is retried, the error is likely to reoccur. --tr:Failed to verify writers state. --tr:Failed to perform pre-backup tasks.

If we unload Trend Micro from both, backup goes smoothly. We followed this article to configure the AV exclusions: https://www.veeam.com/kb1999 but the same issue persists.

Any suggestion will be much appreciated.
stack111
Enthusiast
 
Posts: 27
Liked: 1 time
Joined: Thu Jan 18, 2018 8:15 am
Full Name: Ahmad Jayyusi

Re: VMs Backup fails due to Trend Micro, Case ID# 02472655

Veeam Logoby Mike Resseler » Thu Jan 25, 2018 7:34 am

Hi Ahmad,

First: Welcome to our forums!

Glad to see that you have a case ID, thanks for posting it and keep us informed about the progress.
Did you also excluded Hyper-V specific? (See here: https://support.microsoft.com/en-us/hel ... er-v-hosts)

Cheers
Mike
Mike Resseler
Veeam Software
 
Posts: 4225
Liked: 471 times
Joined: Fri Feb 08, 2013 3:08 pm
Location: Belgium, the land of the fries, the beer, the chocolate and the diamonds...
Full Name: Mike Resseler

Re: VMs Backup fails due to Trend Micro, Case ID# 02472655

Veeam Logoby stack111 » Sat Jan 27, 2018 6:52 pm

Hi,

In fact the Trend Micro AV is uninstalled now from all Hyper-V servers, but problem still exists as long as Trend Micro AV is loaded in guest VMs, backup for those VMs fails.

But if the guest VM does not have Trend Micro or Trend Micro is unloaded, backup goes smoothly.
stack111
Enthusiast
 
Posts: 27
Liked: 1 time
Joined: Thu Jan 18, 2018 8:15 am
Full Name: Ahmad Jayyusi

Re: VMs Backup fails due to Trend Micro, Case ID# 02472655

Veeam Logoby stack111 » Sun Jan 28, 2018 2:32 pm

Hi all,

Appreciate your guidance on this issue..
stack111
Enthusiast
 
Posts: 27
Liked: 1 time
Joined: Thu Jan 18, 2018 8:15 am
Full Name: Ahmad Jayyusi

Re: VMs Backup fails due to Trend Micro, Case ID# 02472655

Veeam Logoby Mike Resseler » Mon Jan 29, 2018 6:08 am

Does this happen only when guest processing is enabled? Or even without it?

Also, is this only AV or is it the suite with firewall and so on?
Mike Resseler
Veeam Software
 
Posts: 4225
Liked: 471 times
Joined: Fri Feb 08, 2013 3:08 pm
Location: Belgium, the land of the fries, the beer, the chocolate and the diamonds...
Full Name: Mike Resseler

Re: VMs Backup fails due to Trend Micro, Case ID# 02472655

Veeam Logoby stack111 » Mon Jan 29, 2018 6:46 am

It happens either guest processing is enabled or not enabled.

AV enabled features are as below:

1. Behavior Monitoring
2. Device Control
3. Real-Time Scan
4. Suspicious Connection Service
5. Web Reputation

AV disabled features are below:

1. Data Loss Prevention
2. Firewall
3. Outbreak Prevention Policy
4. Smart Scan
stack111
Enthusiast
 
Posts: 27
Liked: 1 time
Joined: Thu Jan 18, 2018 8:15 am
Full Name: Ahmad Jayyusi

Re: VMs Backup fails due to Trend Micro, Case ID# 02472655

Veeam Logoby Mike Resseler » Mon Jan 29, 2018 6:51 am

I am assuming that something gets blocked by behavior monitoring. I am surprised though that it doesn't work when you disable guest processing. Make sure you update the case with all that information.

Please keep working with support to figure out what is happening, and can you also check the logs of Trend to see if it is notifying you that it stops something?

Thanks
Mike
Mike Resseler
Veeam Software
 
Posts: 4225
Liked: 471 times
Joined: Fri Feb 08, 2013 3:08 pm
Location: Belgium, the land of the fries, the beer, the chocolate and the diamonds...
Full Name: Mike Resseler

Re: VMs Backup fails due to Trend Micro, Case ID# 02472655

Veeam Logoby stack111 » Mon Jan 29, 2018 7:03 am

Hi,
I updated the case but still waiting support reply since last week, hopefully they get back to me today.
For trend micro logs, I will check with our customer and report back findings here.
stack111
Enthusiast
 
Posts: 27
Liked: 1 time
Joined: Thu Jan 18, 2018 8:15 am
Full Name: Ahmad Jayyusi

Re: VMs Backup fails due to Trend Micro, Case ID# 02472655

Veeam Logoby stack111 » Wed Jan 31, 2018 2:52 pm

Hi,

As per our customer Trend Micro partner, please find the below:

1.They updated the Trend Micro license
2.Firewall already disabled.
3.There is no logs in Antivirus on of the affected VMs that are not getting backed up.
4.Collected the logs (Excel files), please send it to Veeam team to check if there is directory belong to Veeam is blocked or no.
stack111
Enthusiast
 
Posts: 27
Liked: 1 time
Joined: Thu Jan 18, 2018 8:15 am
Full Name: Ahmad Jayyusi

Re: VMs Backup fails due to Trend Micro, Case ID# 02472655

Veeam Logoby Mike Resseler » Thu Feb 01, 2018 8:13 am

Ahmad,

Sorry, I am a bit confused at this time. Is that what Trend said? Did you added all that information to the support call?

Thanks
Mike
Mike Resseler
Veeam Software
 
Posts: 4225
Liked: 471 times
Joined: Fri Feb 08, 2013 3:08 pm
Location: Belgium, the land of the fries, the beer, the chocolate and the diamonds...
Full Name: Mike Resseler

Re: VMs Backup fails due to Trend Micro, Case ID# 02472655

Veeam Logoby stack111 » Thu Feb 01, 2018 9:22 am

Yes, this is what Trend Micro partner told our customer.

I added already all these information to the support call.
stack111
Enthusiast
 
Posts: 27
Liked: 1 time
Joined: Thu Jan 18, 2018 8:15 am
Full Name: Ahmad Jayyusi

Re: VMs Backup fails due to Trend Micro, Case ID# 02472655

Veeam Logoby Mike Resseler » Thu Feb 01, 2018 9:23 am

OK. Cool. Thanks
Mike Resseler
Veeam Software
 
Posts: 4225
Liked: 471 times
Joined: Fri Feb 08, 2013 3:08 pm
Location: Belgium, the land of the fries, the beer, the chocolate and the diamonds...
Full Name: Mike Resseler

Re: VMs Backup fails due to Trend Micro, Case ID# 02472655

Veeam Logoby stack111 » Fri Feb 02, 2018 4:24 pm

This is last message from Veeam support:

[b]From the case notes backup does not work for vms with Trendmicro AV installed. Job fails with following message
"Failed to create snapshot (Microsoft CSV Shadow Copy Provider) (mode: Hyper-V child partition snapshot) Details: Writer 'Microsoft Hyper-V VSS Writer' is failed at 'VSS_WS_FAILED_AT_PREPARE_SNAPSHOT'.

According to the logs, veeam component is not uploaded/installed on the guest machine, so AV cannot block any communication from veeam server to guest machine, because connect is not set up.

In the event logs of the guest vm I see
Volume Shadow Copy Service error: Error calling a routine on a Shadow Copy Provider {74600e39-7dc5-4567-a03b-f091d6c7b092}. Routine details PostFinalCommitSnapshots({92203ad0-6e94-424e-9372-58e05258aa30}, 3) [hr = 0x80070005, Access is denied.

When the backup is taken 2 checkpoint files are created on the host, then one of the checkpoints is mounted to the guest vm. AD service opens AD Database on this new volume to finish VSS operation.
It seems that AV blocks the access to this disk.

Usually if the issue is isolated to AV software and there are no veeam components installed on the system where AV prevents the backup we recommend to open a case with AV vendor so that it can help to tune the Antivirus.

I can offer to setup a remote session to help you with collecting additional details before addressing the issue to the AV vendor.
stack111
Enthusiast
 
Posts: 27
Liked: 1 time
Joined: Thu Jan 18, 2018 8:15 am
Full Name: Ahmad Jayyusi

Re: VMs Backup fails due to Trend Micro, Case ID# 02472655

Veeam Logoby stack111 » Fri Feb 02, 2018 4:36 pm

I asked the support to collect needed information to be shared with Trend Micro vendor in order to tune the Antivirus.

In fact, Trend Micro vendor is waiting for us to provide needed exclusions in order for Veeam backup to work
stack111
Enthusiast
 
Posts: 27
Liked: 1 time
Joined: Thu Jan 18, 2018 8:15 am
Full Name: Ahmad Jayyusi

Re: VMs Backup fails due to Trend Micro, Case ID# 02472655

Veeam Logoby Mike Resseler » Mon Feb 05, 2018 6:32 am

Ahmad,

Could you look into event viewer (both host and guest) and look for VSS errors? Second. I just see AD. So we are talking about a domain controller? Any change this VM also runs DHCP/ DNS?

And last, now is the time to ask what OS the host and what OS the guest is :-)
Mike Resseler
Veeam Software
 
Posts: 4225
Liked: 471 times
Joined: Fri Feb 08, 2013 3:08 pm
Location: Belgium, the land of the fries, the beer, the chocolate and the diamonds...
Full Name: Mike Resseler

Next

Return to Microsoft Hyper-V



Who is online

Users browsing this forum: No registered users and 7 guests