Trend Micro DS and Veeam Backup issues

VMware specific discussions

Trend Micro DS and Veeam Backup issues

Veeam Logoby soylent » Sun Jan 01, 2017 3:51 pm

I just installed Trend Micro Deep Security (agentless for VMWare NSX). The following scheduled backup window killed many of the VM guest networks. The VM's were still running and had no errors, but they completely lost network access. This happened across all the hosts in the cluster, with different OS's, and different dVS port groups. So far only rebooting the host resolves the issues.

I have the Veeam server and proxy server excluded from Deep Security. What else can be causing this issue?
soylent
Enthusiast
 
Posts: 50
Liked: 4 times
Joined: Wed Aug 01, 2012 8:33 pm
Full Name: Max Strzelecki

Re: Trend Micro DS and Veeam Backup issues

Veeam Logoby Mike Resseler » Mon Jan 02, 2017 7:01 am

Max,

I'm afraid I don't understand the entire scenario. So the moment you start a backup, the guest VM networks are gone? Is it Trend that blocks the access? Anything that can seen in the logs over there? So maybe we can identify what is causing this

Thanks
M/
Mike Resseler
Veeam Software
 
Posts: 3165
Liked: 362 times
Joined: Fri Feb 08, 2013 3:08 pm
Location: Belgium, the land of the fries, the beer, the chocolate and the diamonds...
Full Name: Mike Resseler

Re: Trend Micro DS and Veeam Backup issues

Veeam Logoby soylent » Mon Jan 02, 2017 3:58 pm

Yes the guest VMs simply could not communicate of the network. They still had their VMXNET adapters correctly configured, but they were unable to ping anything at all. They were effectively offline.

I think I may have found the issue, and I don't remember seeing it in any documentation on either Veeam or Trend sites. Deep Security has an "agentless" antimalware feature if you are running VMWare NSX. Basically it creates ESX Agents on each host ("guest introspection" and a Trend agent), instead of an agent on each guest. These ESX agents get placed in a resource pool called ESX Agents, and are small Linux-based appliance VMs. Trend's documention explicitly states that these agents must never get vMotioned to another host.

Veeam is set up to automatically back up new VMs. The network failures happened immediately when Veeam started backing up one of the ESX agents. I don't know if it's the snapshot or something else in the process.

After this happened two nights in a row, I deleted the ESX agents and disable all of it. At this point I don't know if simply excluding these agents from Veeam is the solution.

One possible theory I have is when Veeam attempts to mount the snapshot of the agent when it's on a different host. For example Veeam server one Host A attempts to grab the agent on Host B, causes some failure because these agents are designed to never leave either host. That would explain why it seemed to happen on random VMs.
Vcenter/NSX prevents manual vmotion, even when placing the host in maintanence mode it just shuts them down instead.

Anyway the result is catastrophic. My fault for just throwing Deep Security in production, but you may want to include this in your documentation or update to prevent Veeam from touching NSX-created agents.
soylent
Enthusiast
 
Posts: 50
Liked: 4 times
Joined: Wed Aug 01, 2012 8:33 pm
Full Name: Max Strzelecki

Re: Trend Micro DS and Veeam Backup issues

Veeam Logoby Mike Resseler » Mon Jan 02, 2017 4:02 pm

Max,

This is good information. Please keep us up-to-date with more information. It seems indeed that you will need to exclude those agents. I am not aware of how those NSX-created agents look like and what type of VM's they are, and the potential problem is that we (as Veeam) do not know that it is such a VM. For us a VM is a VM...

Again, please keep us further informed. Then we might even be able to talk further to Trend and potentially come up with a nicer solution

Thanks
Mike
Mike Resseler
Veeam Software
 
Posts: 3165
Liked: 362 times
Joined: Fri Feb 08, 2013 3:08 pm
Location: Belgium, the land of the fries, the beer, the chocolate and the diamonds...
Full Name: Mike Resseler

Re: Trend Micro DS and Veeam Backup issues

Veeam Logoby soylent » Mon Jan 02, 2017 4:13 pm

It's definitely a feature of NSX. I think Trend micro and Symantec are the only solutions with "agentless" support for VMware NSX. And it's actually NSX that creates and controls these agents. By that I mean you have to install them using the NSX management tools in vcenter.

You really should get NSX in your test environment. Then you can just install the built in Guest Introspection service (http://pubs.vmware.com/NSX-61/index.jsp ... 43FEA.html) and you can see what they look like. Trend's agent is installed the same way.

If you want I can send you logs or anything that can help.
soylent
Enthusiast
 
Posts: 50
Liked: 4 times
Joined: Wed Aug 01, 2012 8:33 pm
Full Name: Max Strzelecki

Re: Trend Micro DS and Veeam Backup issues

Veeam Logoby soylent » Mon Jan 02, 2017 4:16 pm

I will be attempting to deploy Deep Security again later this week hopefully, in a much more controlled manner ;)
soylent
Enthusiast
 
Posts: 50
Liked: 4 times
Joined: Wed Aug 01, 2012 8:33 pm
Full Name: Max Strzelecki

Re: Trend Micro DS and Veeam Backup issues

Veeam Logoby Dima P. » Mon Jan 02, 2017 4:43 pm

Hi Max,

Can you run a test: create a snapshot for one of the affected VMs via VMWare native snapshot manager and then remove it? Will it act normally or you experience the same behavior as during backup?
Dima P.
Veeam Software
 
Posts: 6261
Liked: 441 times
Joined: Mon Feb 04, 2013 2:07 pm
Location: SPb
Full Name: Dmitry Popov

Re: Trend Micro DS and Veeam Backup issues

Veeam Logoby Andreas Neufert » Mon Jan 02, 2017 11:57 pm

I found some statements online that state, that in some situations agents are needed for this.
http://www.vmbaggum.nl/2016/07/trend-mi ... 2-3-issue/

As Dima said above, most likely the TrendMicro Appliance and network stack will break when a VM Snapshot is created.

Trend Micro uses Veeam and should be able to figure this out immediately:
https://www.veeam.com/success-stories/t ... iness.html

If you have one please add a TrendMicro or Veeam support case here.

Maybe you can match some of the logfiles of you application and figure out when exactly this network outage happened. If this matches with vCenter SnapShot or SnapShot delete process of the TrendMicro VM in the vcenter log then excluding this ressource pool will be the a good option and we can consider this as default job setting.
Andreas Neufert
Veeam Software
 
Posts: 2201
Liked: 360 times
Joined: Wed May 04, 2011 8:36 am
Location: Germany
Full Name: @AndyandtheVMs Veeam PM

Re: Trend Micro DS and Veeam Backup issues

Veeam Logoby dellock6 » Mon Jan 09, 2017 9:09 pm

I've seen this thread only today as I was away, I deployed DS some times in the past, it was the version before NSX but the basic principles are the same, as far as I remember, you should NOT touch those agent VMs at all with any operations involving their IO, like vmotion or snapshots. As they are all deployed in a specific resource pool, you can simply exclude the entire pool from any backup job. Also because it's pretty useless to backup those VMs, they are just expendable agents controlled by the central console.
Luca Dell'Oca
EMEA Cloud Architect @ Veeam Software

@dellock6
http://www.virtualtothecore.com
vExpert 2011-2012-2013-2014-2015-2016
Veeam VMCE #1
dellock6
Veeam Software
 
Posts: 5055
Liked: 1335 times
Joined: Sun Jul 26, 2009 3:39 pm
Location: Varese, Italy
Full Name: Luca Dell'Oca

Re: Trend Micro DS and Veeam Backup issues

Veeam Logoby deevie » Mon Jan 09, 2017 9:46 pm

Indeed. You should never do any operations on these special machines. All your protected VM's IO (storage and network) passes these special VM's. When you take a snapshot of these VM's, you're actually freezing all IO and you definately don't want to do this.
deevie
Influencer
 
Posts: 10
Liked: never
Joined: Sun Jan 22, 2012 8:58 pm

Re: Trend Micro DS and Veeam Backup issues

Veeam Logoby soylent » Mon Jan 23, 2017 4:57 pm

Just a quick update. I opened a case with Trend Micro (00312690), but they are basically saying the same thing- We don't know anything about Veeam, and snapshots *probably* shouldn't be taken.

I redeployed the agents again, and made sure to exclude them from Veeam. I did notice something new after deploying the ESX Agents the second time. I deployed the Guest Introspection and Trend Micro agents to my cluster of 4 hosts. However, I found that one out of the four has a snapshot that was created during its deployment (one Guest Introspection and one Trend snapshot). The snapshot is called "eam_snapshot", with a description "Snapshot used to create new agent VMs". I *think* the purpose of it is for deploying additional agents, but I'm not sure why it only keeps a snapshot of a single agent, and I don't know if it automatically deletes it over time. It's been there for 3 days now.

Regarding Veeam Backup, doesn't it delete all snapshots when it processes a guest VM, whether it was created by Veeam or not?

I hope Veeam will add some NSX awareness. At least consider an automatic, default rule to exclude ESX agents or anything inside the "ESX Agent" resource pool.
soylent
Enthusiast
 
Posts: 50
Liked: 4 times
Joined: Wed Aug 01, 2012 8:33 pm
Full Name: Max Strzelecki

Re: Trend Micro DS and Veeam Backup issues

Veeam Logoby jmmarton » Mon Jan 23, 2017 5:31 pm

No, VBR doesn't delete all snapshots from a VM, only the snapshot that is created by the backup job.

Joe
jmmarton
Veeam Software
 
Posts: 821
Liked: 89 times
Joined: Tue Nov 17, 2015 2:38 am
Location: Chicago, IL
Full Name: Joe Marton


Return to VMware vSphere



Who is online

Users browsing this forum: No registered users and 16 guests