Trend Micro DS and Veeam Backup issues

[soylent]

I just installed Trend Micro Deep Security (agentless for VMWare NSX). The following scheduled backup window killed many of the VM guest networks. The VM's were still running and had no errors, but they completely lost network access. This happened across all the hosts in the cluster, with different OS's, and different dVS port groups. So far only rebooting the host resolves the issues.

I have the Veeam server and proxy server excluded from Deep Security. What else can be causing this issue?

[Mike Resseler]

Max,

I'm afraid I don't understand the entire scenario. So the moment you start a backup, the guest VM networks are gone? Is it Trend that blocks the access? Anything that can seen in the logs over there? So maybe we can identify what is causing this

Thanks
M/

[soylent]

Yes the guest VMs simply could not communicate of the network. They still had their VMXNET adapters correctly configured, but they were unable to ping anything at all. They were effectively offline.

I think I may have found the issue, and I don't remember seeing it in any documentation on either Veeam or Trend sites. Deep Security has an "agentless" antimalware feature if you are running VMWare NSX. Basically it creates ESX Agents on each host ("guest introspection" and a Trend agent), instead of an agent on each guest. These ESX agents get placed in a resource pool called ESX Agents, and are small Linux-based appliance VMs. Trend's documention explicitly states that these agents must never get vMotioned to another host.

Veeam is set up to automatically back up new VMs. The network failures happened immediately when Veeam started backing up one of the ESX agents. I don't know if it's the snapshot or something else in the process.

After this happened two nights in a row, I deleted the ESX agents and disable all of it. At this point I don't know if simply excluding these agents from Veeam is the solution.

One possible theory I have is when Veeam attempts to mount the snapshot of the agent when it's on a different host. For example Veeam server one Host A attempts to grab the agent on Host B, causes some failure because these agents are designed to never leave either host. That would explain why it seemed to happen on random VMs.
Vcenter/NSX prevents manual vmotion, even when placing the host in maintanence mode it just shuts them down instead.

Anyway the result is catastrophic. My fault for just throwing Deep Security in production, but you may want to include this in your documentation or update to prevent Veeam from touching NSX-created agents.

[Mike Resseler]

Max,

This is good information. Please keep us up-to-date with more information. It seems indeed that you will need to exclude those agents. I am not aware of how those NSX-created agents look like and what type of VM's they are, and the potential problem is that we (as Veeam) do not know that it is such a VM. For us a VM is a VM...

Again, please keep us further informed. Then we might even be able to talk further to Trend and potentially come up with a nicer solution

Thanks
Mike

[soylent]

It's definitely a feature of NSX. I think Trend micro and Symantec are the only solutions with "agentless" support for VMware NSX. And it's actually NSX that creates and controls these agents. By that I mean you have to install them using the NSX management tools in vcenter.

You really should get NSX in your test environment. Then you can just install the built in Guest Introspection service (http://pubs.vmware.com/NSX-61/index.jsp ... 43FEA.html) and you can see what they look like. Trend's agent is installed the same way.

If you want I can send you logs or anything that can help.

[soylent]

I will be attempting to deploy Deep Security again later this week hopefully, in a much more controlled manner

[Dima P.]

Hi Max,

Can you run a test: create a snapshot for one of the affected VMs via VMWare native snapshot manager and then remove it? Will it act normally or you experience the same behavior as during backup?

[Andreas Neufert]

I found some statements online that state, that in some situations agents are needed for this.
http://www.vmbaggum.nl/2016/07/trend-mi ... 2-3-issue/

As Dima said above, most likely the TrendMicro Appliance and network stack will break when a VM Snapshot is created.

Trend Micro uses Veeam and should be able to figure this out immediately:
https://www.veeam.com/success-stories/t ... iness.html

If you have one please add a TrendMicro or Veeam support case here.

Maybe you can match some of the logfiles of you application and figure out when exactly this network outage happened. If this matches with vCenter SnapShot or SnapShot delete process of the TrendMicro VM in the vcenter log then excluding this ressource pool will be the a good option and we can consider this as default job setting.

[dellock6]

I've seen this thread only today as I was away, I deployed DS some times in the past, it was the version before NSX but the basic principles are the same, as far as I remember, you should NOT touch those agent VMs at all with any operations involving their IO, like vmotion or snapshots. As they are all deployed in a specific resource pool, you can simply exclude the entire pool from any backup job. Also because it's pretty useless to backup those VMs, they are just expendable agents controlled by the central console.

[deevie]

Indeed. You should never do any operations on these special machines. All your protected VM's IO (storage and network) passes these special VM's. When you take a snapshot of these VM's, you're actually freezing all IO and you definately don't want to do this.

[soylent]

Just a quick update. I opened a case with Trend Micro (00312690), but they are basically saying the same thing- We don't know anything about Veeam, and snapshots *probably* shouldn't be taken.

I redeployed the agents again, and made sure to exclude them from Veeam. I did notice something new after deploying the ESX Agents the second time. I deployed the Guest Introspection and Trend Micro agents to my cluster of 4 hosts. However, I found that one out of the four has a snapshot that was created during its deployment (one Guest Introspection and one Trend snapshot). The snapshot is called "eam_snapshot", with a description "Snapshot used to create new agent VMs". I *think* the purpose of it is for deploying additional agents, but I'm not sure why it only keeps a snapshot of a single agent, and I don't know if it automatically deletes it over time. It's been there for 3 days now.

Regarding Veeam Backup, doesn't it delete all snapshots when it processes a guest VM, whether it was created by Veeam or not?

I hope Veeam will add some NSX awareness. At least consider an automatic, default rule to exclude ESX agents or anything inside the "ESX Agent" resource pool.

[jmmarton]

No, VBR doesn't delete all snapshots from a VM, only the snapshot that is created by the backup job.

Joe

[SolutionsPT]

we run DS10 here, we don't backup any of the Trend created VM's. we have deleted the EAM snapshot and this has had no ill effect on DS at all.