-
- Enthusiast
- Posts: 61
- Liked: 7 times
- Joined: Aug 01, 2012 8:33 pm
- Full Name: Max
- Location: Fort Lauderdale, Florida
- Contact:
Trend Micro DS and Veeam Backup issues
I just installed Trend Micro Deep Security (agentless for VMWare NSX). The following scheduled backup window killed many of the VM guest networks. The VM's were still running and had no errors, but they completely lost network access. This happened across all the hosts in the cluster, with different OS's, and different dVS port groups. So far only rebooting the host resolves the issues.
I have the Veeam server and proxy server excluded from Deep Security. What else can be causing this issue?
I have the Veeam server and proxy server excluded from Deep Security. What else can be causing this issue?
-
- Product Manager
- Posts: 8191
- Liked: 1322 times
- Joined: Feb 08, 2013 3:08 pm
- Full Name: Mike Resseler
- Location: Belgium
- Contact:
Re: Trend Micro DS and Veeam Backup issues
Max,
I'm afraid I don't understand the entire scenario. So the moment you start a backup, the guest VM networks are gone? Is it Trend that blocks the access? Anything that can seen in the logs over there? So maybe we can identify what is causing this
Thanks
M/
I'm afraid I don't understand the entire scenario. So the moment you start a backup, the guest VM networks are gone? Is it Trend that blocks the access? Anything that can seen in the logs over there? So maybe we can identify what is causing this
Thanks
M/
-
- Enthusiast
- Posts: 61
- Liked: 7 times
- Joined: Aug 01, 2012 8:33 pm
- Full Name: Max
- Location: Fort Lauderdale, Florida
- Contact:
Re: Trend Micro DS and Veeam Backup issues
Yes the guest VMs simply could not communicate of the network. They still had their VMXNET adapters correctly configured, but they were unable to ping anything at all. They were effectively offline.
I think I may have found the issue, and I don't remember seeing it in any documentation on either Veeam or Trend sites. Deep Security has an "agentless" antimalware feature if you are running VMWare NSX. Basically it creates ESX Agents on each host ("guest introspection" and a Trend agent), instead of an agent on each guest. These ESX agents get placed in a resource pool called ESX Agents, and are small Linux-based appliance VMs. Trend's documention explicitly states that these agents must never get vMotioned to another host.
Veeam is set up to automatically back up new VMs. The network failures happened immediately when Veeam started backing up one of the ESX agents. I don't know if it's the snapshot or something else in the process.
After this happened two nights in a row, I deleted the ESX agents and disable all of it. At this point I don't know if simply excluding these agents from Veeam is the solution.
One possible theory I have is when Veeam attempts to mount the snapshot of the agent when it's on a different host. For example Veeam server one Host A attempts to grab the agent on Host B, causes some failure because these agents are designed to never leave either host. That would explain why it seemed to happen on random VMs.
Vcenter/NSX prevents manual vmotion, even when placing the host in maintanence mode it just shuts them down instead.
Anyway the result is catastrophic. My fault for just throwing Deep Security in production, but you may want to include this in your documentation or update to prevent Veeam from touching NSX-created agents.
I think I may have found the issue, and I don't remember seeing it in any documentation on either Veeam or Trend sites. Deep Security has an "agentless" antimalware feature if you are running VMWare NSX. Basically it creates ESX Agents on each host ("guest introspection" and a Trend agent), instead of an agent on each guest. These ESX agents get placed in a resource pool called ESX Agents, and are small Linux-based appliance VMs. Trend's documention explicitly states that these agents must never get vMotioned to another host.
Veeam is set up to automatically back up new VMs. The network failures happened immediately when Veeam started backing up one of the ESX agents. I don't know if it's the snapshot or something else in the process.
After this happened two nights in a row, I deleted the ESX agents and disable all of it. At this point I don't know if simply excluding these agents from Veeam is the solution.
One possible theory I have is when Veeam attempts to mount the snapshot of the agent when it's on a different host. For example Veeam server one Host A attempts to grab the agent on Host B, causes some failure because these agents are designed to never leave either host. That would explain why it seemed to happen on random VMs.
Vcenter/NSX prevents manual vmotion, even when placing the host in maintanence mode it just shuts them down instead.
Anyway the result is catastrophic. My fault for just throwing Deep Security in production, but you may want to include this in your documentation or update to prevent Veeam from touching NSX-created agents.
-
- Product Manager
- Posts: 8191
- Liked: 1322 times
- Joined: Feb 08, 2013 3:08 pm
- Full Name: Mike Resseler
- Location: Belgium
- Contact:
Re: Trend Micro DS and Veeam Backup issues
Max,
This is good information. Please keep us up-to-date with more information. It seems indeed that you will need to exclude those agents. I am not aware of how those NSX-created agents look like and what type of VM's they are, and the potential problem is that we (as Veeam) do not know that it is such a VM. For us a VM is a VM...
Again, please keep us further informed. Then we might even be able to talk further to Trend and potentially come up with a nicer solution
Thanks
Mike
This is good information. Please keep us up-to-date with more information. It seems indeed that you will need to exclude those agents. I am not aware of how those NSX-created agents look like and what type of VM's they are, and the potential problem is that we (as Veeam) do not know that it is such a VM. For us a VM is a VM...
Again, please keep us further informed. Then we might even be able to talk further to Trend and potentially come up with a nicer solution
Thanks
Mike
-
- Enthusiast
- Posts: 61
- Liked: 7 times
- Joined: Aug 01, 2012 8:33 pm
- Full Name: Max
- Location: Fort Lauderdale, Florida
- Contact:
Re: Trend Micro DS and Veeam Backup issues
It's definitely a feature of NSX. I think Trend micro and Symantec are the only solutions with "agentless" support for VMware NSX. And it's actually NSX that creates and controls these agents. By that I mean you have to install them using the NSX management tools in vcenter.
You really should get NSX in your test environment. Then you can just install the built in Guest Introspection service (http://pubs.vmware.com/NSX-61/index.jsp ... 43FEA.html) and you can see what they look like. Trend's agent is installed the same way.
If you want I can send you logs or anything that can help.
You really should get NSX in your test environment. Then you can just install the built in Guest Introspection service (http://pubs.vmware.com/NSX-61/index.jsp ... 43FEA.html) and you can see what they look like. Trend's agent is installed the same way.
If you want I can send you logs or anything that can help.
-
- Enthusiast
- Posts: 61
- Liked: 7 times
- Joined: Aug 01, 2012 8:33 pm
- Full Name: Max
- Location: Fort Lauderdale, Florida
- Contact:
Re: Trend Micro DS and Veeam Backup issues
I will be attempting to deploy Deep Security again later this week hopefully, in a much more controlled manner
-
- Product Manager
- Posts: 14726
- Liked: 1707 times
- Joined: Feb 04, 2013 2:07 pm
- Full Name: Dmitry Popov
- Location: Prague
- Contact:
Re: Trend Micro DS and Veeam Backup issues
Hi Max,
Can you run a test: create a snapshot for one of the affected VMs via VMWare native snapshot manager and then remove it? Will it act normally or you experience the same behavior as during backup?
Can you run a test: create a snapshot for one of the affected VMs via VMWare native snapshot manager and then remove it? Will it act normally or you experience the same behavior as during backup?
-
- VP, Product Management
- Posts: 7081
- Liked: 1511 times
- Joined: May 04, 2011 8:36 am
- Full Name: Andreas Neufert
- Location: Germany
- Contact:
Re: Trend Micro DS and Veeam Backup issues
I found some statements online that state, that in some situations agents are needed for this.
http://www.vmbaggum.nl/2016/07/trend-mi ... 2-3-issue/
As Dima said above, most likely the TrendMicro Appliance and network stack will break when a VM Snapshot is created.
Trend Micro uses Veeam and should be able to figure this out immediately:
https://www.veeam.com/success-stories/t ... iness.html
If you have one please add a TrendMicro or Veeam support case here.
Maybe you can match some of the logfiles of you application and figure out when exactly this network outage happened. If this matches with vCenter SnapShot or SnapShot delete process of the TrendMicro VM in the vcenter log then excluding this ressource pool will be the a good option and we can consider this as default job setting.
http://www.vmbaggum.nl/2016/07/trend-mi ... 2-3-issue/
As Dima said above, most likely the TrendMicro Appliance and network stack will break when a VM Snapshot is created.
Trend Micro uses Veeam and should be able to figure this out immediately:
https://www.veeam.com/success-stories/t ... iness.html
If you have one please add a TrendMicro or Veeam support case here.
Maybe you can match some of the logfiles of you application and figure out when exactly this network outage happened. If this matches with vCenter SnapShot or SnapShot delete process of the TrendMicro VM in the vcenter log then excluding this ressource pool will be the a good option and we can consider this as default job setting.
-
- VeeaMVP
- Posts: 6166
- Liked: 1971 times
- Joined: Jul 26, 2009 3:39 pm
- Full Name: Luca Dell'Oca
- Location: Varese, Italy
- Contact:
Re: Trend Micro DS and Veeam Backup issues
I've seen this thread only today as I was away, I deployed DS some times in the past, it was the version before NSX but the basic principles are the same, as far as I remember, you should NOT touch those agent VMs at all with any operations involving their IO, like vmotion or snapshots. As they are all deployed in a specific resource pool, you can simply exclude the entire pool from any backup job. Also because it's pretty useless to backup those VMs, they are just expendable agents controlled by the central console.
Luca Dell'Oca
Principal EMEA Cloud Architect @ Veeam Software
@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
Principal EMEA Cloud Architect @ Veeam Software
@dellock6
https://www.virtualtothecore.com/
vExpert 2011 -> 2022
Veeam VMCE #1
-
- Influencer
- Posts: 11
- Liked: 1 time
- Joined: Jan 22, 2012 8:58 pm
- Contact:
Re: Trend Micro DS and Veeam Backup issues
Indeed. You should never do any operations on these special machines. All your protected VM's IO (storage and network) passes these special VM's. When you take a snapshot of these VM's, you're actually freezing all IO and you definately don't want to do this.
-
- Enthusiast
- Posts: 61
- Liked: 7 times
- Joined: Aug 01, 2012 8:33 pm
- Full Name: Max
- Location: Fort Lauderdale, Florida
- Contact:
Re: Trend Micro DS and Veeam Backup issues
Just a quick update. I opened a case with Trend Micro (00312690), but they are basically saying the same thing- We don't know anything about Veeam, and snapshots *probably* shouldn't be taken.
I redeployed the agents again, and made sure to exclude them from Veeam. I did notice something new after deploying the ESX Agents the second time. I deployed the Guest Introspection and Trend Micro agents to my cluster of 4 hosts. However, I found that one out of the four has a snapshot that was created during its deployment (one Guest Introspection and one Trend snapshot). The snapshot is called "eam_snapshot", with a description "Snapshot used to create new agent VMs". I *think* the purpose of it is for deploying additional agents, but I'm not sure why it only keeps a snapshot of a single agent, and I don't know if it automatically deletes it over time. It's been there for 3 days now.
Regarding Veeam Backup, doesn't it delete all snapshots when it processes a guest VM, whether it was created by Veeam or not?
I hope Veeam will add some NSX awareness. At least consider an automatic, default rule to exclude ESX agents or anything inside the "ESX Agent" resource pool.
I redeployed the agents again, and made sure to exclude them from Veeam. I did notice something new after deploying the ESX Agents the second time. I deployed the Guest Introspection and Trend Micro agents to my cluster of 4 hosts. However, I found that one out of the four has a snapshot that was created during its deployment (one Guest Introspection and one Trend snapshot). The snapshot is called "eam_snapshot", with a description "Snapshot used to create new agent VMs". I *think* the purpose of it is for deploying additional agents, but I'm not sure why it only keeps a snapshot of a single agent, and I don't know if it automatically deletes it over time. It's been there for 3 days now.
Regarding Veeam Backup, doesn't it delete all snapshots when it processes a guest VM, whether it was created by Veeam or not?
I hope Veeam will add some NSX awareness. At least consider an automatic, default rule to exclude ESX agents or anything inside the "ESX Agent" resource pool.
-
- Veeam Software
- Posts: 2097
- Liked: 310 times
- Joined: Nov 17, 2015 2:38 am
- Full Name: Joe Marton
- Location: Chicago, IL
- Contact:
Re: Trend Micro DS and Veeam Backup issues
No, VBR doesn't delete all snapshots from a VM, only the snapshot that is created by the backup job.
Joe
Joe
-
- Novice
- Posts: 3
- Liked: never
- Joined: Sep 19, 2017 11:43 am
- Full Name: Pete Gibson
- Contact:
Re: Trend Micro DS and Veeam Backup issues
we run DS10 here, we don't backup any of the Trend created VM's. we have deleted the EAM snapshot and this has had no ill effect on DS at all.
Who is online
Users browsing this forum: No registered users and 33 guests