vSphere 5 compatibility patch problems

[arthurp]

Within 24 hours from patch installation we have got a situation when Veeam cannot freeze OS (stuck at 0%) or cannot remove snapshots and complete the job (stuck at 99%). Occasionally, this happens both on backup and replication. Jobs do not react to "Stop" command and the only way to continue is to restart the backup server itself (killing processes will likely work too).
We are using Veeam in near CDP mode. At times this involves several simultaneous jobs. Previously (before patch), we had one VeeamAgent.exe process per job. Now we have 2 such processes per job with quite high CPU utilization, bringing CPU usage to 100% with only two simultaneous jobs.
Just sharing our experience - did anyone else run into these problems?

Thank you

Arthur

[Gostev]

This issue is unlikely to be related to patch (we have been monitoring the situation closely, and so far there is zero issues reported from about 500 customers who had received the patch). Well, there was one where customer also blamed patch, but upon investigation it appeared to be unrelated issue.

By the way, part of the reasons why we only distribute the patch through support - we want to track all customers using it, to see if any issues they report might be connected.

Regaring the agents, hotfix is not exactly something where we would dare to change the architecture there have always been 2 agents per job running on backup server (source and target agents), unless of course backup or replication target is agent-enabled (Linux or fat ESX), in which case the target agent is started remotely. This did not change since version 1, and the hotfix does not change this architecture either.

I would recommend opening a support case for deeper troubleshooting.

[UPDATE] Important note based on further discussion:

If you changed your target from "fat" ESX to ESXi as a part of your vSphere 5 upgrade, you can expect major changes in replication behavior in all cases. This is unrelated to patch, and would be the case even if you simply went from ESX 4.1 to ESXi 4.1 without applying the patch. With Veeam v5, replication to ESXi is not as efficient comparing to ESX (there are multiple topics discussing this on the forums). For v5, we always recommend using "fat" ESX as your replication target. Search for esxi replication for more detailed information. Efficient replication to ESXi targets is a feature of v6 of our product.

[arthurp]

Thank you, Anton. The issue of two agents is now clear as previously we operated fat ESX only and with vSphere 5 it is bound to change. I suspect that the problem may have something to do with capacity planning as our Veeam VM now gets overly too CPU busy. I will try to adjust schedules to spread the load. One particular thing about our environment is that virtual machines from ESX 4.1 hosts are replicated to ESXi 5.0 (we stopped migrating to 5.0 once discovered compatibility issues and never rolled back). Did you test it against such a mixed environment? Anyway, I will dig deeper and will likely open support case tomorrow.

[heldchen]

Hi

I also expected some errors. Replication freezes many times since the patch is installed, and the whole Server which is running Veeam is freezing. This happens since i installed the patch on Monday.
Vcenter is the last Version of Vcenter 5 and my hosts are 4.1.
And in a job which is replicating 5 Mashines, 2 or more cannot be replicated because there is is no FIB.
This error came up after installing the patch. Before it, i had a functional replication.

[Vitaliy S.]

Arthur, we found no issues while testing replication jobs in a 4.1/5.0 mixed environments.

Marco, you definitely want to contact our support team to get quicker assistance with this issue. Also I would appreciate, if you could update this topic once you have your issue resolved.

[Gostev]

Arthur and all,

If you changed your target from "fat" ESX to ESXi as a part of your vSphere 5 upgrade, you can expect major changes in replication behavior in all cases. This is unrelated to patch, and would be the case even if you simply went from ESX 4.1 to ESXi 4.1 without applying the patch. With Veeam v5, replication to ESXi is not as efficient comparing to ESX (there are multiple topics discussing this on the forums). For Veeam v5, we always recommend using "fat" ESX as your replication target. Search for esxi replication for more detailed information. Efficient replication to ESXi targets is a feature of v6 of our product.

Marco, there is no FIB error also does not have to deal with the patch, in fact it was already reported on these forums once earlier this year. Moreover, since you are not using version 5 hosts, the patch will not change how your replication works today at all. Modified code is only used if ESXi 5 is detected.

[jcmachadouga]

We started having issues the moment we applied the patch. Now, every job starts 2 VeeamAgent.exe processes and 1 Veeam.Backup.Manager.exe process.
Running 2 simple jobs brings the CPU to 100% all the time, and jobs that used to take 1-2 hours now takes 12-18 hours. The veeam backup server is a physical dual quad-core with 56G or RAM.. Never had an issue before related to performance cpu/memory.

So, YES, there is a problem with the patch.

[Gostev]

Juan, read the above topic carefully regarding 2 agent processes per job (this is normal), as well as overall performance impact of jobs when switching to ESXi. Also, please do not copy the same post into multiple topics - choose one topic that suits best.

Regarding CPU load and possible immediate workaround, please read this topic carefully.

[ThomasMc]

if your needing more people to test it then I could do it, I've got reams of historic performance data;

Virtual Veeam Server 4vCPU, 4GB RAM
ESXi 4.1 u1 on all hosts