ESX 5.5 Update 1

[Dacuy]

We upgraded our vmware hosts to ESX 5.5 Update 1 (from 5.5) over the weekend. Since then, some of our backups notably one with high I/O VM's have had uncommited snapshots at the onset, and then for the most part, backup times have increased from 40 mins - 1 hour to 3- 7 hours mostly. ALSO yesterday and today, my mid-morning backups are just hanging with 0% completed - these are supposed to go to 2 repositories, one a Dell Poweredge server and another a Dell Dedup applicance. We use Dell Equallogics for pimary storage, Dell for Host servers. I have logged a support case with Veeam support. Has anyone else who who has updated to ESX 5.5 Update 1 had issues similar to these?

[foggy]

Dennis, what are the bottleneck stats for the jobs that have degraded performance and what transport mode is being used? Could you please also double check that CBT is used? And what is your support case ID number?

[Dacuy]

case # 00629031. Last night and today, all problem symptoms have disappeared despite the fact that we havent done anything on Veeam, VCentre, SAN...absolutelty nothing. The high I/O vm's in that one job (SAP Production VM's) which backs up every 3 hours have returned to its normal < 3 hour duration. CBT is used, and historically Target was the usual bottleneck even prior to these issues starting probably because its a reverse incremental.
This particular Job would have its load shared like this: Source 75% Proxy 5% Network 35% Target 50%. It appears the load is mostly shared between Source and Target with the Source bearing the brunt. Occasionally Source and Target would be 50-50. The values for Proxy and Network hardly fluctuate.

[veremin]

I'm wondering whether there had been any tasks taking place inside the said VMs before backup job ran. Those tasks, such as antivirus, defragmentation activities or some other maintenance tasks might have affected number of changed blocks and led to backup jobs taking longer and VMs becoming unresponsive during snapshot commit operation (or even "leftover" snapshots). Thanks.

[ggrice]

Not sure if it's related, but I also had problems when upgrading from esx 5.1 to 5.5. My Veeam backup jobs would not run correctly and they would not detect the VM sizes correctly. But the issues were not limited to Veeam alone, I also saw latency spikes and network counter problems within esx itself.

After some hunting around, I found that the problem was caused by an incompatible NIC driver in the ESX installer (downloaded from VMware's site). We are running Dell servers, I downloaded a Dell VMware 5.5 .iso, reloaded my hosts with that and hey presto we were back in business. Did you install ESX from a web download or a Dell image?

[edfops]

Not sure if it's related, but I also had problems when upgrading from esx 5.1 to 5.5. My Veeam backup jobs would not run correctly and they would not detect the VM sizes correctly. But the issues were not limited to Veeam alone, I also saw latency spikes and network counter problems within esx itself.

After some hunting around, I found that the problem was caused by an incompatible NIC driver in the ESX installer (downloaded from VMware's site). We are running Dell servers, I downloaded a Dell VMware 5.5 .iso, reloaded my hosts with that and hey presto we were back in business. Did you install ESX from a web download or a Dell image?

We also have Dell hosts and have not seen these issues with 5.5U1, but I will say we also updated any and all firmware alongside the upgrade.

[Dacuy]

Thanks for the responses. Strangely enough the symptoms have ceased without us doing any further changes, but I am still closely monitoring. We didn't have issues migrating from pre-5.5 to 5.5, and that was quite a while ago.

[ggrice]

Just to clarify, the problems I had were because we were using a 10gb NIC, the 1gb NIC's ran ok with the standard VMware .iso