after upgrade to v10 during backup its dropping pings

[33minutesinheaven]

Hey guys,

i upgraded our veeam instance to version 10 last week. Ever since then, every night during the backup windows, i will get alerts about servers not being accessible. The times on the alerts correspond to when the servers are being backed up. So i am pretty certain that this has been caused by the veeam update.

Anyone else seeing this? Between veeam and the cluster is 10GBE network. And version 9 didnt do this.
Just hoping its some kind of bug or configuration change is needed. Let me know please.

can we still not attach images? sigh...

[foggy]

Do you mean servers being backed up are not accessible? Have you opened a case already?

[33minutesinheaven]

I am not sure if they are accesible or not as it happens in the middle of the night. The main symptom i am seeing is its driving my alert system (zenoss) crazy. So i wake up with like 50 messages about servers going down, but yet nothing has gone down, the uptimes are all fine. So i would conclude it was a momentary loss of connectivity.

My IPM (UPS monitor) also told me last night that a cluster host went down, but it has 233 days of uptime. So its not just zenoss that is affected. Perhaps when vcenter is making the snapshots?

i can try contacting support, but i was hoping that someone on the forums had a similar experience first. I found one post from 2014 but it doesnt really apply and had no resolution. Im sure it started this weekend as i did the v10 update on friday.

[foggy]

Short VM stun is possible upon snapshot commit, for example, however, the fact that it started after the upgrade is suspicious. Haven't you noticed any other changes in the job processing - duration, processing rates, transport mode change?... Or maybe in VI itself? Anyway, I'd recommend taking a closer look at what happens with the help of our engineers.

[33minutesinheaven]

ok maybe i will create a ticket then.

i did notice that the backups seem to run MUCH quicker now. Which is awesome. Another thing i noticed is that guest credentials for primarily linux VMs now make a big red X when they dont have credentials and make backups "fail". Previously, this was only a warning (yellow icon and backups didnt fail or retry because of that). I was able to put in credentials or exclude where necessary. Only two changes i have noticed so far.

[foggy]

I'd appreciate if you share the case ID once you open it - thus we would be able to track it down for resolution in case someone sees similar behavior in the future. Thanks!

[33minutesinheaven]

the case is 04492602 but not at all helpful so far. He just blames my environment. sent me this document https://www.veeam.com/kb1681
but none of that really applies to me. I have made a VM snapshot and will delete it tomorrow morning, which i think is what is meant by this isolation test. The fact is this did not occur till i upgraded. So its for sure caused by v10. Nothing else changed. My comments on the various bullet points in the article and why I dont think that will be the problem is below.

- no vm snapshots
- happens to all my VMs, so not an orphaned snapshot on a particular vm.
- moving to a different datastore is not possible. It’s a SAN. Everything was fine before v10.
- there are no CPU reservations.
- vsphere is up to date. Vmware tools is up to date. San firmware is up to date. None of that has changed.
- tried to move all VMs to a new cluster with more power, this has not solved the problem.
- this problem occurs from my first backup at 9pm all the way till 7am. So nothing to do with time of day.
- don’t really want to change anything about the working dirs..
- this does not seem to apply. I don’t have any windows 2008 or older servers

i updated vmware tools on all the machines over the weekend and even moved them to a more powerful cluster i have. No change. Veeam is not maxing out any resources when backing up except 50GB of ram. And it pretty much uses as much ram as i give to it. And the jobs all complete fine. But for instance when my web server loses connectivity in the evenings, sometime transactions are lost and create errors for customers. not cool.

Im guessing downgrading back to version 9.5 is not an easy process... I would rather get it resolved and move forward, but not looking at any logs and just sending me to a KB... well thats useless. Like everything, i probably have to fix this on my own.. sigh.

[33minutesinheaven]

Ok looks like i fixed it. I did three things at the same time, so i am not sure which operation fixed the problem here.

First, i had 6 concurrent tasks on my backup repository. I reduced that down to 4.
Second, my veeam instance was running on a separate server. That server was running vmware 6.5 becuase the hardware is too old to take 6.7. So i updated to the latest patch level of 6.5.
Thirdly, i updated the nimble plugin for our SAN to the latest version on the separate veeam server. I had updated the storage plugin on all the other hosts a month or so ago, but never did the separate veeam host.

I am not sure what did it. My backups are running slower now wiht the 4 concurrent tasks, so i have now changed that back to 6 to see if that creates the problem again. Will post back with the results from that test, but it hasnt dropped pings the last two nights at all.

[foggy]

Thanks for the update! That kind of proves the fact that the issue is environmental. Will be interested in the outcome of your further findings though.

[33minutesinheaven]

well not necessarily as veeam likely changed the way they are doing something between releases. Its possible to have multiple factors that contribute to a problem. Just because i was able to work around it, does not mean that there still isn't a problem with the product in some way. It was obviously working fine with the old plugin and old esx version when using B&R 9.5

And the isolation test, did not have the same issue. I consolidated the test VM this morning and no pings were lost.

[foggy]

Given the customer base size, such a change would have definitely shown up much earlier. This makes me think that even if the issue is caused by a change in Veeam B&R, it is most likely triggered by some peculiarities of the environment. I recommend you escalating the case to a higher tier for deeper investigation so it could be tracked down to the actual cause.

[33minutesinheaven]

changing the amount of concurrent tasks on the backup repository had no effect. So this was solved by upgrading vmware to the latest 6.5 version and also upgrading to version 7 of the nimble plugin.

I am done working for these people at the end of the week (tomorrow) so i wont be able to check into anything further sadly on this. Hope this whole discussion helps someone out in the future. The problem is resolved for me now.

[foggy]

Thanks for the feedback anyway. One last thing I'd like to clarify though, if possible - what Nimble plug-in are you talking about?

[33minutesinheaven]

i probably meant the plugin that nimble gives to interface vmware on the SAN. the vmware plugin provided by nimble to interface with their equipment.