Wierd Veeam B&R server after Windows Updates / Reboot

[enoch]

Hi all,

Had a pretty solid B&R running Windows 2012r2 DataCenter Edition with current Veeam 9.5 U1 running.

All Windows patches is up-to-date from Windows Update feature in Windows. Last time I updated the server is months ago. In the window we updated the SQL 2014 SP2 to SP2 CU5.

After this on the B&R server it self I have had some "lag" in the interface and 3-4 times today got the message:

"The connection to backup server has been lost. Attempting to reconnect to your session"

"Connection attepmt: 1 of 10"

All services: console, SQL, B&R is on same server!

A lot of jobs I suddenly gets the error (both VMware, Hyper-V backup jobs and replication jobs):
"Full backup file merge failed error: ChannelError: ConnectionReset"

All Backup repositories is on local storage!

Tried restore from some of the jobs failling and they seems to Work fine!

I have a current open case at Veeam: 02140683 but if anyone have any fix/ideas I'll be glad to hear from you!

[Vitaliy S.]

Hi Peter,

Do you have any SQL Server related error messages in the Windows Event log?

Thanks!

[enoch]

Hi,

I can't find any SQL related errors in the System or Application Event Logs.

It's seems a reboot "fix" the problem, 4-6 hours and then problems again.

A stop of the Veeam Backup Service results in not able to start the service - a server reboot is needed!

[Vitaliy S.]

OK, then let's see what our support engineer says after reviewing the debug log files.

[enoch]

Yes, running the case with Veeam support now.

Hope they find the problem and a fix.

[JosvwHLT]

Last Friday I updated 2 Veeam B&R 9.0 servers to 9.5 u1. One of the 2 is experiencing the same problems. A server reset is the only option. In the systemlog i see evenitid 4227 - TCP/IP Network Connectivity (EVENT_TCPIP_TCP_TIME_WAIT_COLLISION). NIC disable/enable option is not available because server is almost non-responsive.

[enoch]

Hi all,

I've tried REALLY much right now.

I see there is a issue with the Microsoft patch: kb4015553 for a lot of people - Windows 2012r2 - the patch is from late April 2017.

Sadly I cannot remove the patch from the server without an error

Maybe if you have this installed you could try with uninstall?

[JosvwHLT]

Don't want to uninstall any patches right now before knowing what is really going on. We are running a tape backup procedure in which we mount a backup image (with VeeamFLR) and retrieve data from that mounted image and dump it to tape. During this procedure the server crashed already twice. I'm running the Veeam B&R server now without running any tape procedure. Want to wait for that outcome first before i change any thing on that server. We are also experiencing a failed service start of the "VeeamCatalogsvc" service, this service won't start anymore after we experienced the issues with the "unresponsive server" but don't know if it is related. In the application log i see the following "Service Cannot be started", part of the message is the following "Attribute "LastSyncTimeUTC" was not found under the tree".

[JosvwHLT]

Applied TCP/IP settings as described in URL vmware-vsphere-f24/veeam-flr-t32119-15.html This seems to have fixed my issue, server is running for a couple of days now without any issues. Server froze when running a tape procedure which copied a lot of small files to tape. That is running fine now too.

[enoch]

Hi all,

Afters days suppport with Microsoft who didn't have any luck removing the KB4015553 which could not be removed in Windows, I succeed removing the patch.

After the patch was removed my system have now run 24 hours without issues.

Before I could only have the system running about 8 hours before I was needed to do a reboot.

[kstrp]

Hi all,

Afters days suppport with Microsoft who didn't have any luck removing the KB4015553 which could not be removed in Windows, I succeed removing the patch.

After the patch was removed my system have now run 24 hours without issues.

Before I could only have the system running about 8 hours before I was needed to do a reboot.

Mind sharing how you removed it? Couldn't get that to happen on our own and microsoft couldn't figure out when we spoke to them briefly.

[enoch]

I would like to, but not 100% sure what did it

Please try this:
Dism /Online /Cleanup-Image /CheckHealth - check everything is OK

Dism /Online /Cleanup-Image /ScanHealth - check everything is OK

If these are not OK, then: DISM.exe /Online /Cleanup-image /Restorehealth and: sfc /scannow

Download the KB4015553 manually from Microsoft: http://www.catalog.update.microsoft.com ... =KB4015553

Then extract thethe KB4015553.msu package.

Then do a DISM.exe /Online /Add-Package /PackagePath:"path to KB4015553.cab file"

Then do a dism /remove-package with this update. Find the package ID with: Dism /Online /Get-Packages

After this I went in the Add/Remove program an the uninstall worked!

[kstrp]

Hmm - I've tried similar steps already - but I'll give it another go covering what you listed.

Thanks again.

[LittlePiGPiG]

I experienced exactly / very much similar symptoms of everyone above said and sadly the Veeam support had not yet come up with any conclusions.

Then, I tried removing latest "updates" (Except those M$ "security patches") and I get a normal V&R server back.

Very lucky in my case, hope it will be useful to you all as well.

[Mike Resseler]

Can you tell us what patches you removed? It might help other community members also and I don't want everyone to install all patches

Thanks
Mike

[WRLCjjordan]

I have a similar issue (now Case # 02197442) using a SAS attached HP2600 array.

I tried the change to the TCP/IP parameters, however since this is direct attached storage, I don't know that it would make any difference. I also tried upgrading to the latest & greatest 9.5 Update 2 to see if it would make any difference - no dice.

I have not yet removed the MS KB4015553 as I wanted to see if U2 (not the band ) would help.

One question to those that had experienced the error - what was the size of the machine? My other backups are running fine; the VMs in this case are multi-terabyte servers (3TB +) and I did't know if perhaps the file size was an issue.

Thoughts? Open to suggestions...
Thanks!
JJ

[jasonboche]

Long time Veeam user. Just recently deployed a new Veeam B&R 9.5.0.823 installation on Windows Server 2016 and now I'm seeing this problem as well. The backup server is configured with 4 CPUs and 8GB RAM. I'm using it to backup a handful of lab servers, one of which is approaching 1TB of data. It looks like 9.5.0.1038 Update2 is available now. I'm going to give that a try.

[Mike Resseler]

Hi Jordan, I am not aware that the problem would be with the size, but you never know. Please keep working with the support team for the investigation and keep us informed.

@Jason, After you upgrade to Update 2, please log a support case if you are still experiencing the problem. The more information we get, the better we can understand what is going on. Thanks!

[jasonboche]

Thanks Mike. I'm still having an issue after the update. I'll open a support case with Veeam.

[Mike Resseler]

Thanks Jason. Please post the case ID here and the follow-up

Cheers
Mike

[Murigar]

Though I would add my recent similar experience.

After updating to Update 2 i experienced this issue as explained by others.
My assigned support engineer observed I had a System Thread count.
In addition, I had approx 15 deprecated iSCSI targets configured and attempting to reconnect via Windows iSCSI Initiator.


After disconnecting / cleaning up the iSCSI targets, performance returned as expected.

Case ID: 02217124

[veremin]

Based on the issue description (deprecated ISCSI connections), don't you have one of those updates installed by any chance?