Host-based backup of Microsoft Hyper-V VMs.
Post Reply
enoch
Service Provider
Posts: 162
Liked: 11 times
Joined: Jun 29, 2013 12:14 pm
Full Name: Peter Enoch
Contact:

Wierd Veeam B&R server after Windows Updates / Reboot

Post by enoch »

Hi all,

Had a pretty solid B&R running Windows 2012r2 DataCenter Edition with current Veeam 9.5 U1 running.

All Windows patches is up-to-date from Windows Update feature in Windows. Last time I updated the server is months ago. In the window we updated the SQL 2014 SP2 to SP2 CU5.

After this on the B&R server it self I have had some "lag" in the interface and 3-4 times today got the message:

"The connection to backup server has been lost. Attempting to reconnect to your session"

"Connection attepmt: 1 of 10"

All services: console, SQL, B&R is on same server!

A lot of jobs I suddenly gets the error (both VMware, Hyper-V backup jobs and replication jobs):
"Full backup file merge failed error: ChannelError: ConnectionReset"

All Backup repositories is on local storage!

Tried restore from some of the jobs failling and they seems to Work fine!

I have a current open case at Veeam: 02140683 but if anyone have any fix/ideas I'll be glad to hear from you!
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Wierd Veeam B&R server after Windows Updates / Reboot

Post by Vitaliy S. »

Hi Peter,

Do you have any SQL Server related error messages in the Windows Event log?

Thanks!
enoch
Service Provider
Posts: 162
Liked: 11 times
Joined: Jun 29, 2013 12:14 pm
Full Name: Peter Enoch
Contact:

Re: Wierd Veeam B&R server after Windows Updates / Reboot

Post by enoch »

Hi,

I can't find any SQL related errors in the System or Application Event Logs.

It's seems a reboot "fix" the problem, 4-6 hours and then problems again.

A stop of the Veeam Backup Service results in not able to start the service - a server reboot is needed! :cry:
Vitaliy S.
VP, Product Management
Posts: 27055
Liked: 2710 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Wierd Veeam B&R server after Windows Updates / Reboot

Post by Vitaliy S. »

OK, then let's see what our support engineer says after reviewing the debug log files.
enoch
Service Provider
Posts: 162
Liked: 11 times
Joined: Jun 29, 2013 12:14 pm
Full Name: Peter Enoch
Contact:

Re: Wierd Veeam B&R server after Windows Updates / Reboot

Post by enoch »

Yes, running the case with Veeam support now.

Hope they find the problem and a fix.
JosvwHLT
Enthusiast
Posts: 47
Liked: 3 times
Joined: Dec 24, 2015 4:15 am
Full Name: Jos van Wordragen
Contact:

Re: Wierd Veeam B&R server after Windows Updates / Reboot

Post by JosvwHLT »

Last Friday I updated 2 Veeam B&R 9.0 servers to 9.5 u1. One of the 2 is experiencing the same problems. A server reset is the only option. In the systemlog i see evenitid 4227 - TCP/IP Network Connectivity (EVENT_TCPIP_TCP_TIME_WAIT_COLLISION). NIC disable/enable option is not available because server is almost non-responsive.
enoch
Service Provider
Posts: 162
Liked: 11 times
Joined: Jun 29, 2013 12:14 pm
Full Name: Peter Enoch
Contact:

Re: Wierd Veeam B&R server after Windows Updates / Reboot

Post by enoch »

Hi all,

I've tried REALLY much right now.

I see there is a issue with the Microsoft patch: kb4015553 for a lot of people - Windows 2012r2 - the patch is from late April 2017.

Sadly I cannot remove the patch from the server without an error :-(

Maybe if you have this installed you could try with uninstall?
JosvwHLT
Enthusiast
Posts: 47
Liked: 3 times
Joined: Dec 24, 2015 4:15 am
Full Name: Jos van Wordragen
Contact:

Re: Wierd Veeam B&R server after Windows Updates / Reboot

Post by JosvwHLT »

Don't want to uninstall any patches right now before knowing what is really going on. We are running a tape backup procedure in which we mount a backup image (with VeeamFLR) and retrieve data from that mounted image and dump it to tape. During this procedure the server crashed already twice. I'm running the Veeam B&R server now without running any tape procedure. Want to wait for that outcome first before i change any thing on that server. We are also experiencing a failed service start of the "VeeamCatalogsvc" service, this service won't start anymore after we experienced the issues with the "unresponsive server" but don't know if it is related. In the application log i see the following "Service Cannot be started", part of the message is the following "Attribute "LastSyncTimeUTC" was not found under the tree".
JosvwHLT
Enthusiast
Posts: 47
Liked: 3 times
Joined: Dec 24, 2015 4:15 am
Full Name: Jos van Wordragen
Contact:

Re: Wierd Veeam B&R server after Windows Updates / Reboot

Post by JosvwHLT »

Applied TCP/IP settings as described in URL vmware-vsphere-f24/veeam-flr-t32119-15.html This seems to have fixed my issue, server is running for a couple of days now without any issues. Server froze when running a tape procedure which copied a lot of small files to tape. That is running fine now too.
enoch
Service Provider
Posts: 162
Liked: 11 times
Joined: Jun 29, 2013 12:14 pm
Full Name: Peter Enoch
Contact:

Re: Wierd Veeam B&R server after Windows Updates / Reboot

Post by enoch » 1 person likes this post

Hi all,

Afters days suppport with Microsoft who didn't have any luck removing the KB4015553 which could not be removed in Windows, I succeed removing the patch.

After the patch was removed my system have now run 24 hours without issues.

Before I could only have the system running about 8 hours before I was needed to do a reboot.
kstrp
Lurker
Posts: 2
Liked: never
Joined: May 10, 2017 1:59 pm
Full Name: RP
Contact:

Re: Wierd Veeam B&R server after Windows Updates / Reboot

Post by kstrp »

enoch wrote:Hi all,

Afters days suppport with Microsoft who didn't have any luck removing the KB4015553 which could not be removed in Windows, I succeed removing the patch.

After the patch was removed my system have now run 24 hours without issues.

Before I could only have the system running about 8 hours before I was needed to do a reboot.
Mind sharing how you removed it? Couldn't get that to happen on our own and microsoft couldn't figure out when we spoke to them briefly.
enoch
Service Provider
Posts: 162
Liked: 11 times
Joined: Jun 29, 2013 12:14 pm
Full Name: Peter Enoch
Contact:

Re: Wierd Veeam B&R server after Windows Updates / Reboot

Post by enoch »

I would like to, but not 100% sure what did it :D

Please try this:
Dism /Online /Cleanup-Image /CheckHealth - check everything is OK

Dism /Online /Cleanup-Image /ScanHealth - check everything is OK

If these are not OK, then: DISM.exe /Online /Cleanup-image /Restorehealth and: sfc /scannow

Download the KB4015553 manually from Microsoft: http://www.catalog.update.microsoft.com ... =KB4015553

Then extract thethe KB4015553.msu package.

Then do a DISM.exe /Online /Add-Package /PackagePath:"path to KB4015553.cab file"

Then do a dism /remove-package with this update. Find the package ID with: Dism /Online /Get-Packages

After this I went in the Add/Remove program an the uninstall worked!
kstrp
Lurker
Posts: 2
Liked: never
Joined: May 10, 2017 1:59 pm
Full Name: RP
Contact:

Re: Wierd Veeam B&R server after Windows Updates / Reboot

Post by kstrp »

Hmm - I've tried similar steps already - but I'll give it another go covering what you listed.

Thanks again.
LittlePiGPiG
Lurker
Posts: 1
Liked: never
Joined: Jun 02, 2017 3:43 pm
Contact:

Re: Wierd Veeam B&R server after Windows Updates / Reboot

Post by LittlePiGPiG »

I experienced exactly / very much similar symptoms of everyone above said and sadly the Veeam support had not yet come up with any conclusions.

Then, I tried removing latest "updates" (Except those M$ "security patches") and I get a normal V&R server back.

Very lucky in my case, hope it will be useful to you all as well.
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Wierd Veeam B&R server after Windows Updates / Reboot

Post by Mike Resseler »

Can you tell us what patches you removed? It might help other community members also and I don't want everyone to install all patches ;-)

Thanks
Mike
WRLCjjordan
Lurker
Posts: 1
Liked: never
Joined: Jun 23, 2017 3:34 pm
Full Name: J Jordan
Contact:

Re: Wierd Veeam B&R server after Windows Updates / Reboot

Post by WRLCjjordan »

I have a similar issue (now Case # 02197442) using a SAS attached HP2600 array.

I tried the change to the TCP/IP parameters, however since this is direct attached storage, I don't know that it would make any difference. I also tried upgrading to the latest & greatest 9.5 Update 2 to see if it would make any difference - no dice.

I have not yet removed the MS KB4015553 as I wanted to see if U2 (not the band :shock: ) would help.

One question to those that had experienced the error - what was the size of the machine? My other backups are running fine; the VMs in this case are multi-terabyte servers (3TB +) and I did't know if perhaps the file size was an issue.

Thoughts? Open to suggestions...
Thanks!
JJ
jasonboche
Novice
Posts: 5
Liked: 1 time
Joined: Aug 16, 2010 1:43 am
Full Name: Jason Boche
Contact:

Re: Wierd Veeam B&R server after Windows Updates / Reboot

Post by jasonboche »

Long time Veeam user. Just recently deployed a new Veeam B&R 9.5.0.823 installation on Windows Server 2016 and now I'm seeing this problem as well. The backup server is configured with 4 CPUs and 8GB RAM. I'm using it to backup a handful of lab servers, one of which is approaching 1TB of data. It looks like 9.5.0.1038 Update2 is available now. I'm going to give that a try.
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Wierd Veeam B&R server after Windows Updates / Reboot

Post by Mike Resseler »

Hi Jordan, I am not aware that the problem would be with the size, but you never know. Please keep working with the support team for the investigation and keep us informed.

@Jason, After you upgrade to Update 2, please log a support case if you are still experiencing the problem. The more information we get, the better we can understand what is going on. Thanks!
jasonboche
Novice
Posts: 5
Liked: 1 time
Joined: Aug 16, 2010 1:43 am
Full Name: Jason Boche
Contact:

Re: Wierd Veeam B&R server after Windows Updates / Reboot

Post by jasonboche »

Thanks Mike. I'm still having an issue after the update. I'll open a support case with Veeam.
Mike Resseler
Product Manager
Posts: 8044
Liked: 1263 times
Joined: Feb 08, 2013 3:08 pm
Full Name: Mike Resseler
Location: Belgium
Contact:

Re: Wierd Veeam B&R server after Windows Updates / Reboot

Post by Mike Resseler »

Thanks Jason. Please post the case ID here and the follow-up

Cheers
Mike
Murigar
Enthusiast
Posts: 37
Liked: 4 times
Joined: Nov 05, 2014 4:19 pm
Contact:

Re: Wierd Veeam B&R server after Windows Updates / Reboot

Post by Murigar »

Though I would add my recent similar experience.

After updating to Update 2 i experienced this issue as explained by others.
My assigned support engineer observed I had a System Thread count.
In addition, I had approx 15 deprecated iSCSI targets configured and attempting to reconnect via Windows iSCSI Initiator.


After disconnecting / cleaning up the iSCSI targets, performance returned as expected.

Case ID: 02217124
veremin
Product Manager
Posts: 20270
Liked: 2252 times
Joined: Oct 26, 2012 3:28 pm
Full Name: Vladimir Eremin
Contact:

Re: Wierd Veeam B&R server after Windows Updates / Reboot

Post by veremin »

Based on the issue description (deprecated ISCSI connections), don't you have one of those updates installed by any chance?
Post Reply

Who is online

Users browsing this forum: No registered users and 24 guests