Backup jobs failing after uninstalling Backup Exec 2012?

[ChrisL]

Hi all,

Just spent a weekend battling with this one and thought it might be of some use to someone in a similar position. Apologies if this is covered elsewhere, I've had a bit of a dig and haven't found anything myself.

The short version:

Uninstalled Backup Exec 2012, Veeam jobs then failed at next run giving the error:

13/04/2014 20:42:24 :: Error: Client error: ChannelError: ConnectionReset

My (eventual) solution: from C:\Program Files\Veeam\Backup and Replication\Backup\Packages ran the VeeamTransport.msi and vPowerNFS.msi installers. Jobs now running as normal (although I don't know which one actually solved it since I did them both at the same time, sorry, not helpful)

The long version:

Time for a core hardware upgrade, 3 new VMWare hosts, and at the same time, updated to ESXi 5.5.

Along wih the hosts, updated the backup box hardware. Needs to be physycal since we run BE2010 to tape for offsite copies using a Quantum Tape Loader over SCSI.

Took the opportunity to update to BE2012 (bit of a mistake but I won't go into that here) and had it all working fine, Veeam jobs working OK on ESXi 5.5 and a test tape job working OK in BE2012. After finding that BE2012 has removed the concept of Selection Lists (Which I use heavily) and a few other reasons, decided to roll back to the tried and tested BE2010 that we've been using for years.

Uninstalled BE2012, rebooted, installed BE2010 (eventually, after convincing it that the local install of SQL2008R2 Express is a valid instance!) and ran a test tape job, all working fine until the scheduled Veeam jobs ran later in the day, throwing the error above.

On the phone to Veeam Support (Case Ref#: 00549248) digging through a few logs didn't give anything obvious besides the fact that the agent was shutting down the connection for some reason. Oddly, the .vmx, .vmxf and .nvram files were copying OK, so figured it must be working at some level. The jobs were failing immediately at the 'Hard Disk 1' action, apparently not even starting. Left Support with a copy of the logs to dig through.

Digging through myself, the logs showed:

[13.04.2014 20:41:57] <13> Error SrcBackupDiskFull failed (System.Exception)
[13.04.2014 20:41:57] <13> Error at Veeam.Backup.AgentProvider.CBackupClient.SrcBackupDiskFull(String pathToDesc, IFileVfsLink vbkLink, ICTAgentSpec ctkSpec, Boolean dedupEnabled)
[13.04.2014 20:41:57] <13> Error at Veeam.Backup.Core.CAgentDiskTransferTransport.ProcessDiskFromVmToBackupVi(CViDiskBackupObject diskBackupObject, CBackupDiskTarget diskTarget, COibEx prevOib, IDiskInfo prevDiskInfo)
[13.04.2014 20:41:57] <13> Error Client error: ChannelError: ConnectionReset (System.Exception)

Clearly the disk isn't full but it suggests something wrong. Checked the credentials being used by the proxy, no joy, checked the permissions on VCSA, all fine. Figured that they must be fine anyway since the job would fail far sooner in the process if they weren't correct.

After a few hours of poking around by myself and eliminating a few things (redirected the job target to local storage rather than the usual NAS box, started a whole new test job on a spare VM, re-ran the installer for Update 2a, rescanned the source and targets, and even moved the SQL database off to our central SQL2012 server) had no success.

Was about to bite the bullet and backup the config, remove Veeam and start again from scratch, but a random google search resulted in a suggestion about the transport, which I know is installed as separate components. From experience, I know the excellent Veeam installers should make this painless, and probably wouldn't make things worse, and sure enough, after locating the package installers, they went on fine, offering the 'repair' option. Ran the test jobs, fingers crossed, and wouldn't you know, they went on to work OK, harrah!

Now, this may all be entirely circumstantial, and I'm not saying that this is a true solution, and I'm definitely not saying that everyone should do the same thing when facing this error (!) but it certainly worked OK for me in this particular set of conditions. I would be interested to know however, if there is any element of logic to this. I gather that BE2012 offers similar functionality to Veeam in handling VMs and backing up from snapshots, so presumably it's concievable that uninstalling it could break something that Veeam then goes on to use to do its thing..?

I Know you like to keep a nice clean interface, but maybe useful to have a button somewhere to say 'Repair Transport', which simply re-runs the transport in repair mode?

Thanks as always to the excellent Support team, always nice to have them at the end of the phone in a time of crisis.

Chris

[foggy]

I would be interested to know however, if there is any element of logic to this. I gather that BE2012 offers similar functionality to Veeam in handling VMs and backing up from snapshots, so presumably it's concievable that uninstalling it could break something that Veeam then goes on to use to do its thing..?

Could be some redistributable runtime libraries. Thanks for sharing, anyway!

[blacktrainee]

Hallo ChrisL,

thank you for sharing this solution. Running both MSI "solve" the Problem.

Serge

[Threonine]

Chris, thank you very much for troubleshooting this and documenting what you found! Nice work! I just ran into this same strange issue after uninstalling Backup Exec 2014 from the server on which the Veeam Proxy agent was running. Each VM replication failed with "Error: Client error: ChannelError: ConnectionReset". I ran just the Veeam Transport MSI from "\\veeamserver\c$\Program Files\Veeam\Backup and Replication\Backup\Packages\VeeamTransport.msi" and completed a Repair and then retried the job and it completed successfully.

[menthol]

Incase anyone comes across this via Google like me, the fix wasn't to repair those MSIs but to backup through the standalone host. See the below from Vlad Luchanskiy @ Veeam Support:

Thanks for the logs. The root cause for the issue is the vCenter that we are backing up.

When we taking the VSS backup of vCenter's database, it freezes thus breaking the operations of all other backups and resetting the connection.

Please, follow the article <http://www.veeam.com/kb1051> and check how the job will behave after that.

My vCenter is a Virtual Appliance.

Hopefully that helps anyone who comes across this issue!

[Perceptum]

Chris, thank you for figuring this one out. In my case, I only had the Backup Exec 2012 agent installed and just upgraded it to the 2014 agent.

[jndtech]

Chris... You the man bro!!! This was driving me nuts!!!! Thanks for figuring this out... You saved my day!!! Appreciate it!!