Veeam EM 9.5 - Failed to load Msxmlsql.dll Case#02028627

[bsylvain]

Case#02028627

Hi guys,
We've just upgraded yesterday from 9.0 to 9.5, backups and replications are running fine but we have a problem with the Enterprise Manager.

Server :
Windows Server 2008 R2 Standard SP1
CPU Xeon E5-2630L
32 GB of RAM
Enterprise Manager runs on same server as B&R and SQL server Express.

Context :
To upgrade we had to migrate the VeeamBackup (B&R) and VeeamBackupReporting (Enterprise Manager) databases from MSSQL Express 2005 to MSSQL Express 2008 R2 (10.50.6220).
We did it that way :
- Backup DBs from 2005
- Restore DBs to 2008 R2
- Ran Veeam tool "Configuration Database Connection Settings" to select both DBs on 2008R2 instance (by the way "BackupDatabase.xml" and "ReportingDBupdate.xml" requested by the tool, were not at the right place)
- Changed compatibility level of DBs to SQL Server 2008 (100)
- Upgraded to 9.5 without error

Now :
Veeam B&R works fine, backups and replications ran last night without issue.
Veeam Enterprise Manager works partially :
- Dashboard tab is OK
- Reports tab is OK
- Jobs tab fails : Failed to load Msxmlsql.dll. Could not find prepared statement with handle 0. Could not find prepared statement with handle 0. Could not find prepared statement with handle 0. sp_xml_removedocument: The value supplied for parameter number 1 is invalid.
- VMs tab fails : Failed to load Msxmlsql.dll. Could not find prepared statement with handle 0. Could not find prepared statement with handle 0. sp_xml_removedocument: The value supplied for parameter number 1 is invalid. Failed to load Msxmlsql.dll. Could not find prepared statement with handle 0. sp_xml_removedocument: The value supplied for parameter number 1 is invalid. The statement has been terminated. @userSid attribute not found.
- Files tab is OK (I can select VM, restore points, and browse files)
- Items tab is OK
- Requests tab fails : Failed to load Msxmlsql.dll. Could not find prepared statement with handle 0. Could not find prepared statement with handle 0. sp_xml_removedocument: The value supplied for parameter number 1 is invalid.
- Configurations tab is OK

Svc.VeeamBES.log shows :

Code: Select all

[04.01.2017 10:40:29] <15> Error    Failed to execute SQL stored procedure "[dbo].[usp.GridReport.DataSrc.CurrentJobs]".
[04.01.2017 10:40:29] <15> Error    Failed to load Msxmlsql.dll.
[04.01.2017 10:40:29] <15> Error    Could not find prepared statement with handle 0.
[04.01.2017 10:40:29] <15> Error    Could not find prepared statement with handle 0.
[04.01.2017 10:40:29] <15> Error    Could not find prepared statement with handle 0.
[04.01.2017 10:40:29] <15> Error    sp_xml_removedocument: The value supplied for parameter number 1 is invalid. Error: Failed to load Msxmlsql.dll.
[04.01.2017 10:40:29] <15> Error    Could not find prepared statement with handle 0.
[04.01.2017 10:40:29] <15> Error    Could not find prepared statement with handle 0.
[04.01.2017 10:40:29] <15> Error    Could not find prepared statement with handle 0.
[04.01.2017 10:40:29] <15> Error    sp_xml_removedocument: The value supplied for parameter number 1 is invalid. (Veeam.Backup.Common.CRegeneratedUserException)
[04.01.2017 10:40:29] <15> Error       à Veeam.Backup.Reporting.DbLayer.CReportingDbAccessor.RegenUserException(Exception srcExc)
[04.01.2017 10:40:29] <15> Error       à Veeam.Backup.DBManager.CDbExecutorImpl.OnErrorOccured(CSqlExecErrorEventArgs args)
[04.01.2017 10:40:29] <15> Error       à Veeam.Backup.DBManager.CDbExecutorImpl.HandleExeption(Exception sqlExc, Boolean& retryDbOperation)
[04.01.2017 10:40:29] <15> Error       à Veeam.Backup.DBManager.CDbExecutorImpl.HandleExeptionOrLogError(Exception sqlExc, Boolean& retryDbOperation, String msgFormat, Object[] args)
[04.01.2017 10:40:29] <15> Error       à Veeam.Backup.DBManager.CDbExecutorImpl.ExecuteStoredProc(TimeSpan timeout, DataTable trgTable, String procedureName, SqlParameter[] arguments)
[04.01.2017 10:40:29] <15> Error       à Veeam.Backup.Reporting.DbLayer.CDataPartition.ExecuteStoredProc(TimeSpan timeout, String procName, SqlParameter[] args)
[04.01.2017 10:40:29] <15> Error       à Veeam.Backup.Reporting.DbLayer.CDirectGridDataFeederImpl.RetrieveData(Int32 start, Int32 limit, Int32& total)
[04.01.2017 10:40:29] <15> Error    Failed to load Msxmlsql.dll.

continued ...

We ran procmon and we can see that :
sqlservr.exe tries to load C:\Program Files\Microsoft SQL Server\MSSQL10_50.VEEAMSQL2008R2\MSSQL\Binn\msxmlsql.dll
C:\Program Files\Microsoft SQL Server\MSSQL10_50.VEEAMSQL2008R2 is the correct instance
but msxmlsql.dll does not exists there.
We tried to add the missing msxmlsql.dll (taken from C:\Program Files\Microsoft SQL Server\100\Shared\msxmlsql.dll which is also loaded at this time, successfully)
but it does not work (also the file is loaded successfully).

Any clue how we can fix this error?

Thanks

[veremin]

Please, keep working with your support engineer. It's rather hard to investigate this sort of the issues without seeing debug logs and actual infrastructure. Thanks.

[bsylvain]

Alright problem has been solved with Veeam support.

Solution was :
Copy msxmlsql.dll from
C:\Program Files\Microsoft SQL Server\100\Shared\msxmlsql.dll
to
C:\Program Files\Microsoft SQL Server\MSSQL10_50.[veeam-dbinstance]\MSSQL\Binn

then copy "msxmlsql.rll" from
C:\Program Files\Microsoft SQL Server\100\Shared\Resources\1033\msxmlsql.rll
to
C:\Program Files\Microsoft SQL Server\MSSQL10_50.[veeam-dbinstance]\MSSQL\Binn\Resources\1033

Appears to be a bug of SQL server 2008R2

Might it be usefull for someone.

[Mike Resseler]

Glad to see it resolved. Thanks for coming back to us with the resolution

Cheers
Mike