SideBySide issue with VEEAM B&R v7 patch 2 (R2)

[STGdb]

I am having an issue with VEEAM B&R v7 patch 2. I’ve opened support case # 00477536 with VEEAM and we’re still working on it but I wanted to see if anyone else ran into this issue.

I am using VEEAM to back up VM’s on a 5.1C ESXI cluster. I have a VM that has AD running on a 2008 R2 SP1 server. I’m trying to set up a SureBackup job to test the VM backup (and to restore AD objects). There are no dependencies with this VM so it is a simple test. The VM is only a few days old and has AD/DNS services on it. I ran backups of the VM as a test with VEEAM B&R v7 patch #1. I created a dedicated SureBackup job to test those backups and it worked fine (but I had a separate issue with restoring AD objects which is what I was working on). B&R v7 patch 2 came out and I install it on the VEEAM server (hoping it would fix my AD restore issue). Instead, the latest patch broke the SureBackup job. The SureBackup job completed successfully with patch #1 but now fails with patch #2. I’ve tried reinstalling patch #2 but no luck. I’ve tried using the SureBackup job to test a different restore point (both patch #1 and patch #2 restore points, full and incremental) but got the same results. I can successfully restore Windows OS files from backups of this VM that were created when either patch #1 or patch #2 were installed. Since installing patch #2, whenever I run the SureBackup job, I am seeing SideBySide errors in my Windows application event log that weren’t there with patch #1 (during the Configuring DC step of the SureBackup job).

The SureBackup job info is:

Code: Select all

11/15/2013 7:38:50 AM          Publishing
11/15/2013 7:38:38 AM          Backup point: created at 11/12/2013 2:10:48 PM, type full
11/15/2013 7:38:48 AM          Backup repository: Backup Repository (type windows server, server This server, mount server This server)
11/15/2013 7:38:48 AM          Virtual resources: host <removed>, datastore <removed>, VM <removed>
11/15/2013 7:38:48 AM          Virtual lab: <removed>
11/15/2013 7:38:48 AM          VM: <removed> (application group, 4096 MB vRAM)
11/15/2013 7:38:49 AM          OS: Microsoft Windows Server 2008 R2 (64-bit)
11/15/2013 7:38:49 AM          Network adapter 1: MAC '<removed>', type 'vpx', network 'unknown'
11/15/2013 7:38:49 AM          Assigned roles: dns server, domain controller, global catalog
11/15/2013 7:38:49 AM          Maximum boot time: 600 second(s)
11/15/2013 7:38:49 AM          Application initialization: 180 sec
11/15/2013 7:38:49 AM          Heartbeat test: enabled
11/15/2013 7:38:49 AM          Ping test: enabled
11/15/2013 7:38:49 AM          Script tests: enabled, 3 tests
11/15/2013 7:38:49 AM          Backup file validation: disabled
11/15/2013 7:38:59 AM          Reconfiguring
11/15/2013 7:38:59 AM          Network 1: production '<removed>', isolated '<removed>', mapped
11/15/2013 7:38:59 AM          Results: 1/1 network(s) mapped, 0 unmapped
11/15/2013 7:38:59 AM          Summary: 100% total pass rate
11/15/2013 7:39:04 AM          Registering
11/15/2013 7:39:56 AM Error    Configuring DC
11/15/2013 7:39:56 AM Error    Error: Client error: Failed to open VDDK disk [[<removed>] <removed>.vmdk] ( is read-only mode - [false] )
                               Dynamic library cannot be loaded. Path to library: [C:\Program Files (x86)\Veeam\Backup Transport\x64\vddk_5_5\vixDiskLib.dll].
                               Failed to initialize VDDK library. Installation folder: [].
                               Logon attempt with parameters [VC/ESX: [<removed>];Port: 443;Login: [****];VMX Spec: [moref=vm-2611];Snapshot mor: [];Transports: [nbd];Rea
11/15/2013 7:39:57 AM          Unregistering
11/15/2013 7:40:00 AM          Cleaning up redo logs
11/15/2013 7:40:01 AM          Unpublishing
11/15/2013 7:40:01 AM          Task finished

The info from the Windows application event log is (this repeats six times in the log whenever the SureBackup job starts the "Configuring DC" step):

Code: Select all

Log Name:      Application
Source:        SideBySide
Date:          11/15/2013 7:39:05 AM
Event ID:      33
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      <removed>
Description:
Activation context generation failed for "C:\Program Files (x86)\Veeam\Backup Transport\x64\vddk_5_5\vixDiskLib.dll". Dependent Assembly Microsoft.VC90.CRT,processorArchitecture="amd64",publicKeyToken="<removed>",type="win32",version="9.0.30729.4148" could not be found. Please use sxstrace.exe for detailed diagnosis.
Event Xml:
<Event xmlns="http://schemas.microsoft.com/win/2004/08/events/event">
  <System>
    <Provider Name="SideBySide" />
    <EventID Qualifiers="49409">33</EventID>
    <Level>2</Level>
    <Task>0</Task>
    <Keywords>0x80000000000000</Keywords>
    <TimeCreated SystemTime="2013-11-15T12:39:05.000000000Z" />
    <EventRecordID>5526</EventRecordID>
    <Channel>Application</Channel>
    <Computer><removed></Computer>
    <Security />
  </System>
  <EventData>
    <Data>Microsoft.VC90.CRT,processorArchitecture="amd64",publicKeyToken="<removed>",type="win32",version="9.0.30729.4148"</Data>
    <Data>
    </Data>
<…these DATA sections repeat but I truncated it down to save space…>
    <Data>
    </Data>
    <Data>C:\Program Files (x86)\Veeam\Backup Transport\x64\vddk_5_5\vixDiskLib.dll</Data>
    <Data>
    </Data>
<…these DATA sections repeat again but I truncated it down to save space…>
    <Data>
    </Data>
  </EventData>
</Event>

Thanks for any help or input

[foggy]

Daniel, have you successfully upgraded all Veeam B&R components (suggested automatically once you open the console for the first time after v7 R2 installation)?

You can perform components upgrade manually using the corresponding command on the Veeam B&R main menu and see whether this helps.

Thanks.

[STGdb]

Yes, when I launched the console the first time and installing R2, I was presented with a message that my components needed to be upgraded as well. The process ran and reported that it was successful.

I can't run the upgrade manually, the "Upgrade" option is greyed out. Is there possibly a command line that I can use to launch the installer/MSI package?


Thanks

[tsightler]

Can you verify that "Microsoft Visual C++ 2008 Redistributable -x64 9.0.30729.4148" is installed on your systems. Just looking at the error message it seems that the VDDK has a dependency on this and can't find it. You should see this in Control Panel "Program and Features". If not, you can download and try to install if from here:

http://www.microsoft.com/en-us/download ... px?id=2092

Normally I believe that the Veeam installer should take care of this, but that sure is what the error looks like so it's worth checking.

[cdabbicco]

00478439 same problem for me. after update to R2 all backup job failed for same problem!

[STGdb]

The info from tsightler led me down the correct trail of breadcrumbs. I checked and found that I had the following installed (all of them were there prior to the VEEAM v7 R2 update):

Microsoft Visual C++ 2008 Redistributable x86 9.0.21022
Microsoft Visual C++ 2008 Redistributable x86 9.0.30729.6161
Microsoft Visual C++ 2010 Redistributable x64 10.0.40219

The link from tsightler led me to Microsoft Visual C++ 2008 Redistributable 2008 x64 9.0.30729.17 (and I needed 2008 x64 9.0.30729.4148) – but it got me moving in the right direction. For anyone who needs it, you can get the 2008 x64 9.0.30729.4148 release at:

http://www.microsoft.com/en-us/download ... x?id=11895

After I installed 2008 x64 9.0.30729.4148, the SureBackup job works. I’m guessing maybe the R2 installation package thought that I already had a more recent release the library installed and so it didn’t install 2008 x64 9.0.30729.4148 as part of the R2 package prerequisites? Not sure...

And I’m not sure yet if R2 fixed my issue with restoring AD objects but at least I can resume testing on that front.


Thanks much to all

[foggy]

This turns out to be an issue with Veeam installer not placing required redistributable files into the correct folder. You can either run the redistributable setup, as Daniel has described above, or manually copy the msvcr90.dll file from

C:\Program Files (x86)\Veeam\Backup Transport\x64\Microsoft.VC90.CRT
to
C:\Program Files (x86)\Veeam\Backup Transport\x64\vddk_5_5

folder. Either workaround should be performed on all proxy/repository servers.

[tsightler]

Ah, sorry about posting the link to the wrong version. must have been a copy/paste error on my part as I had to work my way through several links to finally find the correct one just as you did. Please feel free to reach out to support if you continue to experience issues with the AD object restore.

Glad to be aware that this is now a known issue for R2 with a simple workaround.

[STGdb]

No worries - thanks for helping point me to the right answer.

R2 didn't help me with the AD restore issue, we're still having problems with the restore stopping at 99% on the SureBackup job (but the SureBackup job itself works fine when run separately), but it sits there at 99% until the restore job time expires. We'll continue working on that one.

Thanks again!!!

[tsightler]

Actually, 99% means that it is "ready" for the recovery to be performed, i.e. all systems in the lab are up and running. I think that's pretty confusing, but that's the way it works. Are you saying that it's not showing up in the Lab Manager as ready? If you actually double click the Surebackup job in the Veeam GUI and right click on the domain controller you should be able to start the Active Directory item recovery wizard from there. It's just a different method to get to the same thing. Not saying that's the final solution, but it would help verify if everything else was working correctly.

[STGdb]

Normally yes, but unfortunately my SureBackup is stuck at 99% and my virtual lab is in perpetual "Preparing" mode (doesn't get beyond that now)

[systeembeheernsd]

We also had a problem after the patch2/r2 update, all jobs failed with the following error:

Processing 'xxxxx' Error: Client error: Failed to open VDDK disk [[datatorexxzz] xxxxxx/xxxxxx_3.vmdk] ( is read-only mode - [true] ) Dynamic library cannot be loaded. Path to library: [C:\Program Files (x86)\Veeam\Backup Transport\x64\vddk_5_5\vixDiskLib.dll]. Failed to initialize VDDK library. Installation folder: []. Logon attempt with parameters [VC/ESX: [xxxxx];Port: 443;Login: [****];VMX Spec: [moref=vm-3691];Snapshot mor: [snapshot-49726];Transports: [nbd];Read Only: [true]] fail

this was also fixed by copying Folder C:\Program Files (x86)\Veeam\Backup Transport\x64\Microsoft.VC90.CRT to C:\Program Files (x86)\Veeam\Backup Transport\x64\vddk_5_5\ like Foggy said earlier in the thread.

[Gostev]

We will be updating the R2 package next week to address this issue for the future customers. I wish we could do this faster, but we spent much time working out reliable solution. Just deploying said redistributable automatically did not work well, as this appeared not very straightforward (things like reboot requirements, and a few other issues). And the issue is a bit more complex than "Veeam installer not placing required redistributable files into the correct folder", so simply copying may not always be sufficient either. And, we did not want to require manual installation. But, we seem to have found a solid solution finally.

As far as I know, the scope is fairly small. I will double check with the QC when I am in the office tomorrow, but my understanding is that this should only affect Windows 2008/2008R2 or earlier physical backup proxy servers backing up from ESXi 5.5. If your backup proxy is virtual and has VMware Tools installed, or runs Windows Server 2012 or later, or you are not yet on vSphere 5.5, then you should not be affected.

For now, the best resolution is to install the package linked above manually on those systems that can potentially be affected. Of course, you can first try copying and see if that works.

[Lewpy]

I came across an issue this weekend when attempting to use Replica Seeding for a replication job which I manage to resolve, so I thought I would post my findings.

VMware Environment - a mixture as we are in the middle of migrating to a new platform
Source - VMware vCenter 5.5a with ESXi 5.1 U1 hosts
Destination - VMware vCenter 4.1 with ESX 4.1 hosts

Veeam Environment
Source - Veeam B&R 7 R2 installed on physical server [dual quad core] running Windows 2012 R2 with Direct SAN access to storage
Source - Veeam B&R 7 R2 Proxy on virtual server [quad vCPU] running Windows 2012 R2
Destination - Veeam B&R 7 R2 Proxy on virtual server [quad vCPU] running Windows 2008 R2 SP1 [not Windows 2012, due to ESX version (see above)]

I had initially created a replication job on another [temporary] Veeam server in the Source environment as the physical server was not yet available [32 VMs totalling 11TB].
Once I had built the physical server, I re-created the replication job on the physical server and used the "Replica Seeding" option to match up the Source and Destination VMs (matched perfectly using the Discover option). I couldn't "copy" the configuration from the VM to the physical server, as I had already configured the physical server with Backup Jobs and set up the Tape Autoloader. As you can't export/import individual jobs [boo! ], I had to manually recreate the job, painstakingly selecting all the same options, locations, etc.

The issue I had was that when the job ran, it would get as far as "Calculating digests Hard Disk x" and then fail this after exactly 1 minute. And it did it for all VMs.
However, performing normal replications (both initial replications and subsequent incremental/CBT replications) worked fine: it was just the process of "Calculating digests" which seemed to fail.

I tried switching between Source proxies (as I have two) and Destination Proxy access methods (hotadd & nbd), but nothing helped.
The error shown in the GUI, and the Veeam Server logs was

30/11/2013 23:31:43 :: Error: Client error: Failed to open VDDK disk [[MY_LUN] MY_VM_replica/MY_VM_1-000002.vmdk] ( is read-only mode - [true] )
Dynamic library cannot be loaded. Path to library: [C:\Program Files (x86)\Veeam\Backup Transport\x64\vddk_5_5\vixDiskLib.dll].
Failed to initialize VDDK library. Installation folder: [].
Logon attempt with parameters [VC/ESX: [my_vc.my_domain];Port: 443;Login: [****];VMX Spec: [moref=vm-190192];Snapshot mor: [snapshot-190953];Transports: [nbd];Read Only: [true]] failed

After sleeping on the issue, I took a fresh look this morning
I went digging in to the Veeam log files (under C:\ProgramData\Veeam\Backup\My_Job), and discovered the following error on the main Veeam Server

[30.11.2013 23:31:00] <51> Info [AP] (0ef2) output: --trace_event:3:\n
[30.11.2013 23:31:00] <51> Info [AgentsReconnectableSession] Client error trace event received from source client. Connection is not recoverable.
[30.11.2013 23:31:00] <51> Info [AP] (0ef2) error: Failed to open VDDK disk [[MY_LUN] MY_VM_replica/MY_VM_1-000002.vmdk] ( is read-only mode - [true] )\n--tr:{\n--tr:The application has failed to start because its side-by-side configuration is incorrect. Please see the application event log or use the command-line sxstrace.exe tool for more detail.\n\nDynamic library cannot be loaded. Path to library: [C:\Program Files (x86)\Veeam\Backup Transport\x64\vddk_5_5\vixDiskLib.dll].\n--tr:vixDiskLib initialization failure.\nFailed to initialize VDDK library. Installation folder: [].\nLogon attempt with parameters [VC/ESX: [my_vc.my_domin];Port: 443;Login: [****];VMX Spec: [moref=vm-190192];Snapshot mor: [snapshot-190953];Transports: [nbd];Read Only: [true]] failed because of the following errors:\n--tr:}\nFailed to process [calcDiskDigestsCli].\n--tr:Client failed to process the command. Command: [calcSignature].\n
[30.11.2013 23:31:00] <44> Info [AP] (0ef2) state: closed

Ah ha, a side-by-side configuration error was something to go on
And sure enough, the application event log on the Destination Proxy server was full of these events

Code: Select all

Log Name:      Application
Source:        SideBySide
Date:          30/11/2013 23:30:42
Event ID:      33
Task Category: None
Level:         Error
Keywords:      Classic
User:          N/A
Computer:      my_proxy.my_domain
Description:
Activation context generation failed for "C:\Program Files (x86)\Veeam\Backup Transport\x64\vddk_5_5\vixDiskLib.dll". Dependent Assembly Microsoft.VC90.CRT,processorArchitecture="amd64",publicKeyToken="1fc8b3b9a1e18e3b",type="win32",version="9.0.30729.4148" could not be found. Please use sxstrace.exe for detailed diagnosis.

Looking in the Programs & Features of the Destination Proxy (very vanilla Windows 2008 R2 SP1 server), there were no "Microsoft Visual C++ 2008 Redistributable" packages installed.
Looking in the Veeam Transport Service directory (C:\Program Files (x86)\Veeam\Backup Transport) there was a Visual C++ file of the right build number under "C:\Program Files (x86)\Veeam\Backup Transport\x64\Microsoft.VC90.CRT", but it didn't appear to be being located when needed.

So I then looked to install the required "Microsoft Visual C++ 2008 Redistributable - x64 9.0.30729.4148" package on the Destination Server, when I noticed it was already installed on the main Veeam Server.
Checking installation date-stamps, I determined it was installed as part of the VMware VI Client installation.

So my solution was to install the VMware VI Client on to the Destination Proxy, which therefore installed the required Microsoft Visual C++ packages, and then my Replica Seeding started working
I could have gone hunting for just the required Microsoft Visual C++ Redistribute package from Microsoft, but this was the simplest fix for me.

I believe this is an issue that Veeam need to look at, as I suspect the Veeam Transport Service is meant to be self-contained and not require manual installation of Microsoft Visual C++ packages. It looks like the required DLL is in the Veeam installation, but perhaps is not "selected" correctly at runtime. This could easily be missed, as that Microsoft Visual C++ package is bundled with many other VMware installation packages: I only was affected as the Destination Proxy VM was a very basic "just OS" build.

[Gostev]

Yes, this is the known issue by now, there is an existing topic about it in the main forum. We will be updating R2 package shortly to make sure future users do not run into this. You are correct, even having VMware Tools installed will fix the issue.

[Lewpy]

Hi Gostev,

I did search initially and didn't find a post about Replica Seeding and Hashing, but now I have located the other posting (OP was about SureBackup issues).
However, I would like to point out that my Destination Proxy server did have VMware Tools installed (ESX 4.1) and the issue still occurred, until I manually installed the required Microsoft Visual C++ redistributable.
So it appears older VMware Tools do not include the required Microsoft Visual C++ redistributable, while newer ones do?

Regards,
Lewis.

[Lewpy]

As far as I know, the scope is fairly small. I will double check with the QC when I am in the office tomorrow, but my understanding is that this should only affect Windows 2008/2008R2 or earlier physical backup proxy servers backing up from ESXi 5.5. If your backup proxy is virtual and has VMware Tools installed, or runs Windows Server 2012 or later, or you are not yet on vSphere 5.5, then you should not be affected.

I can confirm that the issue exists on a Windows 2008 R2 SP1 virtual server on ESX 4.1, running VMware Tools v8.3.2.1593
This VM did not have the required Microsoft Visual C++ Redistributable installed until it was done manually.

[Gostev]

Yes, may be only newer ones.

[daphnis]

I also had the same issue in my lab running B&R R2 patch on a Server 2008 R2 box until I installed the C++ 2008 SP1 binaries.

[Gostev]

We've updated R2 download with the newer build that includes the fix for this issue, so that newly updating users do not run into this. However, installing C++ binaries manually on backup proxy will also remain a supported way to resolve this.

[daphnis]

Thanks, Gostev.

[philscompany]

Hi,

Unfortunately my support tech has stopped responding to me on this one (case 00484089) so I'm hoping that someone can assist!

I have a VMware 5.1 single host environment with hourly replicas on a second host and nightly backups to a NAS. Until recently I had a couple of SureBackup jobs which tested the replicas and backups and it worked very well. Since upgrading to R2 however, they have not worked at all, they fail every time with this error:

Code: Select all

03/12/2013 04:00:06          Getting virtual lab configuration
03/12/2013 04:01:55          Starting virtual lab routing engine
03/12/2013 04:02:09          DC01 - Publishing
03/12/2013 04:02:15          DC01 - Reconfiguring
03/12/2013 04:03:09 Error    DC01 - Configuring DC
03/12/2013 04:03:09 Error    Error: Client error: Failed to open VDDK disk [[Replication Datastore] DC01_replica/DC01-000007.vmdk] ( is read-only mode - [false] )
                             Dynamic library cannot be loaded. Path to library: [C:\Program Files (x86)\Veeam\Backup Transport\x64\vddk_5_5\vixDiskLib.dll].
                             Failed to initialize VDDK library. Installation folder: [].
                             Logon attempt with parameters [VC/ESX: [vmhost02];Port: 902;Login: [****];VMX Spec: [moref=1571];Snapshot mor: [];Transports: [nbd];Read Only: [false]] failed because of the following err
03/12/2013 04:03:10          DC01 - Unpublishing
03/12/2013 04:03:11          Stopping virtual lab routing engine
03/12/2013 04:03:14 Error    Error: Client error: Failed to open VDDK disk [[Replication Datastore] DC01_replica/DC01-000007.vmdk] ( is read-only mode - [false] )
                             Dynamic library cannot be loaded. Path to library: [C:\Program Files (x86)\Veeam\Backup Transport\x64\vddk_5_5\vixDiskLib.dll].
                             Failed to initialize VDDK library. Installation folder: [].
                             Logon attempt with parameters [VC/ESX: [vmhost02];Port: 902;Login: [****];VMX Spec: [moref=1571];Snapshot mor: [];Transports: [nbd];Read Only: [false]] failed because of the following err
03/12/2013 04:03:21          Job finished
03/12/2013 04:03:22          Sending email report

I was asked to check for accumulated snapshots on the source host VMs (there were none) and to install Microsoft Visual C++ 2008 Redistributable 2008 x64 9.0.30729.4148 - I did this and it made no difference.

Is there anything else I can try?

Thanks,

Phil

[philscompany]

The issue's been fixed, just after I posted I got a phone call from Dmitry at Veeam, he took remote control via a Webex session and replaced two files in C:\Program Files (x86)\Veeam\Backup Transport\x64\vddk_5_5 on the Veeam backup server:

Microsoft.VC90.CRT.manifest
libxml2.dll

These files should have been replaced during the R2 update but had not changed.

I'm rerunning a Surebackup job now, the VMs now boot and it has moved well past the point of failure experienced before so looks like this is fixed.

Thought this might be useful if anyone else gets this problem, thanks to Dmitry for sorting this.

[pallingd]

Just for general information: This is still an issue with v/8 Patch #1: VI Client has to be installed for Veeam to work. OK, Veeam can probably be brought to work in other ways, but installing the VI Client is a quick, simple and completely viable solution for most Veeam administrators.

[foggy]

Looks strange, I would recommend contacting support to verify this.

[Didi7]

Well, the SideBySide error happens here as well. Here I have freshly installed Windows 2008 R2 SP1 with Veeam B&R 9.0.0.1491 and all Windows Updates included. Starting an Exchange restore from a disk-based-backup results in ...

Code: Select all

Activation context generation failed for "C:\ProgramData\Veeam\Backup\ExchangeExplorer\ESE\V14\14.3.224.1\ese.dll". Dependent Assembly Microsoft.VC90.CRT,processorArchitecture="amd64",publicKeyToken="1fc8b3b9a1e18e3b",type="win32",version="9.0.21022.8" could not be found. Please use sxstrace.exe for detailed diagnosis.

Veeam.Exchange.Explorer_xxx.log says:
Loading ESE library: C:\ProgramData\Veeam\Backup\ExchangeExplorer\ESE\V14\14.3.224.1\ese.dll...
Module file version: 14.03.0224.001
Error: Failed to load JET library from: C:\ProgramData\Veeam\Backup\ExchangeExplorer\ESE\V14\14.3.224.1\ese.dll
Error: The application has failed to start because its side-by-side configuration is incorrect. Please see the application event log or use the command-line sxstrace.exe tool for more detail. (Exception from HRESULT: 0x800736B1)

So, maybe this bug is present in v9 as well?

Regards,
Didi7

[Didi7]

Here is a snippet from the VBR-error ...



Anybody with the same problem?

Is this a Visual C++ Redistributable Problem?

Regards,
Didi7

[Didi7]

I would like to return to this as well?

Has anybody VBR 9.0.0.1491 running on a fresh Windows Server 2008R2 SP1 machine and does backups of Exchange Server and get the same error, when trying to make a restore with the Veeam Explorer for Microsoft Exchange?

Regards,
Didi7

[Didi7]

I would like to return to this as well?

Has anybody VBR 9.0.0.1491 running on a fresh Windows Server 2008R2 SP1 machine with all regular and recommended Windows Updates and does backups of Exchange Server and get the same error, when trying to make a restore with the Veeam Explorer for Microsoft Exchange?

I suspect, I need to install another Visual C++ Redistributable?

Regards,
Didi7

[Vitaliy S.]

Since there are no replies, it seems like this is not widely seen among our existing customers, might be something specific to this deployment. What does our support team say on this issue?