Replica Jobs Failing after V7 upgrade

[Bunce]

Just wondering if anyone else is encountering the following after V7 upgrade:

Code: Select all

18/08/2013 6:43:05 PM :: Failed to create processing task for VM MyVMName Error: Object reference not set to an instance of an object.

Case: 00430381

Pertinent Details:

• Error occurs across all replication jobs - all using HotAdd mode. All backups jobs are running ok.
• Error occurs across jobs for VM's within the same VCenter and across different VCenters
• Have fully recreated one of the jobs - still receive same error
• Have reentered credentials for VC and Guest jobs
• Proxies were upgraded during install
• Proxies in questions also happen to have full VBR console installed at the other end (to do local jobs)
• All components have been restarted, and connectivity between all sites tested
• No other changes in network or infrastructure at all
• Couldn't see anything in the release notes or KB that appear related, except for hitting 'recalculate' to refresh the cache.

All jobs are using seeds and disk exclusions which I noticed weren't bought across correctly from the upgrade - ie all disks were selected after the upgrade (only selected disks were included prior), however fixing this doesn't rectify the error.

Drilling into the job logs doesn't seem to show much from a cursory look - the above error just appears out of nowhere.

I remember something sort of similar during a previous upgrade that required revoking and reapplying host licenses but all seems to be ok on this occasion.

Cheers,
Andrew

[averylarry]

There is a confirmed small bug with replication and VMs with excluded disks. This may be your issue.

Case # 00429720

paraphrased from tech support:
"We seem to have a small release v7 bug with replications with excluded disks. If you wish to workaround this issue while R&D determines next steps please add thin provisioned disks on your target replicas matching the size of the source VMDKs and map your existing replication job to the replica VM with the empty extra disks We will not fill these disks with data, but it will allow you to replicate in the meantime."

I made the changes to my replica VMs, and replication is working again.

[Bunce]

That did the trick - thanks.

[Bunce]

Well - it worked for the first run after adding the disks, however the job removed those disks after the successful run, so it failed on the next run..

So this means we effectively have to re-add the disks prior to every job run, to get the failing replication jobs to work?

Replica jobs don't have the 'Remove excluded disks from VM configuration' option like backup jobs do, which could have been handy.

Where are the powershell guru's..

[Gostev]

I've seen a build notification with this issue listed, so presumably you will be getting a hotfix soon, after its testing completes.

[averylarry]

I've been given a fix for this issue (new veeam.backup.core.dll). I haven't tested it yet (waiting for a replication to finish). Nor have I tried a 2nd replication after the 1st worked.

[averylarry]

Also note -- this failure can be triggered if you replicate your backup proxy VMs and use hotadd. The foreign disks will cause the replication to fail if it tries to replicate the proxy VM while it's running a job.

[odge]

we've only upgraded one customer so far, there are 2 replication jobs (for the same VM) that exclude one of its disks. The job seems to be working fine.

What are the causes for disk exclusion not to work?
(we have another site that is using it quite heavily, so we are not sure if its going to work or not)

[averylarry]

The first fix didn't quite work with a standalone vmware host as a destination. A 2nd fix from today so far is working.

odge -- Initial v.7 relase code replications simply fail any VM that has an excluded disk (replications only, not backups). Other VMs (even in the same job) can work just fine (at least in reference to this specific issue).
My educated speculation leads me to believe that the process to check for an accurate replication destination VM (whether a seed or an existing replicated VM) runs before checking for excluded disks. Therefore the job is looking for the excluded disk on the replica VM, which (obviously) does not exist, and subsequently fails thinking that it's not actually a useable replica of the VM it's trying to process. I suspect that Veeam just needed to calculate the replica VMs configuration before comparing to verify a valid destination.

Bunce -- I can confirm the workaround of adding thin disks to the destination replica VM will only work one time. Of course, you can add them again after that and get another run to go. (Or get the fix from tech support.)

[odge]

thanks averylarry,

there must be one other variable, because I'm looking at my v7 job, and its run more than once with

2013/08/20 11:42:58 PM :: Releasing guest
2013/08/20 11:43:29 PM :: Disk '[Perf-iSCSI-3] db-server/db-server.vmdk' has been skipped because it was excluded from processing by user
2013/08/20 11:43:43 PM :: Processing configuration

2013/08/21 11:25:10 PM :: Releasing guest
2013/08/21 11:25:38 PM :: Disk '[Perf-iSCSI-3] db-server/db-server.vmdk' has been skipped because it was excluded from processing by user
2013/08/21 11:25:44 PM :: Preparing replica VM

Eventually gets to success (i see as well, that the output wasn't quite the same, maybe tonight is going to fail?

[Bunce]

Replica Mapping?
Hot add?

Fix looks to have worked on some jobs but not others for us. Difference between them looks to be the failed jobs use destinations in a different VCenter.

I'm not sure if we were given the first or second patch though, so will follow up.

[odge]

Just as a note, we dont have replica mapping on these jobs. They do use hotadd on the target (and SAN on the source). They are in the same VCenter

[Bunce]

Most likely the mapping then.

2nd revision of patch looks to have fixed it for us.

[averylarry]

All of my jobs were successful overnight. So -- yeah for me!!

[terrance101]

Hi guys,

I am still doing the trial for veeam 7 and I have an issue after deleting the a job then recreating it utilising the current replica as a seed. It comes up with the error "Task failed Error: Object reference not set to an instance of an object." when I try run the new job. I went down the process of deleting the snapshots off the job and still had issues ultimately when this happened the last time. (These are my sql server replications and I am excluding the backup drive and apparently this is a known bug with veeam ver 7).
Since I am trialing the software, do I have the right to raise a ticket for support on this?
Or does anyone have a patch I could try for 7 to see if it helps?

[foggy]

Everyone on the official evaluation period is eligible for technical support, so feel free to contact them with this issue.

[terrance101]

ticket submitted. Thinking it's probably going to be a patch thing so hopefully... holding thumbs

[foggy]

I would appreciate if you could update the thread with the case ID number. Thanks!

[terrance101]

ID#00439091

[terrance101]

Hey guys, as thought, and for all those interested, it was caused by jobs that do not replicate all disks. The veeam team gave me the updated dll required to fix this and voila it's resolved. So yeah a bug in the current version.

[foggy]

Thanks for the update, much appreciated. Yes, there is a known issue where replica mapping could fail if the VM has some disks excluded. Currently support has a private fix available for that and the issue will be also fixed in the upcoming patch. I'm now going to merge this thread to an older existing discussion regarding that problem. Thanks.

[mfc]

Foggy,

I have a case open on this exact issue (Case #00447045), the tech assigned didn't mention the private fix as an available option. Any chance you could take a look at this case and see if that would be an option? As of right now I'm told the only workaround is to abandon the seed and run a full replication without the disk exclusion, two things I'd like to avoid unless absolutely necessary.

[foggy]

Mike, I will check on that.

[mfc]

Mike, I will check on that.

The hotfix corrected the problem, thanks for your help.

[daniel.negru]

Hi everyone,

From you guys and gals receiving the hot fix DLL, did any of you test their guest replica after?
I advice you to do so.
I had a huge surprise, not sure if the v7 upgrade or the hot fix is at fault, but it used to work before.

Replicated win2k3, Disk 0:0 having 3 partitions leaving WSUS repository Disk 0:1 out of it.
The replica seems to contain only partition 0 and 1, leaving the windows install partition (number 2) out of it and hence not booting up. It may be a combination of my setup with the P2V guest and veeam, but I urge everyone to test it out.

Lesson learned! Upgrade or pathch: test after like it would be a new product. Veeam cannot replicate all the possible scenarios in the field.

Daniel.

[averylarry]

Does anyone know if Patch 1 includes this hotfix?

[Gostev]

Yes. But it was mistakenly listed under Hyper-V in patch notes... moved to VMware.

[KevinBeaumont]

Folks - does anybody know if this hotfix is still in v7 R2?

We're running into (potentially!!) the same issue. We never had the hotfix installed on plain v7, but we did upgrade to R2.

Our replication job fails with:

Code: Select all

27.11.2013 22:23:15] <01> Error    Task failed
[27.11.2013 22:23:15] <01> Error    Object reference not set to an instance of an object. (System.NullReferenceException)
[27.11.2013 22:23:15] <01> Error       at Veeam.Backup.Core.CSnapReplicaVmSpec.FindDiskDatastore(CVirtualDisk disk)
[27.11.2013 22:23:15] <01> Error       at Veeam.Backup.Core.CSnapReplicaVmSpec.GetDiskDatastore(CVirtualDisk disk)
[27.11.2013 22:23:15] <01> Error       at System.Linq.Enumerable.WhereSelectArrayIterator`2.MoveNext()
[27.11.2013 22:23:15] <01> Error       at System.Linq.Buffer`1..ctor(IEnumerable`1 source)
[27.11.2013 22:23:15] <01> Error       at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)
[27.11.2013 22:23:15] <01> Error       at Veeam.Backup.Core.CViTaskBuilder.GetReplicaTargetProxies(IVmSpec vmSpec, CVm vm, CHierarchyCache cache, CDiskFilter diskFilter)
[27.11.2013 22:23:15] <01> Error       at Veeam.Backup.Core.CViTaskBuilder.CreateDisksResourceData(CBackupTaskSession taskSession, VmEntry vmEntry, IVmSpec vmSpec, CDiskFilter diskFilter, CHierarchyCache cache, Boolean& useSanSnapshot)
[27.11.2013 22:23:15] <01> Error       at Veeam.Backup.Core.CViTaskBuilder.CreateViVmTasks(CViTaskBuilderEntry entry, Guid vappId)

[Gostev]

Yes, our patches are cumulative, so anything in Patch #1 is also included in R2.

[foggy]

Probably this one is a different issue. Better verify that with support.