Comprehensive data protection for all workloads
Bunce
Expert
Posts: 259
Liked: 8 times
Joined: Sep 18, 2009 9:56 am
Full Name: Andrew
Location: Adelaide, Australia
Contact:

Replica Jobs Failing after V7 upgrade

Post by Bunce » Aug 18, 2013 9:26 am

Just wondering if anyone else is encountering the following after V7 upgrade:

Code: Select all

18/08/2013 6:43:05 PM :: Failed to create processing task for VM MyVMName Error: Object reference not set to an instance of an object.
Case: 00430381

Pertinent Details:
  • Error occurs across all replication jobs - all using HotAdd mode. All backups jobs are running ok.
  • Error occurs across jobs for VM's within the same VCenter and across different VCenters
  • Have fully recreated one of the jobs - still receive same error
  • Have reentered credentials for VC and Guest jobs
  • Proxies were upgraded during install
  • Proxies in questions also happen to have full VBR console installed at the other end (to do local jobs)
  • All components have been restarted, and connectivity between all sites tested
  • No other changes in network or infrastructure at all
  • Couldn't see anything in the release notes or KB that appear related, except for hitting 'recalculate' to refresh the cache.
All jobs are using seeds and disk exclusions which I noticed weren't bought across correctly from the upgrade - ie all disks were selected after the upgrade (only selected disks were included prior), however fixing this doesn't rectify the error.

Drilling into the job logs doesn't seem to show much from a cursory look - the above error just appears out of nowhere.

I remember something sort of similar during a previous upgrade that required revoking and reapplying host licenses but all seems to be ok on this occasion.

Cheers,
Andrew

averylarry
Expert
Posts: 264
Liked: 28 times
Joined: Mar 22, 2011 7:43 pm
Full Name: Ted
Contact:

Re: Replica Jobs Failing after V7 upgrade

Post by averylarry » Aug 21, 2013 3:04 am

There is a confirmed small bug with replication and VMs with excluded disks. This may be your issue.

Case # 00429720

paraphrased from tech support:
"We seem to have a small release v7 bug with replications with excluded disks. If you wish to workaround this issue while R&D determines next steps please add thin provisioned disks on your target replicas matching the size of the source VMDKs and map your existing replication job to the replica VM with the empty extra disks We will not fill these disks with data, but it will allow you to replicate in the meantime."

I made the changes to my replica VMs, and replication is working again.

Bunce
Expert
Posts: 259
Liked: 8 times
Joined: Sep 18, 2009 9:56 am
Full Name: Andrew
Location: Adelaide, Australia
Contact:

Re: Replica Jobs Failing after V7 upgrade

Post by Bunce » Aug 21, 2013 5:34 am

That did the trick - thanks.

Bunce
Expert
Posts: 259
Liked: 8 times
Joined: Sep 18, 2009 9:56 am
Full Name: Andrew
Location: Adelaide, Australia
Contact:

Re: Replica Jobs Failing after V7 upgrade

Post by Bunce » Aug 21, 2013 6:26 am

Well - it worked for the first run after adding the disks, however the job removed those disks after the successful run, so it failed on the next run..

So this means we effectively have to re-add the disks prior to every job run, to get the failing replication jobs to work?

Replica jobs don't have the 'Remove excluded disks from VM configuration' option like backup jobs do, which could have been handy.

Where are the powershell guru's.. :P

Gostev
SVP, Product Management
Posts: 25492
Liked: 3868 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Replica Jobs Failing after V7 upgrade

Post by Gostev » Aug 21, 2013 9:05 am

I've seen a build notification with this issue listed, so presumably you will be getting a hotfix soon, after its testing completes.

averylarry
Expert
Posts: 264
Liked: 28 times
Joined: Mar 22, 2011 7:43 pm
Full Name: Ted
Contact:

Re: Replica Jobs Failing after V7 upgrade

Post by averylarry » Aug 21, 2013 4:18 pm

I've been given a fix for this issue (new veeam.backup.core.dll). I haven't tested it yet (waiting for a replication to finish). Nor have I tried a 2nd replication after the 1st worked.

averylarry
Expert
Posts: 264
Liked: 28 times
Joined: Mar 22, 2011 7:43 pm
Full Name: Ted
Contact:

Re: Replica Jobs Failing after V7 upgrade

Post by averylarry » Aug 21, 2013 4:30 pm

Also note -- this failure can be triggered if you replicate your backup proxy VMs and use hotadd. The foreign disks will cause the replication to fail if it tries to replicate the proxy VM while it's running a job.

odge
Veeam ProPartner
Posts: 76
Liked: 7 times
Joined: Apr 14, 2011 3:20 pm
Full Name: Matthew Ogden
Contact:

Re: Replica Jobs Failing after V7 upgrade

Post by odge » Aug 22, 2013 12:20 pm

we've only upgraded one customer so far, there are 2 replication jobs (for the same VM) that exclude one of its disks. The job seems to be working fine.

What are the causes for disk exclusion not to work?
(we have another site that is using it quite heavily, so we are not sure if its going to work or not)

averylarry
Expert
Posts: 264
Liked: 28 times
Joined: Mar 22, 2011 7:43 pm
Full Name: Ted
Contact:

Re: Replica Jobs Failing after V7 upgrade

Post by averylarry » Aug 22, 2013 4:44 pm

The first fix didn't quite work with a standalone vmware host as a destination. A 2nd fix from today so far is working.

odge -- Initial v.7 relase code replications simply fail any VM that has an excluded disk (replications only, not backups). Other VMs (even in the same job) can work just fine (at least in reference to this specific issue).
My educated speculation leads me to believe that the process to check for an accurate replication destination VM (whether a seed or an existing replicated VM) runs before checking for excluded disks. Therefore the job is looking for the excluded disk on the replica VM, which (obviously) does not exist, and subsequently fails thinking that it's not actually a useable replica of the VM it's trying to process. I suspect that Veeam just needed to calculate the replica VMs configuration before comparing to verify a valid destination.

Bunce -- I can confirm the workaround of adding thin disks to the destination replica VM will only work one time. Of course, you can add them again after that and get another run to go. (Or get the fix from tech support.) ;-)

odge
Veeam ProPartner
Posts: 76
Liked: 7 times
Joined: Apr 14, 2011 3:20 pm
Full Name: Matthew Ogden
Contact:

Re: Replica Jobs Failing after V7 upgrade

Post by odge » Aug 22, 2013 5:20 pm

thanks averylarry,

there must be one other variable, because I'm looking at my v7 job, and its run more than once with

2013/08/20 11:42:58 PM :: Releasing guest
2013/08/20 11:43:29 PM :: Disk '[Perf-iSCSI-3] db-server/db-server.vmdk' has been skipped because it was excluded from processing by user
2013/08/20 11:43:43 PM :: Processing configuration

2013/08/21 11:25:10 PM :: Releasing guest
2013/08/21 11:25:38 PM :: Disk '[Perf-iSCSI-3] db-server/db-server.vmdk' has been skipped because it was excluded from processing by user
2013/08/21 11:25:44 PM :: Preparing replica VM

Eventually gets to success (i see as well, that the output wasn't quite the same, maybe tonight is going to fail?

Bunce
Expert
Posts: 259
Liked: 8 times
Joined: Sep 18, 2009 9:56 am
Full Name: Andrew
Location: Adelaide, Australia
Contact:

Re: Replica Jobs Failing after V7 upgrade

Post by Bunce » Aug 22, 2013 9:03 pm

Replica Mapping?
Hot add?

Fix looks to have worked on some jobs but not others for us. Difference between them looks to be the failed jobs use destinations in a different VCenter.

I'm not sure if we were given the first or second patch though, so will follow up.

odge
Veeam ProPartner
Posts: 76
Liked: 7 times
Joined: Apr 14, 2011 3:20 pm
Full Name: Matthew Ogden
Contact:

Re: Replica Jobs Failing after V7 upgrade

Post by odge » Aug 23, 2013 7:56 am

Just as a note, we dont have replica mapping on these jobs. They do use hotadd on the target (and SAN on the source). They are in the same VCenter

Bunce
Expert
Posts: 259
Liked: 8 times
Joined: Sep 18, 2009 9:56 am
Full Name: Andrew
Location: Adelaide, Australia
Contact:

Re: Replica Jobs Failing after V7 upgrade

Post by Bunce » Aug 23, 2013 8:34 am

Most likely the mapping then.

2nd revision of patch looks to have fixed it for us.

averylarry
Expert
Posts: 264
Liked: 28 times
Joined: Mar 22, 2011 7:43 pm
Full Name: Ted
Contact:

Re: Replica Jobs Failing after V7 upgrade

Post by averylarry » Aug 23, 2013 3:20 pm

All of my jobs were successful overnight. So -- yeah for me!! ;-)

terrance101
Enthusiast
Posts: 33
Liked: 2 times
Joined: May 17, 2013 7:59 am
Contact:

[MERGED] still trialing veeam 7 and had some queries

Post by terrance101 » Sep 04, 2013 11:27 am

Hi guys,

I am still doing the trial for veeam 7 and I have an issue after deleting the a job then recreating it utilising the current replica as a seed. It comes up with the error "Task failed Error: Object reference not set to an instance of an object." when I try run the new job. I went down the process of deleting the snapshots off the job and still had issues ultimately when this happened the last time. (These are my sql server replications and I am excluding the backup drive and apparently this is a known bug with veeam ver 7).
Since I am trialing the software, do I have the right to raise a ticket for support on this?
Or does anyone have a patch I could try for 7 to see if it helps?

foggy
Veeam Software
Posts: 18649
Liked: 1628 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: still trialing veeam 7 and had some queries

Post by foggy » Sep 04, 2013 11:34 am

Everyone on the official evaluation period is eligible for technical support, so feel free to contact them with this issue.

terrance101
Enthusiast
Posts: 33
Liked: 2 times
Joined: May 17, 2013 7:59 am
Contact:

Re: still trialing veeam 7 and had some queries

Post by terrance101 » Sep 04, 2013 1:06 pm

ticket submitted. Thinking it's probably going to be a patch thing so hopefully... holding thumbs :)

foggy
Veeam Software
Posts: 18649
Liked: 1628 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: still trialing veeam 7 and had some queries

Post by foggy » Sep 04, 2013 1:11 pm

I would appreciate if you could update the thread with the case ID number. Thanks!

terrance101
Enthusiast
Posts: 33
Liked: 2 times
Joined: May 17, 2013 7:59 am
Contact:

Re: still trialing veeam 7 and had some queries

Post by terrance101 » Sep 04, 2013 2:49 pm

ID#00439091

terrance101
Enthusiast
Posts: 33
Liked: 2 times
Joined: May 17, 2013 7:59 am
Contact:

Re: still trialing veeam 7 and had some queries

Post by terrance101 » Sep 10, 2013 11:29 am 1 person likes this post

Hey guys, as thought, and for all those interested, it was caused by jobs that do not replicate all disks. The veeam team gave me the updated dll required to fix this and voila it's resolved. So yeah a bug in the current version.

foggy
Veeam Software
Posts: 18649
Liked: 1628 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: still trialing veeam 7 and had some queries

Post by foggy » Sep 10, 2013 12:09 pm

Thanks for the update, much appreciated. Yes, there is a known issue where replica mapping could fail if the VM has some disks excluded. Currently support has a private fix available for that and the issue will be also fixed in the upcoming patch. I'm now going to merge this thread to an older existing discussion regarding that problem. Thanks.

mfc
Influencer
Posts: 14
Liked: never
Joined: Sep 17, 2013 12:48 pm
Full Name: Mike Czajkowski
Contact:

Re: Replica Jobs Failing after V7 upgrade

Post by mfc » Sep 18, 2013 6:01 pm

Foggy,

I have a case open on this exact issue (Case #00447045), the tech assigned didn't mention the private fix as an available option. Any chance you could take a look at this case and see if that would be an option? As of right now I'm told the only workaround is to abandon the seed and run a full replication without the disk exclusion, two things I'd like to avoid unless absolutely necessary.

foggy
Veeam Software
Posts: 18649
Liked: 1628 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Replica Jobs Failing after V7 upgrade

Post by foggy » Sep 19, 2013 7:15 am

Mike, I will check on that.

mfc
Influencer
Posts: 14
Liked: never
Joined: Sep 17, 2013 12:48 pm
Full Name: Mike Czajkowski
Contact:

Re: Replica Jobs Failing after V7 upgrade

Post by mfc » Sep 19, 2013 7:34 pm

foggy wrote:Mike, I will check on that.
The hotfix corrected the problem, thanks for your help.

daniel.negru
Influencer
Posts: 17
Liked: 1 time
Joined: Feb 13, 2013 5:36 pm
Full Name: Daniel Negru
Contact:

Re: Replica Jobs Failing after V7 upgrade

Post by daniel.negru » Sep 20, 2013 1:20 pm 1 person likes this post

Hi everyone,

From you guys and gals receiving the hot fix DLL, did any of you test their guest replica after?
I advice you to do so.
I had a huge surprise, not sure if the v7 upgrade or the hot fix is at fault, but it used to work before.

Replicated win2k3, Disk 0:0 having 3 partitions leaving WSUS repository Disk 0:1 out of it.
The replica seems to contain only partition 0 and 1, leaving the windows install partition (number 2) out of it and hence not booting up. It may be a combination of my setup with the P2V guest and veeam, but I urge everyone to test it out.

Lesson learned! Upgrade or pathch: test after like it would be a new product. Veeam cannot replicate all the possible scenarios in the field.

Daniel.

averylarry
Expert
Posts: 264
Liked: 28 times
Joined: Mar 22, 2011 7:43 pm
Full Name: Ted
Contact:

Re: Replica Jobs Failing after V7 upgrade

Post by averylarry » Oct 03, 2013 4:20 pm

Does anyone know if Patch 1 includes this hotfix?

Gostev
SVP, Product Management
Posts: 25492
Liked: 3868 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Replica Jobs Failing after V7 upgrade

Post by Gostev » Oct 03, 2013 7:17 pm

Yes. But it was mistakenly listed under Hyper-V in patch notes... moved to VMware.

KevinBeaumont
Influencer
Posts: 12
Liked: never
Joined: Apr 28, 2011 3:04 pm
Full Name: Kevin Beaumont
Contact:

Re: Replica Jobs Failing after V7 upgrade

Post by KevinBeaumont » Dec 02, 2013 4:24 pm

Folks - does anybody know if this hotfix is still in v7 R2?

We're running into (potentially!!) the same issue. We never had the hotfix installed on plain v7, but we did upgrade to R2.

Our replication job fails with:

Code: Select all

27.11.2013 22:23:15] <01> Error    Task failed
[27.11.2013 22:23:15] <01> Error    Object reference not set to an instance of an object. (System.NullReferenceException)
[27.11.2013 22:23:15] <01> Error       at Veeam.Backup.Core.CSnapReplicaVmSpec.FindDiskDatastore(CVirtualDisk disk)
[27.11.2013 22:23:15] <01> Error       at Veeam.Backup.Core.CSnapReplicaVmSpec.GetDiskDatastore(CVirtualDisk disk)
[27.11.2013 22:23:15] <01> Error       at System.Linq.Enumerable.WhereSelectArrayIterator`2.MoveNext()
[27.11.2013 22:23:15] <01> Error       at System.Linq.Buffer`1..ctor(IEnumerable`1 source)
[27.11.2013 22:23:15] <01> Error       at System.Linq.Enumerable.ToArray[TSource](IEnumerable`1 source)
[27.11.2013 22:23:15] <01> Error       at Veeam.Backup.Core.CViTaskBuilder.GetReplicaTargetProxies(IVmSpec vmSpec, CVm vm, CHierarchyCache cache, CDiskFilter diskFilter)
[27.11.2013 22:23:15] <01> Error       at Veeam.Backup.Core.CViTaskBuilder.CreateDisksResourceData(CBackupTaskSession taskSession, VmEntry vmEntry, IVmSpec vmSpec, CDiskFilter diskFilter, CHierarchyCache cache, Boolean& useSanSnapshot)
[27.11.2013 22:23:15] <01> Error       at Veeam.Backup.Core.CViTaskBuilder.CreateViVmTasks(CViTaskBuilderEntry entry, Guid vappId)

Gostev
SVP, Product Management
Posts: 25492
Liked: 3868 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Replica Jobs Failing after V7 upgrade

Post by Gostev » Dec 02, 2013 7:51 pm

Yes, our patches are cumulative, so anything in Patch #1 is also included in R2.

foggy
Veeam Software
Posts: 18649
Liked: 1628 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Replica Jobs Failing after V7 upgrade

Post by foggy » Dec 03, 2013 6:13 am

Probably this one is a different issue. Better verify that with support.

Post Reply

Who is online

Users browsing this forum: aj43567, eternal_noob, foggy, Gostev, tsightler, zahradnik and 76 guests