9.5 Broke all of my CSV jobs

Hyper-V specific discussions

9.5 Broke all of my CSV jobs

Veeam Logoby JCauthen » Fri Nov 18, 2016 2:53 am

Upgrade to 9.5 today and now all jobs are failing with the three messages below for each CSV.

11/17/16 9:22:23 PM :: Task failed. Failed to expand object C:\ClusterStorage\Volume12. Error: Cluster volume is no longer available.
11/17/16 9:22:23 PM :: Network traffic verification detected no corrupted blocks
11/17/16 9:22:23 PM :: Processing finished with errors at 11/17/16 9:22:23 PM

The main job status reports all of the VMs contained in that job's CSV "is no longer processed by this job. Make sure this change is intentional"

It appears that Jobs that backup by VM vs. CSV are going to run correctly.

Some report
11/17/16 9:34:45 PM :: Error: Cannot determine extent for storage ID GUID in scale-out backup repository Scale-out Backup Repository 1
11/17/16 9:34:48 PM :: Job finished with error at 11/17/16 9:34:48 PM

Other than installing 9.5. the only change to the system was connecting to my Nimble. This has now been deleted.

I did rescan my scale-out repository before the first retries, to no avail.

OPEN CASE ID: 01979673
JCauthen
Novice
 
Posts: 5
Liked: never
Joined: Tue May 12, 2015 1:31 pm
Full Name: Joshua Cauthen

Re: 9.5 Broke all of my CSV jobs

Veeam Logoby JCauthen » Fri Nov 18, 2016 2:56 am

> It appears that Jobs that backup by VM vs. CSV are going to run correctly.

3/4 ran without error. The third failed the same way. No errors from the Vmware job.
JCauthen
Novice
 
Posts: 5
Liked: never
Joined: Tue May 12, 2015 1:31 pm
Full Name: Joshua Cauthen

Re: 9.5 Broke all of my CSV jobs

Veeam Logoby foggy » Fri Nov 18, 2016 3:30 pm

Hi Joshua, we weren't able to reproduce this "out of the box", so please continue working with your support engineer on this issue.
foggy
Veeam Software
 
Posts: 14742
Liked: 1080 times
Joined: Mon Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson

Re: 9.5 Broke all of my CSV jobs

Veeam Logoby Gostev » Mon Nov 21, 2016 1:23 pm

From the logs it looks like you may not have completed the upgrade fully, and your Hyper-V hosts are still running Hyper-V Intergration components from the previous Veeam Backup & Replication version. Did you follow the Upgrade wizard fully (it should pop up when you first open the console after upgrade)?
Gostev
Veeam Software
 
Posts: 21390
Liked: 2349 times
Joined: Sun Jan 01, 2006 1:01 am
Location: Baar, Switzerland

Re: 9.5 Broke all of my CSV jobs

Veeam Logoby Line_Rider » Mon Nov 21, 2016 11:18 pm

Same thing happened to me after upgrading today. I did complete the upgrade wizard after opening the console and all fields showed success.

Creating a new job and selecting a CSV volume was successful so I just created new backup jobs and mapped them to the corresponding backup repositories. I didn't have many jobs so it only took me about 10 minutes and backups of CSV's are running again.

Hyper-V Cluster 2012 R2 with all patches current, my integration services on the vm's are current too.
Line_Rider
Influencer
 
Posts: 17
Liked: 3 times
Joined: Thu Mar 10, 2016 7:05 pm
Full Name: Jon

Re: 9.5 Broke all of my CSV jobs

Veeam Logoby gairys » Tue Nov 22, 2016 1:26 pm

I finished upgrading all of my Veeam servers to 9.5 yesterday and I'm having the same issue as well. All backup jobs were configured based on the CSVs for each cluster. All HV servers were added to Veeam as a cluster instead of stand-alone or SCVMM. Working on resolving the issue now.

Another strange issue I'm having is one of the HV clusters is sitting at the "Collecting disks and volumes info" screen for over 30 minutes now, yet I cannot find any logs as to what is happening. Not sure if this is normal as all of the other clusters updated their information within 3-5 minutes. Not sure if this is related, but it is observed.
gairys
Novice
 
Posts: 8
Liked: 2 times
Joined: Mon Nov 24, 2014 4:27 pm
Full Name: Gairy S

Re: 9.5 Broke all of my CSV jobs

Veeam Logoby gairys » Tue Nov 22, 2016 2:15 pm

Update - Case Number 01983152
gairys
Novice
 
Posts: 8
Liked: 2 times
Joined: Mon Nov 24, 2014 4:27 pm
Full Name: Gairy S

Re: 9.5 Broke all of my CSV jobs

Veeam Logoby JCauthen » Tue Nov 22, 2016 4:30 pm

Support has collected logs and also take a copy of my database for lab testing.

SQL Backup via Studio:
https://www.veeam.com/kb1471

5th Logging Level:
https://www.veeam.com/kb1247

They came back and asked me to create a temp database to test with.

To temporarily use a new database -
stop any veeam jobs
stop the veeam backup service
go to the registry HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication
for SqlDatabaseName fill in a new name like VeeamBackup_test and start the veeam backup service
When you open the console , you can add your cluster to the veeam db and create a dummy job with csv volumes as a source to see if we get the same result or not.
After the test you can stop the veeam backup service, fill in the original database name in the registry and start the service again.

Then today they had me edit my existing jobs to re-add my CSVs to each job. So I took the CSVs for that job out, added a different one, saved it, then re-edited and put the originals back in and saved again.

After this I am so far able to get the jobs to run. I'm only testing 1 out of the 6 and I'm going to let the rest run tonight on schedule.
JCauthen
Novice
 
Posts: 5
Liked: never
Joined: Tue May 12, 2015 1:31 pm
Full Name: Joshua Cauthen

Re: 9.5 Broke all of my CSV jobs

Veeam Logoby gairys » Tue Nov 22, 2016 5:38 pm 2 people like this post

I had the same solution from support. Support explained that enumeration of the CSVs has changed, and this is a known bug. You can open the job, remove the CSV, wait till the space is calculated as 0.0 KB, then re-add the CSV. I have tested with two jobs and it has worked so far. And I am waiting for the next scheduled run for the rest like JCauthen.
gairys
Novice
 
Posts: 8
Liked: 2 times
Joined: Mon Nov 24, 2014 4:27 pm
Full Name: Gairy S

Re: 9.5 Broke all of my CSV jobs

Veeam Logoby JSi » Wed Nov 30, 2016 8:21 am

Same problem here. Resolved by gairys solution. Thanx.
JSi
Influencer
 
Posts: 14
Liked: 1 time
Joined: Wed Oct 23, 2013 9:02 am
Full Name: Jan Simko


Return to Microsoft Hyper-V



Who is online

Users browsing this forum: No registered users and 4 guests