Discussions specific to the Microsoft Hyper-V hypervisor
Post Reply
JCauthen
Novice
Posts: 6
Liked: never
Joined: May 12, 2015 1:31 pm
Full Name: Joshua Cauthen
Contact:

9.5 Broke all of my CSV jobs

Post by JCauthen » Nov 18, 2016 2:53 am

Upgrade to 9.5 today and now all jobs are failing with the three messages below for each CSV.

11/17/16 9:22:23 PM :: Task failed. Failed to expand object C:\ClusterStorage\Volume12. Error: Cluster volume is no longer available.
11/17/16 9:22:23 PM :: Network traffic verification detected no corrupted blocks
11/17/16 9:22:23 PM :: Processing finished with errors at 11/17/16 9:22:23 PM

The main job status reports all of the VMs contained in that job's CSV "is no longer processed by this job. Make sure this change is intentional"

It appears that Jobs that backup by VM vs. CSV are going to run correctly.

Some report
11/17/16 9:34:45 PM :: Error: Cannot determine extent for storage ID GUID in scale-out backup repository Scale-out Backup Repository 1
11/17/16 9:34:48 PM :: Job finished with error at 11/17/16 9:34:48 PM

Other than installing 9.5. the only change to the system was connecting to my Nimble. This has now been deleted.

I did rescan my scale-out repository before the first retries, to no avail.

OPEN CASE ID: 01979673

JCauthen
Novice
Posts: 6
Liked: never
Joined: May 12, 2015 1:31 pm
Full Name: Joshua Cauthen
Contact:

Re: 9.5 Broke all of my CSV jobs

Post by JCauthen » Nov 18, 2016 2:56 am

> It appears that Jobs that backup by VM vs. CSV are going to run correctly.

3/4 ran without error. The third failed the same way. No errors from the Vmware job.

foggy
Veeam Software
Posts: 18028
Liked: 1532 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: 9.5 Broke all of my CSV jobs

Post by foggy » Nov 18, 2016 3:30 pm

Hi Joshua, we weren't able to reproduce this "out of the box", so please continue working with your support engineer on this issue.

Gostev
SVP, Product Management
Posts: 24445
Liked: 3410 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: 9.5 Broke all of my CSV jobs

Post by Gostev » Nov 21, 2016 1:23 pm

From the logs it looks like you may not have completed the upgrade fully, and your Hyper-V hosts are still running Hyper-V Intergration components from the previous Veeam Backup & Replication version. Did you follow the Upgrade wizard fully (it should pop up when you first open the console after upgrade)?

Line_Rider
Influencer
Posts: 21
Liked: 3 times
Joined: Mar 10, 2016 7:05 pm
Full Name: Jon
Contact:

Re: 9.5 Broke all of my CSV jobs

Post by Line_Rider » Nov 21, 2016 11:18 pm

Same thing happened to me after upgrading today. I did complete the upgrade wizard after opening the console and all fields showed success.

Creating a new job and selecting a CSV volume was successful so I just created new backup jobs and mapped them to the corresponding backup repositories. I didn't have many jobs so it only took me about 10 minutes and backups of CSV's are running again.

Hyper-V Cluster 2012 R2 with all patches current, my integration services on the vm's are current too.

gairys
Influencer
Posts: 11
Liked: 3 times
Joined: Nov 24, 2014 4:27 pm
Full Name: Gairy Spiers
Contact:

Re: 9.5 Broke all of my CSV jobs

Post by gairys » Nov 22, 2016 1:26 pm

I finished upgrading all of my Veeam servers to 9.5 yesterday and I'm having the same issue as well. All backup jobs were configured based on the CSVs for each cluster. All HV servers were added to Veeam as a cluster instead of stand-alone or SCVMM. Working on resolving the issue now.

Another strange issue I'm having is one of the HV clusters is sitting at the "Collecting disks and volumes info" screen for over 30 minutes now, yet I cannot find any logs as to what is happening. Not sure if this is normal as all of the other clusters updated their information within 3-5 minutes. Not sure if this is related, but it is observed.

gairys
Influencer
Posts: 11
Liked: 3 times
Joined: Nov 24, 2014 4:27 pm
Full Name: Gairy Spiers
Contact:

Re: 9.5 Broke all of my CSV jobs

Post by gairys » Nov 22, 2016 2:15 pm

Update - Case Number 01983152

JCauthen
Novice
Posts: 6
Liked: never
Joined: May 12, 2015 1:31 pm
Full Name: Joshua Cauthen
Contact:

Re: 9.5 Broke all of my CSV jobs

Post by JCauthen » Nov 22, 2016 4:30 pm

Support has collected logs and also take a copy of my database for lab testing.

SQL Backup via Studio:
https://www.veeam.com/kb1471

5th Logging Level:
https://www.veeam.com/kb1247

They came back and asked me to create a temp database to test with.

To temporarily use a new database -
stop any veeam jobs
stop the veeam backup service
go to the registry HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication
for SqlDatabaseName fill in a new name like VeeamBackup_test and start the veeam backup service
When you open the console , you can add your cluster to the veeam db and create a dummy job with csv volumes as a source to see if we get the same result or not.
After the test you can stop the veeam backup service, fill in the original database name in the registry and start the service again.

Then today they had me edit my existing jobs to re-add my CSVs to each job. So I took the CSVs for that job out, added a different one, saved it, then re-edited and put the originals back in and saved again.

After this I am so far able to get the jobs to run. I'm only testing 1 out of the 6 and I'm going to let the rest run tonight on schedule.

gairys
Influencer
Posts: 11
Liked: 3 times
Joined: Nov 24, 2014 4:27 pm
Full Name: Gairy Spiers
Contact:

Re: 9.5 Broke all of my CSV jobs

Post by gairys » Nov 22, 2016 5:38 pm 2 people like this post

I had the same solution from support. Support explained that enumeration of the CSVs has changed, and this is a known bug. You can open the job, remove the CSV, wait till the space is calculated as 0.0 KB, then re-add the CSV. I have tested with two jobs and it has worked so far. And I am waiting for the next scheduled run for the rest like JCauthen.

JSi
Influencer
Posts: 17
Liked: 2 times
Joined: Oct 23, 2013 9:02 am
Full Name: Jan Simko
Contact:

Re: 9.5 Broke all of my CSV jobs

Post by JSi » Nov 30, 2016 8:21 am

Same problem here. Resolved by gairys solution. Thanx.

Post Reply

Who is online

Users browsing this forum: No registered users and 15 guests