9.5 Broke all of my CSV jobs

[JCauthen]

Upgrade to 9.5 today and now all jobs are failing with the three messages below for each CSV.

11/17/16 9:22:23 PM :: Task failed. Failed to expand object C:\ClusterStorage\Volume12. Error: Cluster volume is no longer available.
11/17/16 9:22:23 PM :: Network traffic verification detected no corrupted blocks
11/17/16 9:22:23 PM :: Processing finished with errors at 11/17/16 9:22:23 PM

The main job status reports all of the VMs contained in that job's CSV "is no longer processed by this job. Make sure this change is intentional"

It appears that Jobs that backup by VM vs. CSV are going to run correctly.

Some report
11/17/16 9:34:45 PM :: Error: Cannot determine extent for storage ID GUID in scale-out backup repository Scale-out Backup Repository 1
11/17/16 9:34:48 PM :: Job finished with error at 11/17/16 9:34:48 PM

Other than installing 9.5. the only change to the system was connecting to my Nimble. This has now been deleted.

I did rescan my scale-out repository before the first retries, to no avail.

OPEN CASE ID: 01979673

[JCauthen]

> It appears that Jobs that backup by VM vs. CSV are going to run correctly.

3/4 ran without error. The third failed the same way. No errors from the Vmware job.

[foggy]

Hi Joshua, we weren't able to reproduce this "out of the box", so please continue working with your support engineer on this issue.

[Gostev]

From the logs it looks like you may not have completed the upgrade fully, and your Hyper-V hosts are still running Hyper-V Intergration components from the previous Veeam Backup & Replication version. Did you follow the Upgrade wizard fully (it should pop up when you first open the console after upgrade)?

[Line_Rider]

Same thing happened to me after upgrading today. I did complete the upgrade wizard after opening the console and all fields showed success.

Creating a new job and selecting a CSV volume was successful so I just created new backup jobs and mapped them to the corresponding backup repositories. I didn't have many jobs so it only took me about 10 minutes and backups of CSV's are running again.

Hyper-V Cluster 2012 R2 with all patches current, my integration services on the vm's are current too.

[gairys]

I finished upgrading all of my Veeam servers to 9.5 yesterday and I'm having the same issue as well. All backup jobs were configured based on the CSVs for each cluster. All HV servers were added to Veeam as a cluster instead of stand-alone or SCVMM. Working on resolving the issue now.

Another strange issue I'm having is one of the HV clusters is sitting at the "Collecting disks and volumes info" screen for over 30 minutes now, yet I cannot find any logs as to what is happening. Not sure if this is normal as all of the other clusters updated their information within 3-5 minutes. Not sure if this is related, but it is observed.

[gairys]

Update - Case Number 01983152

[JCauthen]

Support has collected logs and also take a copy of my database for lab testing.

SQL Backup via Studio:
https://www.veeam.com/kb1471

5th Logging Level:
https://www.veeam.com/kb1247

They came back and asked me to create a temp database to test with.

To temporarily use a new database -
stop any veeam jobs
stop the veeam backup service
go to the registry HKEY_LOCAL_MACHINE\SOFTWARE\Veeam\Veeam Backup and Replication
for SqlDatabaseName fill in a new name like VeeamBackup_test and start the veeam backup service
When you open the console , you can add your cluster to the veeam db and create a dummy job with csv volumes as a source to see if we get the same result or not.
After the test you can stop the veeam backup service, fill in the original database name in the registry and start the service again.

Then today they had me edit my existing jobs to re-add my CSVs to each job. So I took the CSVs for that job out, added a different one, saved it, then re-edited and put the originals back in and saved again.

After this I am so far able to get the jobs to run. I'm only testing 1 out of the 6 and I'm going to let the rest run tonight on schedule.

[gairys]

I had the same solution from support. Support explained that enumeration of the CSVs has changed, and this is a known bug. You can open the job, remove the CSV, wait till the space is calculated as 0.0 KB, then re-add the CSV. I have tested with two jobs and it has worked so far. And I am waiting for the next scheduled run for the rest like JCauthen.

[JSi]

Same problem here. Resolved by gairys solution. Thanx.