Veeam Backup 4.0 Upgrade Issue < READ IF UPGRADING FROM v3

[jai64]

Has anyone had any problem upgrading to v4?

I upgraded last night and only 1 of 5 backup jobs ran. The rest hung for hours @ 0%.

I restarted and rebooting the server. Then tried to run again (and new jobs), but now the are all still stuck at 0%.

It says "Repairing Backup from Previous Rollback". It is also eating space like crazy and leaving .update files.

Called support at 4:30AM. Still no word...or help.

~joe

[Gostev]

Joe, Veeam Backup 4.0 has completely new backup storage format (transactional storage) and the upgrade procedure of v3 storage is very complex. Storage upgrade happens on the first launch of the specific job after the product upgrade (this is why you see the job "hanging" at 0%). Based on our earlier investigation of couple of similar support cases, storage upgrade may be unable to update certain larger v3 backup files to v4 format.

If possible, I recommend that you create the new jobs and delete existing v3 ones for which upgrade failed, this is an immediate workaround. We are already testing fix for this issue, and support will contact you as soon as the fix will be available.

Sorry for this inconvenience, upgrade is always most complex thing to test

[Gostev]

Thread stickied.

For future readers: this is one-time upgrade issue does not affect all backups, but only some. So, if your job are running fine after upgrade, you do not need any fixes or anything.

UPDATE - IMPORTANT - PLEASE READ THIS POST CAREFULLY
The download package has been updated on http://www.veeam.com at 10pm GMT Oct 29
Current package does not have this issue.

If you have run into the above mentioned upgrade problem with the original RTM package, follow this procedure:
1. Uninstall the existing version.
2. Download and install updated package (the new package has "_u" in its name), pointing it to the same configuration database.
3. Rename or copy away the .upgrade files next to VBK files for all failed jobs.
4. Run all jobs, first run should now complete successfully.
5. If all is well, delete .upgrade files preserved on step 3.

If you have replication jobs affected by installing the original RTM package, please contact support for assistance.

If you upgraded using the original RTM package, and first run for each job after upgrade completed successfully (which is more common situation), or if you decided to create "fresh" jobs instead - you do not need to download and reinstall the new package. It has all the same components as the original package, except one upgrade-related component.

[jai64]

That puts me in a bit of a pickle.

I have 5TB of backups for my 70 servers and a bunch of complex jobs setup. I wish I would have read about the conversion in the Upgrade instructions???? It was only a few lines and seemed like a minor patch. It said nothing about a backup conversion processes?

I only have enough space to do new backups for a few days. I will have to delete all my old backup files, because it will not properly prune the old stuff out.

And yes my backup files are large? I suspect most peoples are. I have a lot of big servers.

How long until a fix?

~ Not happy

[Gostev]

The fix will be available later today.

[toreagan]

I have the same issue. Half of my jobs are stuck at 0% and I've tried "stopping" the job(s), but no luck... they are still hung. How can I successfully stop the job(s) without having to reboot the server? where will the patch be available from... new full download?

Please Advise... Thanks,
~Trent

[jai64]

Does that also mean that until this conversion is completed on a job, you can also not do a restore from that job?

~ joe

[Gostev]

Patch is already available and verified with one of the customers over webex.
Support will contact you shortly with the patch (2 files) and instructions on applying the fix.

Joe, you can restore from backup even if storage update procedure failed, no problems there. You just cannot make new backups on v3 jobs, until the fix is applied.

[BarryCoombs]

Is this patch going to be made available to everyone, or rolled up into a new release, just thinking about the planned customer upgrades I have over the next few weeks. Would prefer to not need to log a support call if I experience this.

[Gostev]

Barry, we will update the package on our web-site tonight. Anyone who will download the product after this will not run into the issue. But overall, this issue does not affect everyone who is upgrading, it is quite rare: so far just 3 support cases for a few hundreds of successful upgrades.

I will update this thread once the new package is available for download.

[BarryCoombs]

ok cheers, I will re-download tomorrow. I came across the issue on our demo Veeam box but just deleted the old backups as they weren't important anyway.

[jai64]

The update worked great. Thank you for the quick response.

~joe

[Gostev]

Thank you for updating the topic

[donikatz]

Wish I'd read this before upgrading, since I had this same problem. I have enough storage for new backups, though, so I just started fresh ones. Thanks for the update, good info. Loving 4.0 speed!

[Gostev]

The download package has been updated on http://www.veeam.com

If you have run into the above mentioned problem with the previous package:
1. Uninstall the existing version.
2. Download and install updated package (the new package has "_u" in its name), pointing it to the same configuration database.
3. Rename or copy away the .upgrade files next to VBK files for all failed jobs.
4. Run all jobs, first run should now complete successfully.
5. If all is well, delete .upgrade files preserved on step 3.

If you have replication jobs affected by installing the original RTM package, please contact support for assistance.

If you upgraded using the original RTM package, and first run for each job after upgrade completed successfully (which is more common situation), or if you decided to create "fresh" jobs instead - you do not need to download and reinstall the new package. It has all the same components as the original package, except one upgrade-related component.

Thanks Doni

[HenrikS]

I totally agree, this should have been mentioned in the upgrade notes.
I also ran into this problem, but since We've only been using veeam in a test environment, i just purged all old backups and reinstalled veeam and the DB.

[tjestr]

For the upgrade process, do i need the double space (for vbk files) to complete the upgrade succesfully?

[HenrikS]

If you have installed the updated package from veeam.com, it should not need the double space; no!

[Vitaliy S.]

Falko,

Henrik is right, you don't need to have double space for the proper upgrade of the VBK files. Just use the the new package with "_u" in its name.

[Felix]

Argh, had I read this thread earlier, just yesterday thrashed two weeks of v3 Backups, because Veeam Backup 4.0 was trying to upgrade a 800GB VBK which exceeded the free space on the backup raid. This *really* should've been noted in the release notes.

[colsztyn]

After upgrading to the original 4.0.80 and running my first test job, an existing job, it has been "hanging" at the 0 percent complete and "initializing target session". Is it anticipated that this will eventually complete or do I need to restart the server.

Also, can I get the patch to fix this issue.

Sincerely,

Chris

[Felix]

Read the thread, redownloading the latest version and following the steps in the thread should (supposedly) fix your problems.

[Gostev]

Thanks Felix! Correct.

[jeff@pcb.ca]

I have updated to the latest Backup 4.0 (just downloaded, looks to be the same build file however re-installed over the existing). I still have 2 servers that are stuck on the 0% problem. And after the update are still stuck on this. Should I leave them for the weekend? Or is there a way to force kill the .update process, clear the job and start with a new job?

I agree that some heads up on this issue would have been helpful.

[Vitaliy S.]

Hello Jeff, have you exactly followed the instructions posted by Anton here around what to do with the .upgrade files?

[Gostev]

Upgrade shoud complete pretty fast, no need to leave the jobs hanging for the weekend, it won't help. If you follow the procedure I listed above closely, the upgrade should complete successfully. Or yes, alternatively you can always simply delete the jobs for which automatic VBK storage upgrade failed, and create the new job - if this works for you.

[jeff@pcb.ca]

I can't stop the jobs in the console, otherwise I would and simply delete them. However I will go through the step as listed over the weekend and see if that clears things up.

Thanks

[Gostev]

Jeff, if you need any help at all, please do not hesitate to open a support case, and they will assist you. If the job won't stop, I believe you can simply kill veeam manager and agent processes with the task manager. Or, reboot Veeam Backup server. Then, you will be able to delete the job.

[jingxi02]

I ran into the same issue after I upgraded it to version 4. The job just hung on "Constructing roll back file" for over day. It won't stop or cancel. I end up have to move the old backup files to a different folder and delete the backups in the Veeam console and start the new Full back on the same job. If I need anything from the old backup, I just import them again.

Anyway, Veeam 4 works much better and solved alot of old issues. Some of my job was running only 15MB/s. The same job is running 120MB/s with Veeam 4. Huge improvement. Thanks alot.

[Gostev]

I end up have to move the old backup files to a different folder and delete the backups in the Veeam console and start the new Full back on the same job. If I need anything from the old backup, I just import them again.

Yep, this is exactly what I would do too. Despite major storage changes, Veeam Backup 4.0 can still import backups made by any previous version of Veeam Backup. Glad you are enjoying the new version!