Problems after upgrading to V6.1

[Moebius]

I upgraded from v6 to v6.1 last week. While the upgrade process itself was smooth, I'm receiving errors running the existing jobs.

1. Most jobs were configured as forward incrementals with Enable Synthetic Fulls for every day and Transform also enabled. After the upgrade, all of these jobs started failing after at transform time with the message "Full point was not found", even if the single VMs in the job backed up correctly. The transform phase was not performed and the .vib file was left in the backup chain.
Performing an active full corrects the problem. However, the backup chain still retains some leftover .vib files and an older .vbk that will eventually fall of the chain. Space is critical at this point.
I am going back to the Reverse Incremental mode. Switching to Forward with daily transforms has given me only trouble.

2. Fixed the above problem forcing active fulls, now I am getting a lot (about 70% of the VMs) failing due to VSS issues (either "failed to freeze guest" or "unfreeze error"). I often had those errors in the past but they had mostly gone away in recent times. The job retries sometimes succeed, sometimes fail again. Has anything been changed in the VSS mechanism or am I having some network-related issues here? Is anybody else getting back VSS problems?

[Moebius]

Update: issue n.2 was due to the vCenter that was almost unresponsive for unknown reason. Rebooting the vCenter made the problem disappear. All jobs are working fine now.

Issue n. 1 however remains. I had to force an Active Full on all of my jobs that were configured for Forward Incrementals with Daily Transform to get rid of the "Full point not found" error.
I ended up switching back to Reverse Incremental (which triggered another full) for all jobs. Incidentally, the jobs seem to take much less time this way: a daily job with 21 VMs that used to take about 5:30 hours (3 of which just for the transform) took less than 3 hours to complete. I understand, however, that there are many variables that dictate whether backup type is faster.

[Gostev]

I will check on #1 with devs.

[rvopat]

I am having this same problem with error 1. If i run an incremental job that does not have a synthetic full for today, it works but i wonder if it is going to fail on the days it is scheduled to run the synthetic full.

i upgraded from 6 to 6.1 just today and i am seeing these problems.

The jobs seem to run - it just seems to have a problem with the transform - the new jobs state they are of type Veeam Backup & REplication incremental backup file and the old job files state they are Veeam Backup & Replication reversed incremntal file.

[Gostev]

Please always include support case ID when posting about technical issue.

I see this exact error on the bug tracker, but it was logged pretty long ago, before 6.0 was released. Transform will fail if actual VBK is missing (VBK can be deleted by retention policy in very rare cases, this requires certain conditions to achieve - basically, if all VMs have their full backups in the VIB files, VBK may get cleaned up). Try doing active full backup to force create new VBK, and the error should go away.

[Moebius]

My case ID was #5196803 (now closed). The actual VBK was present in all my jobs, and they were configured as stated above: forward incremental with daily transform. Also, all recent jobs prior to the upgrade went fine. That means that just before upgrading, the full backup (.vbk) was the most recent object of the chain for each job.
Immediately after upgrading they started failing as described.

I forced an active full for each job and the error went away. However, running a full on some jobs was a pain (30+ hours), and I think that this issue should be stated clearly in the release notes.

I went back to reverse incrementals for all my jobs since.

[homerjnick]

Upgraded to 6.1 and my replication job for a W2003 server 64 bit with SQL 2008 hangs at 100% ans has hung for 80 hours now.

The job is at 99% yet in the activity window it is at 100% with all three harddrives having a green tick so replication has completed but the job hasn't finished...the backup job is awaiting for the lock on the VM to be released.

Never happened on 6.0.

Case ID is #5198797.

Can't see anyone else having the issue though so not a 6.1 bug perse...

[murdocmk]

Had this same problem (the "Full point was not found" problem) happen at a customer the other day. They upgraded and followed the upgrade procedure and all jobs were stopped. The upgrade happened during the week so it was in the middle of a VIB chain. On the following weekend when the synthetic full and daily transform was supposed to occur, they received the "Full point was not found" error. We just set the existing backups aside (removed them from Veeam) and started a new set because we had the space to do so. We'll know by this next weekend if the synthetic full and daily transform works as expected. If not, we'll open a case (but I expect it to).

Since Moebius and this customer experienced the same thing after the 6.1 upgrade, it certainly seems like the upgrade has a part in the problem. I'd be interested to hear if anyone else has jobs set up in this manner (forward incremental with weekly synthetic fulls and daily transform) and has upgraded to 6.1 without issue. We've got other customers to upgrade but are holding off a bit until the new version has had some more time in the field.

[nreutemann]

Hello,
This weekend i got the message "Full point was not found." on some backup jobs.
The weekend before this, everything goes fine.

Note: the jobs that fails, was the jobs that I change the location of the backup files. I move some backups folders between backup repositories trying to optimize the distribution of space and IOs.

tequesta

[Vitaliy S.]

Nicolas, have you opened a support ticket with our technical team? I would appreciate if you could let our team investigate the debug log files.

[nreutemann]

A few minutes ago I post a support case

Nicolas.

[Gostev]

The research for "Full point was not found" issue was completed, thanks for everyone who provided the log files. We were able to reproduce this behavior in house.

Turns out, only the jobs which went through changing its backup mode twice. The only affected job mode changes pattern is the following: incremental backup mode, then reversed incremental backup mode, then back to incremental backup mode with transform enabled, and running the job at least once in between the changes. Only these kind of jobs will be affected by this issue after upgrade to 6.1.

The workaround is very simple, uncheck the transform check box, and retry or run job to make it create a full backup (synthetic or active does not matter). Then, check the transform check box again, and transformation will start working normally.

[murdocmk]

Gostev,

The workaround you provide will result in an additional full or synthetic full that does not get transformed, is that true? For example, assume I do weekly synthetic fulls with daily transform and I have:

Sat:VBK Sun:vib Mon:vib Tues:vib Wed:vib Thurs:vib Fri:vib

And on Saturday the job fails with the "full point was not found" error. I uncheck the transform box and run the job again, and then I have:

Sat:VBK Sun:vib Mon:vib Tues:vib Wed:vib Thurs:vib Fri:vib Sat:VBK

And then I check the box again and the job runs on Sunday, I'll have:

Sat:VBK Sun:vib Mon:vib Tues:vib Wed:vib Thurs:vib Fri:vib Sat:VBK Sun:vib

And that will continue on with vib files until the following Saturday, where the synthetic full will run and the previous Saturday full is transformed into VRB files. The synthetic full from two saturdays ago will still be there, is that not true?

I'm not saying that's a terrible thing -- but in some cases backup space is tight and our customers don't have room for an additional full in their backup files. I think for those customers (that have made backup job mode changes twice) we may need to wait until a patch is released before going to 6.1.

[Gostev]

Correct.

[Tobias_Elfstrom]

Any patches for this issue?

Altough my problem isn't exactly the same i would guess that a patch resolving this would resolv my problem too.
Did a clean install of 6.1, started out with jobs in reversed incremental mode, changed to incremental backup mode with transform enabled and got the "Full point was not found" errors.

So no upgrade, no twice changing mode etc.

BR Tobias.

[foggy]

Further investigation showed that the actual issue has nothing to do with upgrade itself but rather with installation of v6.1 in general and is observed on switching back from reversed to forward incremental mode (which is always a second switch as the default mode is forward incremental). The patch will help but actually the workaround given above also helps, so there's no need to wait for the patch.

[Mindflux]

I got bit by this too. Upgraded mid week, no changing from reverse to incremental or anything like that.

I'm following Anton's recommendation to turn off transform and perform a full backup.. hopefully this will fix it.

[foggy]

Most likely you've switched the modes in this job previously, while running v6. And after upgrading the issue showed up.

[itdirector]

Any update on this issue? Will there be a patch released or is the only option still to create a new Full? We have a 3TB full with 11 100GB+ transforms (this is a monthly job that has a 7 year retention policy). We don't/can't keep a 2nd 3TB full backup file.
On another note, we always had our Monthly jobs as Incremental --> Synthetic fulls --> Transform on version 6.0. With the version 6.1 upgrade, it got changed to "Reverse incremental" - & not by us. We switched it back to synthetic fulls with transforms after the reverse incremental job ran. We now get the "Full point was not found" errors.

Thank you.

[Mindflux]

Most likely you've switched the modes in this job previously, while running v6. And after upgrading the issue showed up.

5.1 had been running non stop for over a year without fiddling. I may have switched modes right at the beginning when I started using VEEAM, but not in the timeframe of the backups listed within the VEEAM console. I never ran 6.0, I went from 5.1 to 6.1!

[foggy]

5.1 had been running non stop for over a year without fiddling. I may have switched modes right at the beginning when I started using VEEAM, but not in the timeframe of the backups listed within the VEEAM console. I never ran 6.0, I went from 5.1 to 6.1!

The version of the previous installation does not really matter, it is v6.1 who does not handle this switch correctly.

Btw, did following the recommendations given above help in your case?

[Mindflux]

The version of the previous installation does not really matter, it is v6.1 who does not handle this switch correctly.

Btw, did following the recommendations given above help in your case?

I won't know until I get another transform on Saturday.

[foggy]

Any update on this issue? Will there be a patch released or is the only option still to create a new Full?

The patch will be definitely released sooner or later (the position regarding patch release is described in this post from Anton). The current workaround is to clear the transform check box and create a full backup.

[AlexL]

Great, also got the "full point not found" error. Nothing to do with changing incremental or reverse or upgrading, all I did was change the repository from a CIFS location to a Windows location in the middle of the week, daily jobs still work fine, transform/full gives the error. Thanks for messing this up VEEAM, this again means overtime this weekend moving over 3TB of backups around.

[Vitaliy S.]

Alex, could you please contact our support team and let our support analysts figure out why this change has caused such a behavior?

[AlexL]

will do so tomorrow

[itdirector]

If I open up a tech support case, does anyone know if there is a hotfix available for this until a patch is releases? Per my previous post. Thx

Any update on this issue? Will there be a patch released or is the only option still to create a new Full? We have a 3TB full with 11 100GB+ transforms (this is a monthly job that has a 7 year retention policy). We don't/can't keep a 2nd 3TB full backup file.

[Gostev]

Let me check, I am back from vacation and finally can talk to devs and QC directly

[Gostev]

OK, so the original problem discussed in this thread (see summary in my post the previous page) is addressed in Patch 1 that should be released soon.

Great, also got the "full point not found" error. Nothing to do with changing incremental or reverse or upgrading, all I did was change the repository from a CIFS location to a Windows location in the middle of the week, daily jobs still work fine, transform/full gives the error. Thanks for messing this up VEEAM, this again means overtime this weekend moving over 3TB of backups around.

This specifically is not a known issue, so please open a support case and provide all logs for further investigation.

[itdirector]

Gostev, thx for the reply. Not sure if you were addressing Alex or my post, but I am having that exact issue that was in your summary from the previous page. I know it is being addresses in Patch 1, but my question is if there is a hotfix addressing this one issue in the meantime? If there is, I can then open up a ticket & get that hotfix so our monthly backups start running again.

Thx