Comprehensive data protection for all workloads
Moebius
Veeam ProPartner
Posts: 208
Liked: 28 times
Joined: Jun 09, 2009 2:48 pm
Full Name: Lucio Mazzi
Location: Reggio Emilia, Italy
Contact:

Problems after upgrading to V6.1

Post by Moebius »

I upgraded from v6 to v6.1 last week. While the upgrade process itself was smooth, I'm receiving errors running the existing jobs.

1. Most jobs were configured as forward incrementals with Enable Synthetic Fulls for every day and Transform also enabled. After the upgrade, all of these jobs started failing after at transform time with the message "Full point was not found", even if the single VMs in the job backed up correctly. The transform phase was not performed and the .vib file was left in the backup chain.
Performing an active full corrects the problem. However, the backup chain still retains some leftover .vib files and an older .vbk that will eventually fall of the chain. Space is critical at this point.
I am going back to the Reverse Incremental mode. Switching to Forward with daily transforms has given me only trouble.

2. Fixed the above problem forcing active fulls, now I am getting a lot (about 70% of the VMs) failing due to VSS issues (either "failed to freeze guest" or "unfreeze error"). I often had those errors in the past but they had mostly gone away in recent times. The job retries sometimes succeed, sometimes fail again. Has anything been changed in the VSS mechanism or am I having some network-related issues here? Is anybody else getting back VSS problems?
Moebius
Veeam ProPartner
Posts: 208
Liked: 28 times
Joined: Jun 09, 2009 2:48 pm
Full Name: Lucio Mazzi
Location: Reggio Emilia, Italy
Contact:

Re: Problems after upgrading to V6.1

Post by Moebius »

Update: issue n.2 was due to the vCenter that was almost unresponsive for unknown reason. Rebooting the vCenter made the problem disappear. All jobs are working fine now.

Issue n. 1 however remains. I had to force an Active Full on all of my jobs that were configured for Forward Incrementals with Daily Transform to get rid of the "Full point not found" error.
I ended up switching back to Reverse Incremental (which triggered another full) for all jobs. Incidentally, the jobs seem to take much less time this way: a daily job with 21 VMs that used to take about 5:30 hours (3 of which just for the transform) took less than 3 hours to complete. I understand, however, that there are many variables that dictate whether backup type is faster.
Gostev
Chief Product Officer
Posts: 31816
Liked: 7302 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Problems after upgrading to V6.1

Post by Gostev »

I will check on #1 with devs.
rvopat
Influencer
Posts: 22
Liked: never
Joined: Mar 22, 2010 3:26 pm
Contact:

Re: Problems after upgrading to V6.1

Post by rvopat »

I am having this same problem with error 1. If i run an incremental job that does not have a synthetic full for today, it works but i wonder if it is going to fail on the days it is scheduled to run the synthetic full.

i upgraded from 6 to 6.1 just today and i am seeing these problems.

The jobs seem to run - it just seems to have a problem with the transform - the new jobs state they are of type Veeam Backup & REplication incremental backup file and the old job files state they are Veeam Backup & Replication reversed incremntal file.
Gostev
Chief Product Officer
Posts: 31816
Liked: 7302 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Problems after upgrading to V6.1

Post by Gostev »

Please always include support case ID when posting about technical issue.

I see this exact error on the bug tracker, but it was logged pretty long ago, before 6.0 was released. Transform will fail if actual VBK is missing (VBK can be deleted by retention policy in very rare cases, this requires certain conditions to achieve - basically, if all VMs have their full backups in the VIB files, VBK may get cleaned up). Try doing active full backup to force create new VBK, and the error should go away.
Moebius
Veeam ProPartner
Posts: 208
Liked: 28 times
Joined: Jun 09, 2009 2:48 pm
Full Name: Lucio Mazzi
Location: Reggio Emilia, Italy
Contact:

Re: Problems after upgrading to V6.1

Post by Moebius »

My case ID was #5196803 (now closed). The actual VBK was present in all my jobs, and they were configured as stated above: forward incremental with daily transform. Also, all recent jobs prior to the upgrade went fine. That means that just before upgrading, the full backup (.vbk) was the most recent object of the chain for each job.
Immediately after upgrading they started failing as described.

I forced an active full for each job and the error went away. However, running a full on some jobs was a pain (30+ hours), and I think that this issue should be stated clearly in the release notes.

I went back to reverse incrementals for all my jobs since.
homerjnick
Expert
Posts: 213
Liked: 35 times
Joined: Feb 20, 2012 4:13 pm
Full Name: Nick Mahlitz
Contact:

Re: Problems after upgrading to V6.1

Post by homerjnick »

Upgraded to 6.1 and my replication job for a W2003 server 64 bit with SQL 2008 hangs at 100% ans has hung for 80 hours now.

The job is at 99% yet in the activity window it is at 100% with all three harddrives having a green tick so replication has completed but the job hasn't finished...the backup job is awaiting for the lock on the VM to be released.

Never happened on 6.0.

Case ID is #5198797.

Can't see anyone else having the issue though so not a 6.1 bug perse...
murdocmk
Enthusiast
Posts: 25
Liked: 4 times
Joined: Dec 14, 2009 5:10 pm
Contact:

Re: Problems after upgrading to V6.1

Post by murdocmk »

Had this same problem (the "Full point was not found" problem) happen at a customer the other day. They upgraded and followed the upgrade procedure and all jobs were stopped. The upgrade happened during the week so it was in the middle of a VIB chain. On the following weekend when the synthetic full and daily transform was supposed to occur, they received the "Full point was not found" error. We just set the existing backups aside (removed them from Veeam) and started a new set because we had the space to do so. We'll know by this next weekend if the synthetic full and daily transform works as expected. If not, we'll open a case (but I expect it to).

Since Moebius and this customer experienced the same thing after the 6.1 upgrade, it certainly seems like the upgrade has a part in the problem. I'd be interested to hear if anyone else has jobs set up in this manner (forward incremental with weekly synthetic fulls and daily transform) and has upgraded to 6.1 without issue. We've got other customers to upgrade but are holding off a bit until the new version has had some more time in the field.
nreutemann
Enthusiast
Posts: 47
Liked: 6 times
Joined: Mar 06, 2012 11:45 pm
Full Name: Nicolas Reutemann
Contact:

Re: Problems after upgrading to V6.1

Post by nreutemann »

Hello,
This weekend i got the message "Full point was not found." on some backup jobs.
The weekend before this, everything goes fine.

Note: the jobs that fails, was the jobs that I change the location of the backup files. I move some backups folders between backup repositories trying to optimize the distribution of space and IOs.

tequesta
Vitaliy S.
VP, Product Management
Posts: 27377
Liked: 2802 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Problems after upgrading to V6.1

Post by Vitaliy S. »

Nicolas, have you opened a support ticket with our technical team? I would appreciate if you could let our team investigate the debug log files.
nreutemann
Enthusiast
Posts: 47
Liked: 6 times
Joined: Mar 06, 2012 11:45 pm
Full Name: Nicolas Reutemann
Contact:

Re: Problems after upgrading to V6.1

Post by nreutemann »

A few minutes ago I post a support case

Nicolas.
Gostev
Chief Product Officer
Posts: 31816
Liked: 7302 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Problems after upgrading to V6.1

Post by Gostev » 2 people like this post

The research for "Full point was not found" issue was completed, thanks for everyone who provided the log files. We were able to reproduce this behavior in house.

Turns out, only the jobs which went through changing its backup mode twice. The only affected job mode changes pattern is the following: incremental backup mode, then reversed incremental backup mode, then back to incremental backup mode with transform enabled, and running the job at least once in between the changes. Only these kind of jobs will be affected by this issue after upgrade to 6.1.

The workaround is very simple, uncheck the transform check box, and retry or run job to make it create a full backup (synthetic or active does not matter). Then, check the transform check box again, and transformation will start working normally.
murdocmk
Enthusiast
Posts: 25
Liked: 4 times
Joined: Dec 14, 2009 5:10 pm
Contact:

Re: Problems after upgrading to V6.1

Post by murdocmk »

Gostev,

The workaround you provide will result in an additional full or synthetic full that does not get transformed, is that true? For example, assume I do weekly synthetic fulls with daily transform and I have:

Sat:VBK Sun:vib Mon:vib Tues:vib Wed:vib Thurs:vib Fri:vib

And on Saturday the job fails with the "full point was not found" error. I uncheck the transform box and run the job again, and then I have:

Sat:VBK Sun:vib Mon:vib Tues:vib Wed:vib Thurs:vib Fri:vib Sat:VBK

And then I check the box again and the job runs on Sunday, I'll have:

Sat:VBK Sun:vib Mon:vib Tues:vib Wed:vib Thurs:vib Fri:vib Sat:VBK Sun:vib

And that will continue on with vib files until the following Saturday, where the synthetic full will run and the previous Saturday full is transformed into VRB files. The synthetic full from two saturdays ago will still be there, is that not true?

I'm not saying that's a terrible thing -- but in some cases backup space is tight and our customers don't have room for an additional full in their backup files. I think for those customers (that have made backup job mode changes twice) we may need to wait until a patch is released before going to 6.1.
Gostev
Chief Product Officer
Posts: 31816
Liked: 7302 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Problems after upgrading to V6.1

Post by Gostev »

Correct.
Tobias_Elfstrom
Enthusiast
Posts: 84
Liked: 8 times
Joined: Jul 04, 2012 6:32 am
Full Name: Tobias Elfstrom
Contact:

Re: Problems after upgrading to V6.1

Post by Tobias_Elfstrom »

Any patches for this issue?

Altough my problem isn't exactly the same i would guess that a patch resolving this would resolv my problem too.
Did a clean install of 6.1, started out with jobs in reversed incremental mode, changed to incremental backup mode with transform enabled and got the "Full point was not found" errors.

So no upgrade, no twice changing mode etc.

BR Tobias.
foggy
Veeam Software
Posts: 21139
Liked: 2141 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Problems after upgrading to V6.1

Post by foggy »

Further investigation showed that the actual issue has nothing to do with upgrade itself but rather with installation of v6.1 in general and is observed on switching back from reversed to forward incremental mode (which is always a second switch as the default mode is forward incremental). The patch will help but actually the workaround given above also helps, so there's no need to wait for the patch.
Mindflux
Enthusiast
Posts: 32
Liked: never
Joined: Nov 10, 2010 7:52 pm
Full Name: RyanW
Contact:

Re: Problems after upgrading to V6.1

Post by Mindflux »

I got bit by this too. Upgraded mid week, no changing from reverse to incremental or anything like that.

I'm following Anton's recommendation to turn off transform and perform a full backup.. hopefully this will fix it.

8)
foggy
Veeam Software
Posts: 21139
Liked: 2141 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Problems after upgrading to V6.1

Post by foggy »

Most likely you've switched the modes in this job previously, while running v6. And after upgrading the issue showed up.
itdirector
Enthusiast
Posts: 59
Liked: 3 times
Joined: Jan 19, 2012 8:53 pm
Full Name: friedman

Re: Problems after upgrading to V6.1

Post by itdirector »

Any update on this issue? Will there be a patch released or is the only option still to create a new Full? We have a 3TB full with 11 100GB+ transforms (this is a monthly job that has a 7 year retention policy). We don't/can't keep a 2nd 3TB full backup file.
On another note, we always had our Monthly jobs as Incremental --> Synthetic fulls --> Transform on version 6.0. With the version 6.1 upgrade, it got changed to "Reverse incremental" - & not by us. We switched it back to synthetic fulls with transforms after the reverse incremental job ran. We now get the "Full point was not found" errors.

Thank you.
Mindflux
Enthusiast
Posts: 32
Liked: never
Joined: Nov 10, 2010 7:52 pm
Full Name: RyanW
Contact:

Re: Problems after upgrading to V6.1

Post by Mindflux »

foggy wrote:Most likely you've switched the modes in this job previously, while running v6. And after upgrading the issue showed up.
5.1 had been running non stop for over a year without fiddling. I may have switched modes right at the beginning when I started using VEEAM, but not in the timeframe of the backups listed within the VEEAM console. I never ran 6.0, I went from 5.1 to 6.1!
foggy
Veeam Software
Posts: 21139
Liked: 2141 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Problems after upgrading to V6.1

Post by foggy »

Mindflux wrote:5.1 had been running non stop for over a year without fiddling. I may have switched modes right at the beginning when I started using VEEAM, but not in the timeframe of the backups listed within the VEEAM console. I never ran 6.0, I went from 5.1 to 6.1!
The version of the previous installation does not really matter, it is v6.1 who does not handle this switch correctly.

Btw, did following the recommendations given above help in your case?
Mindflux
Enthusiast
Posts: 32
Liked: never
Joined: Nov 10, 2010 7:52 pm
Full Name: RyanW
Contact:

Re: Problems after upgrading to V6.1

Post by Mindflux »

foggy wrote: The version of the previous installation does not really matter, it is v6.1 who does not handle this switch correctly.

Btw, did following the recommendations given above help in your case?
I won't know until I get another transform on Saturday.
foggy
Veeam Software
Posts: 21139
Liked: 2141 times
Joined: Jul 11, 2011 10:22 am
Full Name: Alexander Fogelson
Contact:

Re: Problems after upgrading to V6.1

Post by foggy »

itdirector wrote:Any update on this issue? Will there be a patch released or is the only option still to create a new Full?
The patch will be definitely released sooner or later (the position regarding patch release is described in this post from Anton). The current workaround is to clear the transform check box and create a full backup.
AlexL
Service Provider
Posts: 105
Liked: 7 times
Joined: Aug 24, 2010 8:55 am
Full Name: Alex
Contact:

Re: Problems after upgrading to V6.1

Post by AlexL »

Great, also got the "full point not found" error. Nothing to do with changing incremental or reverse or upgrading, all I did was change the repository from a CIFS location to a Windows location in the middle of the week, daily jobs still work fine, transform/full gives the error. Thanks for messing this up VEEAM, this again means overtime this weekend moving over 3TB of backups around.
Vitaliy S.
VP, Product Management
Posts: 27377
Liked: 2802 times
Joined: Mar 30, 2009 9:13 am
Full Name: Vitaliy Safarov
Contact:

Re: Problems after upgrading to V6.1

Post by Vitaliy S. »

Alex, could you please contact our support team and let our support analysts figure out why this change has caused such a behavior?
AlexL
Service Provider
Posts: 105
Liked: 7 times
Joined: Aug 24, 2010 8:55 am
Full Name: Alex
Contact:

Re: Problems after upgrading to V6.1

Post by AlexL »

will do so tomorrow
itdirector
Enthusiast
Posts: 59
Liked: 3 times
Joined: Jan 19, 2012 8:53 pm
Full Name: friedman

Re: Problems after upgrading to V6.1

Post by itdirector »

If I open up a tech support case, does anyone know if there is a hotfix available for this until a patch is releases? Per my previous post. Thx

Any update on this issue? Will there be a patch released or is the only option still to create a new Full? We have a 3TB full with 11 100GB+ transforms (this is a monthly job that has a 7 year retention policy). We don't/can't keep a 2nd 3TB full backup file.
Gostev
Chief Product Officer
Posts: 31816
Liked: 7302 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Problems after upgrading to V6.1

Post by Gostev »

Let me check, I am back from vacation and finally can talk to devs and QC directly :)
Gostev
Chief Product Officer
Posts: 31816
Liked: 7302 times
Joined: Jan 01, 2006 1:01 am
Location: Baar, Switzerland
Contact:

Re: Problems after upgrading to V6.1

Post by Gostev »

OK, so the original problem discussed in this thread (see summary in my post the previous page) is addressed in Patch 1 that should be released soon.
AlexL wrote:Great, also got the "full point not found" error. Nothing to do with changing incremental or reverse or upgrading, all I did was change the repository from a CIFS location to a Windows location in the middle of the week, daily jobs still work fine, transform/full gives the error. Thanks for messing this up VEEAM, this again means overtime this weekend moving over 3TB of backups around.
This specifically is not a known issue, so please open a support case and provide all logs for further investigation.
itdirector
Enthusiast
Posts: 59
Liked: 3 times
Joined: Jan 19, 2012 8:53 pm
Full Name: friedman

Re: Problems after upgrading to V6.1

Post by itdirector »

Gostev, thx for the reply. Not sure if you were addressing Alex or my post, but I am having that exact issue that was in your summary from the previous page. I know it is being addresses in Patch 1, but my question is if there is a hotfix addressing this one issue in the meantime? If there is, I can then open up a ticket & get that hotfix so our monthly backups start running again.

Thx
Post Reply

Who is online

Users browsing this forum: Bing [Bot] and 47 guests