Huge v6 bug with job scheduling

[joshuastenhouse]

I have found a major bug in v6 occurring in both an upgrade and a new install. My ticket ref is 5157348.

On Wednesday I upgraded my v5 Veeam server to v6. Immediatly all of my jobs started showing an incorrect next run time which didnt reflect what was actually set in the shcedule for the job. I thought this may of been a GUI issue, however my jobs then started running on the incorrect next run time. This included major VMs which shouldn't be run on a snapshot in business hours!

To rule out the upgrade as the issue I created a new VM and isntalled V6 from scratch using a local SQL express instance. I created a new job and scheduled it to only run on Wednesdays, the next run column will only show it to run on 03/12/11 rather than 07/12/11 no matter which day I pick that is not today or past Saturday. Thus ruling out any potential upgrade issue as its affecting a completely new install. My Veeam server, vcenter server and ESXi hosts are all showing the correct date and time and none have been altered recently.

I raised a fault at 13:30 on 30/11/12. It has taken until 10:10 02/12/12 for Veeam to acknolwedge they can replicate this fault in one of their test environments. I'm baffled as to how this hasn't been tested? Isn't backup scheduling a key part of backup software?

[Gostev]

Hi Joshua, do you possibly have full backup scheduled to happen on Saturday?

[joshuastenhouse]

It is configured for reversed incremental so no day is specified advanced settings>backup>days tab.

On the active full backup option it is set to "weekly on selected days" and then every day is selected to ensure Veeam performs a full backup each time. If I un-tick this option completely the next run time is still incorrect.

[Gostev]

Hi Joshua, please edit the job settings and under backup modes, switch to Incremental backup option, uncheck "Enable synthetic fulls", and then go back to reversed incremental option (the synthetic full schedule seem to wrongly affect this).

Also, a bit "cleaner" way to setup your desired schedule would be:
1. Use the incremental backup mode, uncheck Enable synthetic full.
3. Check Perform active full, Weekly, Wednesday.
3. On the job schedule step, set your job to run Daily, on these days, Wednesday.

Hope this helps.

[joshuastenhouse]

Haha, fantastic! So the bug is that if the sub check box enable synthetic full is checked then even if reversed incremental is selected then it overrides the scheduled date?

I have just tested this on my production server and it has resolved the issue. I just need to edit the other 38 jobs then and we are good to go.

46 hours since raising a ticket and no resolution. 1 hour since creating a forum post and its solved. Go figure. Thanks.

[Gostev]

Usually it is faster through support, but they have been slammed past 2 days as it usually happens during major releases.

What helped here is I approached this differently - instead of troubleshooting your issue through the logs (which I cannot do anyway), I tried to create the backup schedule you need they way I would do it (as posted above). And I did not see the issue present there. Then, I deduced the difference down to that checkbox

[joshuastenhouse]

Sorry, not working completely in production. I have a job scheduled to run every monday morning at 4:30am. I have unchecked the enable synthetic full box yet it is still saying next run date is 03/12/11 rather than 05/12/11?

[Gostev]

You should select only Wednesday in active full schedule (since this is what you want), and not every day. Otherwise, the active full backup will be created every single day, as instructed.

Full backup schedule now overrides main job schedule, this part is actually by design. Previously, people would often mistakenly schedule full backup for days when the job is not scheduled to run, and full backup would never create. There were support cases with people ending up with chains of hundreds of incremental backups without any fulls so now, full backup is always performed according to the schedule that is set for it (no matter if the job is scheduled to run on this specific day or not).

[joshuastenhouse]

Back through the jobs I go! Thanks.

[ThomasMc]

Back through the jobs I go! Thanks.

Code: Select all

function Amend-SynJob ($jobName) {
	$job = Get-VBRJob -Name $jobName
	$holdingobj = $job | Get-VBRJobOptions
	$holdingobj.BackupTargetOptions.TransformFullToSyntethic = $false
	$holdingobj.BackupTargetOptions.TransformIncrementsToSyntethic = $false
	$job.SetOptions($holdingobj)
}

Code: Select all

Get-VBRJob | %{Amend-SynJob $_.Name}

This will untick Enable Synthetic Full & Transform for all jobs

[Sethbartlett]

You are everywhere Thomas! I feel like a second me has hit the forum with a "Why not use powershell?" method Nicely done.

[Gostev]

Seth, you sound quite jealous

[ThomasMc]

Seth, you sound quite jealous

Seth is my Yoda


and thanks Seth

[real_tarantoga]

Thank you, Anton!
After upgrading this morning there weren't no more jobs scheduled to next sunday ...
unchecking the "enable synthetic fulls" box resolved the behaviour.

bug bug bug

Bye, Stefan

[joshuastenhouse]

[merged]

Hi,

I've created a new job, never run before with no seeding. I've set the schedule to run daily and selected only wednesdays at 7am. No backup window is configured and its not set to retry. This is a new v6 server, not an upgrade.

When I click finish and go back to the main screen, it says the next run date is 24/12/11. This is incorrect, it should be 28/12/11? i've tried removing the scheduling, disabling and re enabling the job to no avail. Please advise.

[Gostev]

Hi - this is the same issue, also described in more details in the sticky "Known Issues" topic. Thanks!

[joshuastenhouse]

Ah yes, didn't see the extended list Can you post a sample of this script rather than having to wait for support?

[Gostev]

Once the script gets sufficient validation, it will be included in the support KB article - until then, you can only get it from support.

[micwe]

[merged]

In the last year I was a very happy Veeam Backup customer
Veeam Backup 4V 5V works grate for my ESXI 4.0 SMB environment

VMWARE vSphere5 new release is available for the last mounts
It seems Veeam has release V6 without enough QA software debug

Last week I upgrade my existing VMWARE environment to V5.0
And install new win2008 r2 for Veeam Backup & Replication 6.0.0.15
Unfortunately my experience with the last Veeam Build is bad

It started with this - Hot add is not supported for this disk - 6.0.0.153
Hot add is not supported for this disk - 6.0.0.153

Then this - v6 recurring VM copy not overwriting previous copies
v6 recurring VM copy not overwriting previous copies

And now new bug from this weekend - all my schedule jobs start automatically not in the correct date
I have set backup tasks - VM will be backup once during week days
But all schedule task start working yesterday one by one in time setting like schedule task but not correct date
I was very surprise to see seven tasks running last night that WAS NOT PLANNED

Is this a known issue and is there a fix for this?

[Gostev]

Hi, see above for the scheduling bug.

Please note that hot add issue is not specific to Veeam v6, and moreover can be reproduced without Veeam B&R present in the environment (there is PowerShell script in the corresponding topic). There are 2 separate issues that can cause this, both are unrelated to issues with the Veeam code.

VM copy not overwriting previous copies was caused by the move to the new architecture with backup repositories (not a bug). However, based on a number of request we will make this behavior optional. With any software, I always recommend testing the new major release before upgrading to it - especially when there are big architecture changes.

If you'd like to discuss these 2 issues in more details, please feel free to post in the corresponding topics (not to derail this thread).

Thanks.

[micwe]

Hi - this is the same issue, also described in more details in the sticky "Known Issues" topic. Thanks!

can you please post a link for this page I can't find it

[micwe]

o.k found it
[BEFORE YOU UPGRADE] Summary of known v6 issues

I am using a VM copy task and replication task
Sorry but there is no "Full Synthetic" option under this tasks
This option is only avalible on backup task

[Gostev]

Please, read the topic that you found.

[micwe]

Hi Joshua, please edit the job settings and under backup modes, switch to Incremental backup option, uncheck "Enable synthetic fulls", and then go back to reversed incremental option (the synthetic full schedule seem to wrongly affect this).

Also, a bit "cleaner" way to setup your desired schedule would be:
1. Use the incremental backup mode, uncheck Enable synthetic full.
3. Check Perform active full, Weekly, Wednesday.
3. On the job schedule step, set your job to run Daily, on these days, Wednesday.

Hope this helps.

Gostev I have read your post several times and can't figure it out
Please bear in mind that I am using a copy vm and vm replication jobs
There is no "synthetic full" option under these jobs
I am doing a vm copy one a week so I don't understand why all job started to work on Saturday as well

[Vitaliy S.]

Hi Mickey, I guess Anton was referring to this line:

Some of these issues are being considered for the next v6 patch, I will update this topic once it is available.

[micwe]

Thanks for replying Vitaliy
Maybe Gostev can clarify this issue

[Gostev]

Issue 9
Jobs scheduled to start on selected days only will always start on synthetic full day when Enable synthetic full checkbox is checked, even if incremental backup mode is not used. To workaround, uncheck this checkbox and save settings. If your replication jobs are also affected, our support can provide a script resolving this.

[micwe]

So only Script can fix this on vm copy and replication tasks ?
Are you referring to the script post on first page in this post ?

[Gostev]

I am referring to the official script from our support.

[micwe]

Hello
For Future readsers
I can confirm that this issue was fixed
After installing Patch 2 (build number 6.0.0.164)