2 questions about synthetic fulls

[tfloor]

Hello,

I'm using veeam 6 patch 3 on a physical windows 2008 server using san-mode with 4 jobs to backup vm's from the same template (linux / oes / win2003 / win2008)
I'm also writing to tape and have a good solution for that with backup exec that depends on datstamps of files.

Question 1: (see joblog)
What i see now is that for 1 job, that is set for 14 restore points there are some .VBK files. (ofcourse)
On friday i schedule the synthetic full. It's now sunday 27 Feb. So i had a new VBK file with a datestamp of 25 Feb.
That's correct. But i see that there are also 2 other previous vbk files that has a new datestamp of 25.

How is that possible?, is there a process that is changing datestamps of previous VBK files?, I only see this so far for 1 job.... Very strange.

24-2-2012 23:41:12 :: Job started at 24-2-2012 23:41:07
24-2-2012 23:41:12 :: Building VM list
24-2-2012 23:41:39 :: SRV4 is no longer processed by this job. Make sure this change is intentional.
24-2-2012 23:41:39 :: VM size: 1,3 TB
24-2-2012 23:41:39 :: Changed block tracking is enabled
24-2-2012 23:41:48 :: Preparing next VM for processing
25-2-2012 0:12:08 :: Processing 'SRV1'
25-2-2012 0:33:41 :: Preparing next VM for processing
25-2-2012 0:33:41 :: Processing 'SRV2'
25-2-2012 0:40:26 :: Preparing next VM for processing
25-2-2012 0:40:26 :: Processing 'SRV3'
25-2-2012 0:44:07 :: Preparing next VM for processing
25-2-2012 0:44:07 :: Processing 'SRV4'
25-2-2012 0:54:55 :: Preparing next VM for processing
25-2-2012 0:54:55 :: Processing 'SRV5'
25-2-2012 0:59:00 :: All VMs have been processed
25-2-2012 0:59:04 :: VM 'SRV4' is outdated and will be deleted
25-2-2012 0:59:21 :: VM 'SRV4' is outdated and will be deleted
25-2-2012 1:04:09 :: VM 'SRV4' is outdated and will be deleted
25-2-2012 1:07:51 :: Synthetic full backup created successfully
25-2-2012 16:33:49 :: Load: Source 9% > Proxy 33% > Network 3% > Target 98%
25-2-2012 16:33:49 :: Primary bottleneck: Target
25-2-2012 16:33:49 :: Job finished at 25-2-2012 16:33:49

I need to find a way that nothing is changing timestamps of files (otherwise the complete backupsolution to tape is useless)

Question 2: (see joblog)
The vm's backup on friday (synthetic full) is normally ready within 2 hours. But to create a VBK file the job finished about 15 hours later. Is that the normal process ? (or is this the time that veeam need to copy the full .VBK file to the target?)
So first it creates the incrementals (as usual), and then create a .VBK file (around 900GB) , that take up to 15 hours? (for about 5 vm's)

I hope someone can explain me this, so i can finalize my backup strategy.

Kind regards,

Tristan.

[Gostev]

Hi,

1. This is not expected - I asked devs to check the code around this.

2. Yes, this is normal process. New synthetic full backup is created from the previous full backup, and incremental backups - so yes, in your case it is about taking 900 GB of data blocks from those files, and creating the new 900 GB file. The time it takes depends only on 2 things: the size of incremental backups to process, and performance of the backup storage. Looks like your backup storage is not very fast?

Thanks!

[tsightler]

1. This is not expected - I asked devs to check the code around this.

I'm suspecting, based on the messages about "SRV4 is outdated", that there was a server named SRV4 deleted/removed from the job that hit the "Deleted VM Retention" period. When you hit this the blocks backing this VM are literally removed from the backup files which does cause the older files to be modified. Just a guess based on the log messages though.

[Gostev]

This should not be happening in the forward incremental backup job - only backup modes with transform (those with a single full backup file) do this in v6.

[tfloor]

1. The Srv4 server was removed from inventory since it was orphaned. But then immediately added back to vcenter.
Because the vmid changed and veeam could not find the vm anymore I removed the server from veeam job and add again.
But it looks like i still get the warning messages in the reports about this vm
My vm retention period is 14 days. (the same as snapshots).

2. My target is a DD610 not very fast indeed. But I get a brand new enterprise class qnap this week. The highest model in range.

[tfloor]

I'm suspecting, based on the messages about "SRV4 is outdated", that there was a server named SRV4 deleted/removed from the job that hit the "Deleted VM Retention" period. When you hit this the blocks backing this VM are literally removed from the backup files which does cause the older files to by modified. Just a guess based on the log messages though.

That's correct. I had to re-add the server.
I still don't understand the vm retention period and what it does exactly.
As long as I have to backup to tape, I don't want modifications to older vbk files.

[tfloor]

It's possible that this was the case, but i'm using forward incrementals, and "GOSTEV" tells us, that this is not normal. For me i also don't want this, not only because of the change in datestamps, but i do not see any reason for us to change the files, because over about 2 weeks the files are deleted automatically. (14 snapshots)

So what is the correct answer for this.
- it's a bug (should not happen with forward-incr.)
- it's normal (maybe there is a way to disable the feature ......)

Thanks
Tristan

[tsightler]

If Gostev says it should not happen, then I would say it must be a bug. I was only intending to indicating that I also see this behavior and describe the circumstances under which I have seen it occur.

[tfloor]

Gostev,

Can you confirm this is a bug? Or do you need more information. Maybe special log files.

Regards
Tristan

[Gostev]

It should not happen, but it is not a bug. How's that

Seriously though - while I am still to confirm this, but from the above it looks like we are seeing left overs of valid v5 functionality which *should* have been removed from v6, but may be were not due to the lack of time or something. I was positive that they were in fact removed from v6, which is why I said "it should not be happening".

I will review the code with the devs (did not get a chance to do this yet), and if this is indeed the case, we will address that in the upcoming minor release. I will post here when I have definite information.

Thanks!

[tfloor]

Gostev,

That explains something. Wondering why this was the case in v5. I don't see it as a feature
Let me know if you have the results.

Regarding to the log file (maybe a bug) :
Why do i still see the this message each day :
24-2-2012 23:41:39 :: SRV4 is no longer processed by this job. Make sure this change is intentional.
While the server is backed up with status "success"

[foggy]

Tristan, as you have stated above, you had to re-add the VM to the backup job. This message refers to the VM that was deleted from the job - just to point you that it is not backed up anymore. From the product's standpoint, the VM that was then added to the job is another VM, as VBR tracks VMs by their uniqie IDs (and the ID for this VM was changed after re-adding it to inventory).

[tfloor]

Foggy,
You are right. I've readd the server. And therefore it had the same name. That was confusing me. Underwater it's another vm according to the Id. So based on that story I think the message will go away after the vm retention period. I hope

Thanks
Tristan

[Gostev]

Yep.

[tfloor]

Yep ?

Is that the definitive answer on question 1 or uhhh a yep only

[Gostev]

The answer to "I think the message will go away after the vm retention period"

[tfloor]

That's clear.
i will wait and see if the message expired

[tfloor]

Gostev,

Any idea when I can receive the information you will get from dev?

I like veeam but i"m using it 2 weeks and have found 3 bugs
I hope there will come patches for that soon. I only have experiences with patch release times of other vendors.

[Gostev]

Hi Tristan, reading through my response above the specific behavior being discussed in this topic is not a bug, but existing functionality of VM-level retention that we had in the product since v5, so this will definitely not get a patch. The current plan is to drop this functionality from the forward incremental backup mode in the next minor release. Thanks.

[tfloor]

Gostev,
So I can Not expect this in patch 4 or 5. I have to wait for 6.1 or something.
But what can I expect. Do I have to wait for weeks or months. Or half a year.
Is there maybe a way to disable the feature? Change the retention to 0 or 9999?

[Gostev]

Good call. Changing the "Deleted VM retention" to a very high number for this job should effectively disable this functionality until 6.1 is released, which should definitely happen sooner than in 9999 days

[tfloor]

Gostev,
I've set it to the max 999 days. So I hope to see the release within 3 years
Thanks to confirm my own workaround haha
Thanks
Tristan

[tfloor]

It would be nice if the "known issues" list can be updated more frequently.
Fox example to add this one
And the schedule bug. And maybe the duplicate email notification,
This are only my own experiences, so there must be more that is known

[Gostev]

Our support KB covers many more known issues and common questions. The known issue list on this forum is only for issues which have caused multiple support cases. I must keep it short and to the point - people don't read walls of text.

[tfloor]

I understand. So there is no place to find the known bugs. There is only a place to search for problems and get a result that could be a bug.

[Gostev]

Correct. The best way is to search is the support KB with your issue/error.

[ddockter]

[merged]

My backups are scheduled to run fulls on Friday. I noticed this morning that one of the full backups also updated the time stamp of a .vib from 02-16. Why is that?