Synthetic full monthly instead of weekly?

[Berkovska]

Hi there,
I was wondering if its possible to run Synthetic full backup monthly instead of weekly, may be can be configured via powershell?

Thanks!

[HannesK]

Hello,
no, this is not possible because it would result in extremely high load on the backup storage. As a result, the creation of synthetic full would probably take longer than 24h with the result that backup jobs fail.

What is your use case for a monthly synthetic full?

Thanks,
Hannes

[Berkovska]

Active full is taking a long time.

[HannesK]

yes, but why no weekly synthetic full? As you are a service provider, I guess you are using ReFS anyway. So space usage would not be an issue with weekly synthetic fulls.

With ReFS also monthly synthetic full would be no technical issue, but it would break all systems with classic file systems.

[Berkovska]

Understand.
The deal is there is limited storage available, i though running monthly full would be great solution.

[HannesK]

due to the fact that this setting would break many installations, we do not plan that.

to save disk space in your situation I see the following options:
- use an incremental forever mode (forward or reverse)
- use ReFS (well, I don't see any good argument to point to the same blocks, but it seems to make people happy )

[Berkovska]

Thank you Hannesk!
The whole deal is retention for 365 days. I dont want to do forever forward because i have no idea if that will work ever at all.
So I have option with monthly full. Active full takes a long time to process but I might go with it as last resort, other than that I have no options.
I also thought of doing backup copy job, I was able even to make a script to force it to run active full monthly but since it takes over 1 day (interval) to process - it fails.

[HannesK]

because i have no idea if that will work ever at all.

well, for storage corruption we have the backup copy job (you already mentioned it) and the storage level corruption guard - always mind the 3-2-1 rule

In general, 365 days is not the longest chain I have seen at customers...technically not a big deal from a Veeam perspective.

If your source is too slow to finish active full in 24h, I can only suggest to split the active fulls in smaller jobs. Example: run 20 VMs active full on 1st, 20VMs on 2nd and in one month you will have backup for 600 VMs (just adjust the value to your environment)

[Berkovska]

I hear you, unfortunately there is 1 VM that takes up 80% of the backup size.
Does WAN accelerator help for Active full runs or it more for increments?

[foggy]

WAN acceleration is not available for regular backup jobs. Btw, this script allows running synthetic fulls on a less frequent basis.

[Berkovska]

Thats a great thing! I will sure try it - seems like its what I was looking for.
Thank you Hannesk

[olafurh]

Sorry for opening this thread up again "extremely high load on the backup storage" is a reason not to implement monthly synthetic fulls? Why would we transfer the load to the production storage, datacenter links, firewalls even WAN links? Is anyone thinking about network 95th costs?

Running synthetic fulls operation 4-5x a month instead of 1x a month, I'm not buying it that this takes less resources.

Or running a copy-job to achieve this, bit for a bit?

Now in 2021, we have ultra-fast backup storage systems, even all-flash systems/NMVe systems to act like a backup target, built up with amazing tech as ReFS or XFS ref link.

GFS are now an option in "normal" backup jobs, and more and more are moving the GFS operation from copy jobs to normal jobs, this would benefit a fast and modern backup backend.

Data corruption can affect all kinds of backup mode, incremental forever being one of them. Running health checks with Surebackup jobs should be how we check for corruption in 2021, not running active fulls, as data can come corrupt in transit.

Please bring us monthly (even yearly) synthetic fulls.

+1

[cpfleger]

Hm, I'd say that when a synthetic full backup has to check out the most recent data from not just 5+1 (5 increments + 1 full) but up to 29+1 restore points this probably makes some difference in load.
And the same doom (data corruption) you mention to maybe happen to forever fwd incremental chains lurks for synthetic full backup as well - the longer the period the more likely.

btw does restore happen faster with shorter chains.

Just my 2 cents - and a -1

[HannesK]

Hello,
@olafurh: if you use REFS / XFS... what's the point of not enabling synthetic fulls every week? It's for free on XFS / REFS. And what would be the benefit of monthly synthetic full in your use-case?

Agree that monthly synthetic full would work fine with REFS / XFS. But adding that option would cause much more trouble when customers start asking "why is my backup chain so long? I configured 14 days with monthly synthetic full and now I have 44 restore points and I'm running out of disk space".

So keeping weekly as requirement seems to have more benefit that adding a new scheduling option.

Best regards,
Hannes

[olafurh]

Hi and thanks for having this discussion, and 100% valid points from Hannes and cpfleger. But every coin has two sides..

One problem is policy, old policies that have been put in place for a long time, negotiating policies take time.

Other thing is that clients start asking "why do I have weekly fulls? I just want to keep 90 days incremental, 12 monthly fulls and 25 yearly."

This is now solved by having 90 days incremental, and a copy job for GFS. It gets the job done, but it has overhead in it. Think about 200 backup jobs, 200 GFS jobs and 200 Offsite jobs (copy or tape). This adds up quickly.

We could do 90 days incremental, GFS with Active full each month and then offsite jobs, but it creates unnecessary strain on production storage and DC links. My vision is to do this inside the backup environment without the Active full.

[HannesK]

Hello,

why do I have weekly fulls?

because it's possible and it doesn't cost extra The policy of 90 days is met, even if there are weekly fulls additionally now.

Having an extra backup copy job just to get rid of weekly backups sounds overkill to me. I would just ignore the fact that there are some additional fulls. I know customers that do daily synthetic fulls since years just to move backups to object storage "as soon as possible"

Best regards,
Hannes

[olafurh]

Then we have the problem about cost.. You have no-idea if the backup job (syntectic or not) landing on NTFS or ReFS, or is block cloned when you are calculating storage cost

[HannesK]

built up with amazing tech as ReFS or XFS ref link.

which "cost problem"? I'm confused. It's 2021 and I understood that you use block-cloning.

With NTFS, monthly synthetic fulls would cause performance issues. Even today, it takes ages to do synthetic full weekly on SSDs without fastclone. So monthly synthetic fulls could only be allowed on REFS / XFS anyway. Which again makes it pointless because weekly fulls are achieving the same.

You have no-idea if the backup job (syntectic or not) landing on NTFS or ReFS

it's 2021. NTFS is dead for Veeam repositories.

[olafurh]

which "cost problem"? I'm confused. It's 2021 and I understood that you use block-cloning.

In multilevel organization and MSP, you will need to provide cost-control/chargeback/show back of exactly how much storage (GB/TB) your org/department/tenant is consuming of total investment.

There is no way (as far as I know) to measure the benefits of a relink/block clone on the backup chain. It will always show up just as a synthetic full backup on a non relink/block clone FS when you measure the byte length of the backup chain.

[HannesK]

yes, showing the space saving of block cloning is a long term feature request from everybody who does chargeback. That's something we plan to improve.

But it's the same with monthly synthetic fulls and all GFS restore points. So I still miss the information which problem a monthly synthetic full really solves (except that it looks nicer having only the fulls a customer requested).

[robnicholsonmalt]

I guess you are using ReFS anyway. So space usage would not be an issue with weekly synthetic fulls.

Why is ReFS not an issue? I know little about the format. Deduplication?

[HannesK]

Block cloning / fastclone

https://www.veeam.com/blog/advanced-ref ... suite.html - same for XFS

[ebondi]

So I still miss the information which problem a monthly synthetic full really solves

Hi, in my case I've huge amount of data di backup stored on a file server and I manage local snapshots in order to keep last 60 days locally.
This is a disaster recovery server, duplication of production file server.

I backup those dr server files already splitted on several backup jobs in order to have less data to backup for each job, but the biggest amount of data for a single job is about 90 TB and incremental can take also more than a week to complete.
I would run 1 incremental backup at a month because I already have local snapshots on the dr server and it is enough, and I would like that incremental backup will be translated into synthetic full every time it run. In brief, 1 time at a month.

If the job runs every week (due to current limitation/specs of Veeam software) and takes longer than a week, the next week will not happen.

My GFS policy is setup to keep last 12 months backup on a SOBR that makes offload to AWS S3, then Glacier.

Making an example: GFS policy is setup to tag the Fourth week full backup as Monthly backup to keep for retention and archive. If incremental starts on the Third Week and will take longer than a week to complete, the next Fourth week scheduled job will not run and at same time the GFS policy will not tag any backup to keep for that month!

This is why I'm looking for a setting to create Synthetic full backup once a month, and NOT every week.

I see on Veeam 12 there is already a setting to reach this for the Active Full backup that allows to create Active Full backups periodically and specify Monthly schedule (not only weekly).
So why is it so difficult to add that option to Synthetic??

[HannesK]

Hello,

and takes longer than a week, the next week will not happen.

That sounds longer than we expect an incremental backup to run when designing the software. Can you maybe explain the reasons for long running incremental backups? Undersized hardware? Bandwidth? I mean, if an incremental backup takes more than a week, how long would a full restore take then? That sounds like months for restore

So why is it so difficult to add that option to Synthetic??

monthly synthetic full available in V12 for VMs. We added it "by popular request" which does not exist for Veeam Agent for Windows (if you are asking for that)

Best regards,
Hannes

[ebondi]

I forgot to specify that in my case I have Linux Agent Backup jobs. In those kind of backup it is not available the mentioned option to choose to make monthly Synthetic full.
I see that this option is available only on VMware Backup jobs!
So I really don't understand why Veeam cannot add this simple option to Linux Agent Backup jobs too.
Please, add it in next software updates.

[ebondi]

That sounds longer than we expect an incremental backup to run when designing the software. Can you maybe explain the reasons for long running incremental backups? Undersized hardware? Bandwidth? I mean, if an incremental backup takes more than a week, how long would a full restore take then? That sounds like months for restore

As said I've huge amount of linux file server data to backup, moreover we backup the disaster recovery server in order not to bother production servers. Dr server is slower and is for DR purposes.

We use Veeam to offload data on cloud object storage repository in case of ransomware attack and offsite backup store. In case of restore we will save back data on production server.

We need just one incremental backup at a month in order to offload data from main site just 1 time at a month for those huge amount of data, and at same time we cannot have complex Veeam backup Jobs configuration: just that option to create Synthetic full 1 time at a month also for Linux Agent Backup job!

[HannesK]

about performance: doing a 90TB full backup in 24h requires less than 10Gbit/s. That sounds reasonable in 2023. For incremental backup it depends on the configuration. File-based backup is probably too slow (sounds like what you see). With volume-based backup there would be change block tracking (CBT) available but it gets reset after reboot. If I assume a 10% change rate per week, that would be 9TB. 9TB in 24h would be less than 1Gbit/s.

[ebondi]

Sorry but why you drive speaking about other... Ok, I answer you also on this: I make filelevel backup due to the file servers have zfs file system unsupported by Veeam for block based backup (I know from Veeam support I can do it) so it is slower.

Now: here the need is to have just that option Once A Month also for Linux Agent based backup jobs, nothing more! The action to configure scheduling jobs is the most personal customization that backup operator customer can do!Forcing me to make synthetic full every week and not also once a month is like force someone to choose black haired woman when he like blonde ones.
The worst is that option is already available for Active Full in agent based backup jobs and for all kind of VM based ones.

Please, please, please

[HannesK]

I'm mentioning it because in many cases we get feature requests that are caused by misconfigurations / design issues. We only want to add functionality that is useful and not something that only exists because of misconfigurations.

The root cause seems to be backups running longer than a week. So there seem to be two options how we can solve it: make backup faster or add the option you ask for (or both )

With the ZFS and file-based backup explanation the request makes much more sense now. Thanks!

[ebondi]

So, can I expect in next Veeam software release/version to have this setting to create Synthetic full backup once a month, and NOT every week?