Forever Forward with Active Full

[lobo519]

I love veeam - I really do

But do anyone else find it impossibly frustrating that you can't do a periodic active full with forever incremental and have it follow the actual retention policy??

Wish I had the option to count the previous chain in the number of restore points and the old incrementals would just continue to merge into the old full.

[Gostev]

Well, the challenge here is that one of the most typical use cases for Active Fulls is when storage is too slow to handle full backup transformations aka merges.

What is your use case for using an Active Full with forever-incremental chain? May be I will be able to suggest the alternative.

[lobo519]

I like the warm fuzzy feeling of a periodic active full to protect against corruption.

Ideally - Forever incremental with quarterly active full while maintaining the correct number of restore points.

I know corruption guard exists but is quite slow and prevents jobs from running. I would rather spend that time getting a fresh active full than getting no backups at all while the health check runs. Also trying to run a synthetic full + roll back + health check in a weekend is generally not enough time.

I'm open to suggestions.

[spiritie]

That's why Sure Backup exists

[lobo519]

Which is great - but not really the same thing.

[Gostev]

Indeed, not the same thing - SureBackup is better! Just making an Active Full does not automatically guarantee it's recoverable... only SureBackup can guarantee the recoverability. Thing is, your Active Full may potentially already be unreadable from storage the moment it lands there, if storage is experiencing silent data corruption issues. So you can do an Active Full every day, and not one of those backups will be recoverable.

[Gostev]

Ideally - Forever incremental with quarterly active full while maintaining the correct number of restore points.

How about this idea: a separate job running quarterly and doing Active Fulls. Ideally to a different storage device, for added warm fuzzy feeling (this way you will not have all your backups hit at once by some bug in a RAID controller).

[lobo519]

Fair statement regarding Surebackup - but still I still stand by my request.

Not a bad idea running a separate job - still could have corruption on the forever chain

Additionally - you are saying I can run surebackup - not the health check - and the backup is guaranteed good running forever incremental?

[foggy]

Nothing can be guaranteed, since backups can be corrupted by the storage device itself upon landing. But with SureBackup, you will immediately know that your latest backup is not recoverable.

[lobo519]

can't Surebackup with agent jobs right right?

[foggy]

Could you please re-phrase the question?

[lobo519]

surebackup can't be used to verify veeam agent for windows backups still correct?

[Gostev]

Correct, it's not possible currently, but is something we're working on.

[lobo519]

There's that active full request creeping in again

[dimaslan]

The only solution I see is ReFS with weekly synthetic full backups.
The benefits are:
- ReFS does not have the 15.6TB limitation per volume NTFS has.
- Backup synthetic operations are done almost instantly (synthetic full backups can take <1 minute to create)
- Synthetic full backups are created only by writing new blocks and updating metadata for existing same blocks, so new Synthetic full backups take roughly the space of an incremental.
- ReFS filesystem has native error detection

What you need:
- A backup server or backup repository server with Windows 2016 or newer, fully updated.
- Rule of thumb is 1GB of RAM for every 1TB of backup space (excluding OS ram). Metadata operations are handled by memory, so if you have a 20TB backup repo with only 8 GB of ram on the server, it may result in issues with updating metadata.
- Make sure to format ReFS with 64k size blocks, not the default 4k.

You cannot have synthetic fulls further apart than a week, so if you lose something it should not be larger than a week's worth of backups.

[lobo519]

I was looking into switching to ReFS over the weekend to take advantage of the clone feature.

Biggest issue I have is the whole chain needs to be on a single volume - I have 3 volumes in a scale-out repo.

I know I can use data locality but it doesn't guarantee and limits the flexibility some.

[lobo519]

The only solution I see is ReFS with weekly synthetic full backups.

I have decided its not currently possible to forklift to ReFS and there seem to be quite a few issues out there with reliability.

I will restate my opinion that we should have at least the option to have active full without affecting the retention policy.

What was the reasoning behind not having this behavior?

[foggy]

The reasoning is explained in the first Anton's post above - it's storage performance considerations in the first place (identifying possible adoption rate).

[lobo519]

So catering to smaller customers with slow storage?

I'm not trying to start a war - but that's your statement says to me.

[Gostev]

Catering to customers with slow storage, which are common in every market segment, because people tend to stuff their backup storage with big fat slow drives. Periodic active fulls with classic retention (no merges) is the only way to achieve decent backup window with such storage.

But in any case, ReFS/XFS is the future (along with object storage, where we work the same way). The benefits of "nodupe" are just too overwhelming.

[lobo519]

Fair enough.

I wish ReFS gave me the warm fuzzy feeling but it doesn't - even talking to some veeam employees they don't recommend ReFS due to reliability.

I Appreciate the conversation!

[tsightler]

It's certainly true that ReFS has experience it's share of issues, as does every new filesystem, and ReFS pretty much did trial by fire with thousands of Veeam users using it nearly immediately with large volumes and high amount of cloned blocks stressing ReFS in significant and perhaps somewhat unexpected ways immediately out of the gate. Add to that that Windows hadn't had a new filesystem in decades, and the kernel had been highly tuned for NTFS behaviors over that time, it was almost sure that ReFS would have some issues.

That being said, even back almost 2 years ago, an analysis of ~13,000 Veeam instances showed that ~17% of those environmenst were using ReFS repos (this was shared in a Gostev newsletter years ago). When applied to our entire customer base, that would extrapolate to around 50,000 customers using ReFS, the vast majority with no issues and another smaller, segment with only minor, performance related issues. In the 2 years since this ReFS has stabilized significantly and it's quite rare for me to work with customers that aren't using ReFS unless they are backing up directly to a dedupe appliance. Note that the vast majority of clients I work with have PB's of repos all running ReFS.

I'd say that ReFS on 2016 has been quite solid for a while now. Windows 2019 had a lot of fixes and performance enhancements for ReFS which led to a resurgence of some issues when it was released, but Microsoft has worked quite diligently to solve those and 2019 ReFS is now running better than ever as far as I can tell.

Perhaps this still won't give you the warm and fuzzies for ReFS, and I can understand that, but any new technology can have it's bumps. I actually think ReFS held up pretty well as an entirely new filesystem looking to break the scale limits of NTFS and I have no doubt it will continue to improve going forward.

[Jeff M]

I can confirm. As an end user, I have been using REFS from the beginning. I had many open tickets and several forum posts due to REFS issues. Those have all become a thing of the past. As tsightler stated, Server 2019 REFS is now better than ever. In fact NOW Veeam and REFS repos give me lots of warm fuzzies. And yes we too have PBs of repos running REFS.

[mkretzer]

Can confirm, we had more issues then we can count in the beginning, but everything is running perfectly now (2019 and 1903/1909).

What gave me a good feeling was working with the ReFS team of microsoft to fix these issues. This team is really dedicated in their work (i am not the biggest MS fan but i am a fan of this development team now).