V10 & XFS - all there is to know (?)

[crackocain]

Hello Tom. For example, i assume Veeam compression applies 4bit, not 1MB

1 Data block compression applies 4 bit
ABAC-BBBB-CBAC-DDDD-FCAB-GGGG. B, D and G blocks are repeating in same 4 bit block. So Veeam compress data. ABACBCBACDFCABG final result

But one week later Veeam Proxy process this data different alignment. Veeam chunk has different block start data. B at first

BABA-CBBB-BCBA-CDDD-DFCA-BGGG. 4bit compression can't applied so final data is the same output. Because new block has no repeating pattern. You see B, D and G blocks aren't in 4 bit compression block. So Veeam Proxy thinks "this is new data i can't use Block Clone".

But compression disable scenario Veeam clone every block. Because compression changes nothing. Only incremental changes are consume space. Maybe i'm wrong

[Gostev]

Why would Veeam Proxy suddenly process the same VMDK block using a different alignment a week later? This simply cannot happen. So, unless the VMDK block's content actually changes, then source data will remain the same throughout all future runs. And as Tom correctly stated, whether compression or encryption is used does not make any difference to block cloning, because it uses a hash of the original raw data of the block, and does not care about the content of the block stored in the backup file (compressed and/or encrypted).

[crackocain]

No, they can't. Fast cloning is available for image-level backups only, which plug-in backups are not - they are streaming backups.

Hello
Right now we could use backup copy for application files (SAP HANA for example).
Could we send via backup copy RMAN and HANA backups to XFS immutable repository. Could we use XFS block clone? Backup is not stream with this scenario.
This feature is very good for many of customer. Everyone ask immutable repository for SAP HANA & Oracle workloads with Veeam.
We can't use VM image backup because of customer&SAP consultants requests.

[Gostev]

XFS block cloning specifically cannot be used regardless, because application backups are not image-level. They are still streams not images, you just make a copy of them.

If you are asking if you can use the same repository to host them anyway, without any special XFS integrations - the answer is YES.

If you are asking if application backups support v11 immutability feature - then I need to defer to @Andreas Neufert for the answer.

[Andreas Neufert]

XFS Immutability will not used for *.vab Application Backups (SAP HANA / Oracle RMAN) and Image Backup based Log Shipping *.vlb files.

Application Backups (SAP HANA/Oracle RMAN) can use Immutability defined in Scale-out-Backup Repository Capacity Tier (object storage).

[jasonede]

We're using XFS on ubuntu and are really happy with the performance. We were looking at REFS on Windows Server 2019, but this way saves on the windows licensing, has less overheads and performs well. We've only about 60TB of data though on this repo.

[crackocain]

XFS Immutability will not used for *.vab Application Backups (SAP HANA / Oracle RMAN) and Image Backup based Log Shipping *.vlb files.

Application Backups (SAP HANA/Oracle RMAN) can use Immutability defined in Scale-out-Backup Repository Capacity Tier (object storage).

Thank you Andreas. But I need you to clarify something.

Application backups can use XFS immutable repositories only immutable function not worked.
or
Application backups can't use what so ever XFS immutable repositories.

Because if can't use we need add new repositories.

BTW, streaming type can't use BlockClone but is immutability is in the roadmap for XFS repository?

[Andreas Neufert]

Both vlb and vab based backups can write to the Immutable XFS but will not set the XFS filesystem immutable flag.

[mkretzer]

Why?

[Gostev]

I'm not surprised because the whole immutability feature was the very last minute idea that came to me towards the end of v11 development cycle. So normally it should have been v12, but we were able to squeeze it into v11 for image-level backups, because it is such a big deal (and because image-level backups currently represent >99% of all backups created by Veeam users).

For example, I did know right away that the NAS backup team will not be able to support immutability in v11 already - actually it would require some significant architecture changes from them to support. I just was not sure about the application plug-ins team... but now we have the answer.

[crackocain]

Thanks Anton, immutability is very popular my customers.
Also XFS and ReFS blockclone. This two is deadly weapon

Maybe backup copy process change the type stream mode to image mode by Veeam? Is there any possibility? Special backup copy process for streaming backups?

[Gostev]

Honestly, I can't be sure if there are any benefits to using block cloning for those specific backups in the first place... we will need to research if they actually do have repeating matching blocks of fairly large size that we use, as without this block cloning integration makes zero sense.

[pirx]

I'm not sure if I should open a new thread for this, so I'm just posting it here....

As our NetApp + CIFS setup is not working very good for any synthetic operation, we've to think what we could use instead. I'm quite sure that it will be ReFS oder XFS based. I know the famous ReFS thread, I also know that XFS is probably not used as much as ReFS. Personally I would prefer XFS anytime as I was using it a couple of years for large filesystems and it didn't fail me.

Can some of people that are already using XFS with reflinks share their setups?

- How much data is stored, what retention time?
- How long is this setup already running, any problems?
- What storage do you use? Apollo like servers with local disks or SAN based?
- which kind of disks?
- which Linux Distro?
- how much space do you safe by using reflinks?

Our primary storage (400TB) is also CIFS based but with flash, so this is working ok for backups and we probably can live with it. But the secondary storage is all Nearline SAS (2PT) and there is no option to switch to AFA. Budget is currently an issue because a replacement is not planned for next 12 months. So we have to check if we you could improve situation with a limited budget, like Apollos + N-SAS disks. But this makes only sense if it solves our problems. And yes, we can approach our vendors, but I trust the community here a lot, and like to get some real world feedback.

[Andreas Neufert]

Regarding the block cloning for RMAN/HANA/BRTOOLS. The challenge here is that we are not the backup application and "just" forward data (compressed sometimes encrypted) and write it in our own backup file container. The Plug-in APIs from those vendors do not allow us to have the same visibility into the data format as with our image level backup. However we are speaking with the vendors on this topic. I think the best way to get additional data reduction is to use deduplication appliances like Exagrid that have as well backed in immutability that workd transparent for Veeam.

[orb]

I'm not sure if I should open a new thread for this, so I'm just posting it here....

As our NetApp + CIFS setup is not working very good for any synthetic operation, we've to think what we could use instead. I'm quite sure that it will be ReFS oder XFS based. I know the famous ReFS thread, I also know that XFS is probably not used as much as ReFS. Personally I would prefer XFS anytime as I was using it a couple of years for large filesystems and it didn't fail me.

Can some of people that are already using XFS with reflinks share their setups?

- How much data is stored, what retention time?
- How long is this setup already running, any problems?
- What storage do you use? Apollo like servers with local disks or SAN based?
- which kind of disks?
- which Linux Distro?
- how much space do you safe by using reflinks?

Our primary storage (400TB) is also CIFS based but with flash, so this is working ok for backups and we probably can live with it. But the secondary storage is all Nearline SAS (2PT) and there is no option to switch to AFA. Budget is currently an issue because a replacement is not planned for next 12 months. So we have to check if we you could improve situation with a limited budget, like Apollos + N-SAS disks. But this makes only sense if it solves our problems. And yes, we can approach our vendors, but I trust the community here a lot, and like to get some real world feedback.

Implementing NetApp with CIFS and a synthetic method isn't the most efficient way. We tested a NetApp FAS2720 out of curiosity made of 12*10TB (8 spindles, raid-tec + spare), a SOBR composed of 2 NFS (same volume, 2 qtrees), in-line compression, post-compression and deduplication on the volume. The storage manages to keep over 3000 IOPS and about 250MB/s of data ingest from 2 VM proxies with 8 vCPU. As long as you have 10GbE links, per-vm and decompress/ align block and use full active job only, you do fine. we were surprised we manage to reach 50% space-saving just with the in-line compression.

The synthetic approach will shine on a pure block approach (SAN, iSCSI) with ReFS or XFS. It is maybe a bit of waste while using a FAS model from NetApp especially because your increment won't be deduped at the volume level.

Oli

[Andreas Neufert]

I am not sure about this setup.
Are you saying you use uncompressed data to write it into the FAS2720 and then get 50% reduction with inline compressions? This is expected.

When you would use the Veeam compression instead of the FAS2720 one you will end up with 2x the backup target performance as the target has to handle only half of the data.

As well make sure that the post dedup run is scheduled to be run outside of the backup window.

I would always use in this setup iSCSI + XFS block cloning with Veeam compression. If you really like you can enable deduplication within the storage afterwards. Not ideal but it should give you good performance with good space savings.

[orb]

I am not sure about this setup.
Are you saying you use uncompressed data to write it into the FAS2720 and then get 50% reduction with inline compressions? This is expected.

I was not sure either but very curious about it

When you would use the Veeam compression instead of the FAS2720 one you will end up with 2x the backup target performance as the target has to handle only half of the data

The source is the bottleneck and throttling pop up quickly since they have a high average load. Write speed averages 100MB/s and Network bandwidth isn't really an issue.

As well make sure that the post dedup run is scheduled to be run outside of the backup window.

I would always use in this setup iSCSI + XFS block cloning with Veeam compression. If you really like you can enable deduplication within the storage afterwards. Not ideal but it should give you good performance with good space savings.

Yes post-dedupe,post-compression outside the hours and set on the best effort so It should not impact performance much even if it runs at the same time. I did that test before, it gives you a little extra 15-18% indeed to enable post-dedup with block-cloning.

I tend to use block cloning as well because e don't have many customers who can afford FAS type as a backup repo and when they do we have a Snapshot, Snapmirror or Snapvault involved at some point.

The "plus" of this approach is we get back dedupe/compression lost by per-vm settings since it is applied a volume level. The trade-off is your basically cut in half your maximum write speed as you mentioned it, no Synthethic allowed and you need to keep enough space on your volume for the post-processing.

I will be happy to share some data later.

Oli

[kspare]

Is it possible to uncheck the use fast cloning box, run an active full backup and just use xfs without fast cloning, but still have access to the older backups that used fast cloning?

[orb]

Is it possible to uncheck the use fast cloning box, run an active full backup and just use xfs without fast cloning, but still have access to the older backups that used fast cloning?

@kspare
Fast block cloning is only available on a full synthetic operation, so per definition an active full ignores fast block cloning.
You can have a synthetic full and an active full in the same job as long they don’t run the same day or you always have the option to trigger the full active manually on a job via the context menu.

Oli.

[kspare]

we want to *stop* using fast cloning for our backups and only use them for back up copy jobs. we're finding that our jobs are actually slowing down. So if I uncheck that box, and run an active full, all the reverse incrementals going forward will not use fast cloning, but I can still access the old jobs? or do I even need to run a new active full after I uncheck it?

[orb]

@kspare,
I would recommend you create a separate post for that, even better open a call. It sounds you have more a setup problem / job configuration situation than a block cloning one.

Here you give only half of the story

Oli

[evilaedmin]

...we're finding that our jobs are actually slowing down....

Disclaimer: Not a block clone user in either ReFS or XFS

But one impression I came away with in the ReFS threads is that when it comes to "fragmentation" there was no free lunch: doing block clone operations over time would lead to fragmentation (from the perspective of the spinning rust), operations that used to look like sequential reads would be random reads, slowing down synth fulls. So spindle counts and overall random read performance is important to a healthy synth full implementation using block clone.

Is it a fair assumption that this would be true for XFS also?

[Gostev]

Yes, XFS is no different from ReFS in this regard.

Backup file fragmentation is of course real, since when multiple backup files refer to the same shared blocks, it is impossible to have them "defragmented"

"Slowing down synth fulls" part is NOT true, as on ReFS/XFS this is a metadata-only operation which doesn't do ANY actual data movement. In fact, synthetic fulls take no physical disk space on ReFS/XFS, so there are no physical reads/writes of backup data blocks to start with. As such, spindle counts and overall random read performance cannot possibly matter much for the synthetic fulls performance (unless it is so bad that it impacts file system metadata updates).

[kspare]

My issue right now is that the reverse incrementals have slowed down now that we are about 60 days into XFS with linked clones. We have all our jobs set to do health checks and file maint ever 3 months, so that hasn't happened yet.

We also do a backup copy once a week to another nas and this has gotten pretty slow. but we don't even do synthetic fulls. But I don't think we need to because we are doing RI?

[kspare]

our storage servers are synology RS3617RPxs with 11 8tb barracuda pro running raid 5, and 2 1tb ssd cache drives in read only mode with 10gb networking...it really shouldn't be this slow. but it keeps indicating that the source is the bottle neck with the new xfs link clone volume....

[evilaedmin]

"Slowing down synth fulls" part is NOT true, as on ReFS/XFS this is a metadata-only operation which doesn't do ANY actual data movement. In fact, synthetic fulls take no physical disk space on ReFS/XFS, so there are no physical reads/writes of backup data blocks to start with. As such, spindle counts and overall random read performance cannot possibly matter much for the synthetic fulls performance (unless it is so bad that it impacts file system metadata updates).

Apologies for my mistake. Then @kspare 's issue, are reverse incrementals expected to be completely metadata-only? Trying to understand the use of the quote word "injects" here

During subsequent backup job sessions, Veeam Backup & Replication copies only VM data blocks that have changed since the last backup job session. Veeam Backup & Replication “injects” copied data blocks into the full backup file to rebuild it to the most recent state of the VM. Additionally, Veeam Backup & Replication creates a reverse incremental backup file containing data blocks that are replaced when the full backup file is rebuilt, and adds this reverse incremental backup file before the full backup file in the backup chain.

[Gostev]

No - not completely, because the reverse incremental backup mode does two things:
1. VRB file creation: this is metadata-only, because what it does is moving the existing blocks which are already on disk (currently in a VBK file) into the newly created rollback file.
2. VBK file update: this, on the other hand, does involve physically writing (injecting) new blocks into the full backup file, as this is the brand new data captured from the production environment.

To be honest, it's a terrible backup mode for fragmentation regardless of the file system. Our own admins had an Exchange VM restore speed issue on NTFS due to it about 10 years ago, restoring from the reverse incremental chain that did not see an active full or a compact/defrag operation (this feature did not exist at the time anyway) for over a year. They did an active full after, and in the following test the restore performance improved like 10x. Keep in mind Veeam was quite small at the time, and we used pretty shitty backup storage

[kspare]

are you recommending that I switch away from reverse incremental s for my larger customers?

[Andreas Neufert]

Reverse Incremental is the oldest Veeam backup modes and addressed some specific situations and technologies available back in time. Now with the storage system evaluation and that Veeam has implemented Forever Forward Incremental mode plus block cloning the situation has changed.

Forever Forward Incremental backup has some advantages in bigger environments as the Proxy/Repository task slots are only blocked for a VM for the time of a incremental backup (1 IO) while they are blocked for the whole processing with Reverse Incremental (3 IO per Source block). A lot of customers were able to reduce the backup window siginificantly by the change from Reverse Incremental to Forever Forward Incremental (Incremental without selected Synthetic or Active Full).

As well it has some additional advantages mentioned by Anton above.

[Seve CH]

My issue right now is that the reverse incrementals have slowed down now that we are about 60 days into XFS with linked clones. We have all our jobs set to do health checks and file maint ever 3 months, so that hasn't happened yet.

If you have XFS and it supports reflink/block-cloning (? linked clones is something else), why are you using reverse incrementals? Use frequent synthetic fulls instead. With block cloning they are "free" .

Anyway, whenever I find a reverse incremental job, if I have enough space I switch that to standard incrementals. Reverse incrementals take forever and overload your storage. It is practical to have the latest backup as a single logical file, but that was IMHO the only advantage before block-cloning arrived.