V10 & XFS - all there is to know (?)

[Gostev]

I would like to add to what Andreas said above.

Veeam started as a very-SMB vendor 15 years ago, and reversed incremental backup mode checked all the boxes for vSMB customers:
- No multiple full backups (because they didn't have enough storage to store for those)
- Latest backup file is always the full backup file (very easy to manage: copy, transport etc.)
- Predictable backup window (each run is incremental, so there are no days when backup takes 10x longer due to the full backup)
While performance did not really matter as much due to the size of those tiny environments, and large backup windows available since almost no SMB business requires that performance SLAs are met 24/7. Also, retention policy were typically extremely simple in these environments: practically always just last 7-14 days worth of backups.

However, as Veeam moved up market, we saw virtually all larger customers requiring GFS retention due to internal policies or external regulations. This is when we started looking for an space-efficient way to store those multiple GFS full backups, and as the result came out with the whole ReFS/XFS integration and fast cloning.

This is the very reason why the current default backup mode is forward incremental with period synthetic fulls. We also guide users to ensure they leverage fast cloning, for example you will get a warning if you create a Windows-based repository on a volume that is not ReFS.

Having said that, even back just a couple of years ago @tsightler liked to repeat how at least some of his largest enterprise customers love reverse incremental backup based on their backup storage and other requirements, and that we should not even think of discontinuing it I wonder if ReFS/XFS has changed this lately Tom?

[orb]

our storage servers are synology RS3617RPxs with 11 8tb barracuda pro running raid 5, and 2 1tb ssd cache drives in read only mode with 10gb networking...it really shouldn't be this slow. but it keeps indicating that the source is the bottle neck with the new xfs link clone volume....

11*8TB / RAID5,seagate barracuda which are desktop class drive. I hope you have a second copy somewhere.

Oli

[tsightler]

@Gostev Since block clone was first introduced over 4 years ago and has reached quite high levels of stability for most deployments over the last 2-3 years, save some pretty good bumps with Windows 2019 early adopters, forward incremental with synthetic full is the go-to recommended architecture, perhaps occasionally with straight forever incremental based on retention requirements. At this point I personally can't think of any large clients still using reverse incremental as a matter of course, and most of the corner case uses that did exist (for example, offloading daily fulls to tape) have been addressed. I consider reverse incremental a legacy backup mode at this point and it's certainly not part of our standard designs.

[lolbebis]

We are using reverse incrementals to flash disk and tier the increments to on prem S3 that means we can increase the retention almost litmitless without using more space on the primary flash backup system. (The S3 system has lots of disk for PACS images so our backups are more less a drop in the ocean).
I think that works very good for us.

I dont know how implementing forward incremental with synthetic full would affect that setup, but I asume that we would have to send full backups to the S3 storage aswell then?

[aich365]

We use Reverse Incrementals where customers want a Daily Tape backup - This ensures the latest RP is copied without having the overhead of synthesizing

[Gostev]

@lolbebis backup mode makes no difference to the object storage offload engine, it is always forever-incremental. You would not be able to "send full backups" even if you wanted.

@aich365 there's definitely no "overhead of synthesizing" in v10, and in fact it is completely the opposite: this approach performs a few times faster than copying an existing backup file to tape. But in any case, reversed incremental is not the requirement for your approach, as any backup mode with periodic fulls would do - you just have to time those fulls to the day when tape export happens.

[aich365]

Hi Gostev
Thanks for the response. The problem with the tape job waiting on a "periodic full" is if the PF fails to be created the Tape job will try and synthesize the last RP (or sit waint for the PF until it times out.
In a Reverse Incremental job the last RP is always a Full.

Regards

[tsightler]

Daily full backup to tape is indeed the most common use case I've seen for reverse incremental these days, but synthesized full to tape daily has largely addressed this use case now that it performs so well in v10 and beyond. You don't have to synthesize the full on disk for this function to work.

[DonZoomik]

IMHO reverse incremental also works better for long chains (100+ RP) without sythetic fulls. With forever incremental, jobs spend a looong time (and a lot of read IO) reading embedded metadata in VIBs and VBK, before the job actually starts. With some basic testing, this start slowdown was noticeable at already 20-30 RPs (if I were to do weekly synthetics) - just some numbers from top of my head, don't remember details. I had a support case for this when forever incremental job did several times more read IO on repository than total writes.

[furyflash]

Hi,
I am trying to convert traditional repository to XFS (Fast clone). We have one backup job with 14 TB full backup file and 30 incremental files 1-2 TB size. We can't create active full backup for this job (slow channel). we are trying additional option with compact full.
Following on recommendation of the veeam support. we have moved full backup file to the XFS repository and run compact full file. It takes 8 days and didn't help. Then veeam support said what we need to move all incremental backups too.
When I copied files to the XFS repository, it's ready for block cloning. I checked it with linux commands. But we need to run compact backup for get some flag on job, why?
Is it possible to change this flag somehow in job file and don't run compact or active full jobs?

[JTT]

If i deploy a Ubuntu 20.04 LTS repo with XFS today for Veeam V10, then does it mean, that after V11 is released, the same repo gets immutable option automatically or should i redeploy the repo? My main goal is to know, does the Linux repo get upgraded with immutable option or should i wait for the V11 release?

[Gostev]

You can go ahead and deploy now. There will be a path to convert it to a hardened repository without re-deploying. If I remember correctly, you will need to register the server again with Veeam using the new "single-use credentials", plus run a console command to change existing backup files ownership to the limited account under which the newly deployed persistent components will run. So fairly quick and simple process.

@HannesK sounds like a how-to forum post or a KB article is due!

[Riley]

So I found the command like to create the xfs file directory from veeam. What I have not found is the underlying systems that peeps are putting xfs on. i.e. processors, raid, etc. I am Exceptionally excited about immutability in 11 and I want to get a head start on creating a system. Those of you with XFS already, what did you set up?? Or is there a post somewhere I am not finding on systems being used??

[Gostev]

Any general-purpose server that meets our System Requirements for backup repositories will do, so everyone is using whatever they have
Here's the big thread that span many years with people sharing all the different options (XFS does not bring any special requirements).

[evilaedmin]

XFS on top of ZFS volumes (zvol) or is that just asking for trouble?

[Seve CH]

Those of you with XFS already, what did you set up??

During testing, whatever we could found

In production, we run on the iron directly. We wanted to be able to interchange the hardware of the Veeam Server and the Linux repo if we needed to, so the repo server is a bit overkill.
OS: Ubuntu 20.04 LTS Multipath+LVM2+XFS.
Server: HPE DL380 Gen10, 2xIntel Xeon Silver 4215R (2x8 Cores @3.2Ghz), 128GB RAM (1x DIMM per channel-> 2x6 DIMMS @2933Mhz)
Storage: Dell ME4012 with 2xSAS 12Gbit, 30x HD 16TB (480TB RAW) NL-SAS (7200RPM) in a single pool/disk group with dynamic RAID (ADAPT). This is the standard 10x Data + 2x Redundancy + 32TB Spare capacity (=2 disks).

[nitramd]

XFS on top of ZFS volumes (zvol) or is that just asking for trouble?

Eugene,

Take a look at this thread - it should answer your question: servers-workstations-f49/v11-and-linux-t71075.html

Mr. Sightler is testing what you're asking about.

[tsightler]

I actually have a full presentation on XFS on ZFS at last years VeeamON. Admittedly this was prior to v11 and the hardened repository, but I still think there can be some additional value to combining both options since ZFS can effectively protect the entire repository and revert it to a prior state easily. The biggest issue with XFS on ZVOL that I've seen is the performance characteristics of ZVOLs themselves. If you can design a ZVOL solution that meets your performance requirements, it's an excellent solution.

[orb]

and still one of my favourite! I agree with the performance bit. ZFS or CEPH are great tools but you need to know your subject from the hardware, storage to application. Today, they are not out of box Veeam ready solution but I am confident we see soon some experience shared on the matter.

Oli

[pirx]

I see that RHEL 8.2+ is supported for reflinks. I also see that it comes with an rusty 4.18 kernel, which means no iomap which was introduced in 5.4 (https://blogs.oracle.com/linux/xfs-data ... ng-reflink). I can't see if iomap was backported to RHEL 4.18 though (anybody knows how to check this?). At least there is an elrepo 5.4 kernel https://elrepo.org/linux/kernel/el8/x86_64/RPMS/

How important is kernel 5.4/iomap for reflink performance? As RHEL is the distribution that we use for all linux based servers, Ubuntu or any other disti will not be possible.

[Gostev]

Testing to this point has shown only minimal difference with kernel 5.4 for most real-world Veeam use cases. So unless you are planning to do something extreme, like keeping a large number of synthetic GFS points, there should not be much difference and earlier kernels will likely meet your performance requirements too.

[pirx]

What could me more extreme than using SMB protocol for backing up PT of data...

I guess we are not average size, 1600 VM's our copy repository size would be ~400-500TB, maybe with S3 offloading, but this in mainly done on primary backup target. 10 weeks retention with 14 dailies. Retention time might change and be longer in future. What would you call a large number of GFS points?

[Gostev]

Well, it would be unfair not to call your deployment large but the GFS schedule on the other hand is quite short. So from what we know now, I think you will be fine.

[pirx]

According to the the Apollo/ReFS thread 192GB RAM would be enough for a 500TB repository with ReFS. What about XFS? Is this also a reasonable number for xfs with reflinks?

[Gostev]

Neither ReFS nor XFS have special RAM requirements associated with them. Just make sure you meet the system requirements for Veeam roles you're going to deploy on the server, and keep in mind they vary depending on the number of concurrent tasks you want to run.

[pirx]

I remember the rule of thumb 1 GB RAM per 1 TB used. We're are talking about two systems each with 2x26 cores as copy targets, it's hard to compare this with our current setup with 2x10x100TB SMB shares where every extent has 30 concurrent tasks configured. The gateway servers are used as proxies too (7 servers ~120 cores for everything).

[Gostev]

I remember the rule of thumb 1 GB RAM per 1 TB used.

Me too, but I think I still had hair when this rule existed it was a temp workaround for a Microsoft bug that was fixed a few years ago.

[tsightler]

Also, a lot of that memory recommendation on Windows was attempts to work around the crazy Windows buffer behavior, which was about 10x worse with ReFS due to various bugs. Now, with v11 bypassing the OS buffer for writes on Windows, memory usage should be more normal. Still I would never recommend to go with less than 4GB per core recommended in best practice, and, arguably 4 GB per planned task (generally core = task, but if you want to oversubscribe cores on your repo, having more memory is the best way to make sure you can do this). Admittedly, 4GB is probably overkill, but if your environment is large, VeeamAgent process can grow quite a bit larger. There can be quite a difference in memory usage when you are backing up 50 VMs with 100GB disk vs 50 VMs with 4TB of disks.

I can pretty much say this, I've had far more customers regret not putting memory in the box vs the other way around! If you want to have less problems, don't skimp on it. Admittedly, I don't think you need to go overboard either. I've had customers buy boxes with ~1TB of RAM, which seems maybe the other extreme! If I was buying a big box like that, I'd probably look at no less than 256GB if it's repo only, 384GB if it's proxy+repo.

Note that the customers I work with are on the larger side, almost all have >10,000 systems being protected, some a LOT more, and many have dozens of PB of data under protection, but the lessons are mostly valid for customers of all sizes.

[dcolpitts]

So to share my XFS setup, I just converted a very small customer from a DL380 Gen9 running Win2016 using BackupExec 20 (to RDX only) to ESXi on a DL380 Gen10 (1 x Xeon 5218, 192GB, 8 x 960GB SSD in R6) with 536FLR-T 10GbE adapter, RDX (via USB passthru to the VBR VM), a Buffalo Terrastation (TS4510R with 4 x 8TB drives in R5) and of course VBR now. The TS5410R has as single 10GbE port and 2 x 1GbE ports. In ESXi I created 2nd vSwitch with just one port of 536FLR-T in and I cross connected the 10GbE port in the TS5410R to it. I created a Ubuntu VM (Server 20.04.2 LTS, 2 vCPUs, 8GB RAM, 40GB thin provisioned drive) with two VMXNET3 NICs - one connected to each vSwitch (VLAN0 and the 2nd one connected to the TS5410R). I provisioned the entire TS5410R as iSCSI and presented it to the Ubuntu VM, which I then formatted with XFS. I then presented this as my Linux repository with immutability.

The customer has around 940GB of data at present in 13 VMs (which is crazy since there was only a single Win2016 in the environment until I came along a few weeks ago). The initial full on the first day processed 940GB, read 904GB, and transferred 704GB in 59m49s at a rate of 280 MB/s. Daily incrementals are around 15GB in 28 minutes at 640 MB/s. Friday night synthetics are completing quicker than I can get up, go to the fridge and get a beer and get back to my desk to see it end (3/13/2021 9:28:25 PM :: Synthetic full backup created successfully [fast clone] - 00m53s).

I currently have 19 restore points so far (I only installed this a couple weeks of ago), and my repository is reporting 2.3TB space used (as shown from the Backup Repositories screen), while my capacity is 16.3TB and my free is 15.2TB (so 1.1TB capacity is used, also what shows when I do a df -h in the Ubuntu shell).

I also keep the the two 1GbE switch ports disabled that the TS5410R is plugged into, so it's management interface is not accessible on the LAN unless I specifically SSH the switch and enable them. In testing (before delivering to the customer), I deleted the Ubuntu VM (as if I were an attacker), and I was able to reinstall and gain access back to the iSCSI volume within 20m, and from there I was able to access my backups.

Overall, I'm pretty happy with the performance so far, and it was a cost effect solution for the customer on a limited budget (normally I'd just toss a StoreOnce 3520 at it, but $3000 vs $30000 makes a huge difference).

I am however currently struggling to figure out how to display the XFS statistics for dedup though from the Ubuntu console - if anyone knows and would share, I'd appreciate it.

dcc

[soncscy]

>(normally I'd just toss a StoreOnce 3520 at it, but $3000 vs $30000 makes a huge difference).

Your post is great, but this is honestly the biggest shocker for me -- is this really what a Storeonce is running nowadays? How are they even justifying it?

Just a small note, I love your setup in general. When you speak about checking the XFS stat for dedup, you mean just checking the savings?

The filefrag command should show that you're using the same extents (use the -v flag with a list of files)

https://unix.stackexchange.com/question ... les-on-xfs

Maybe someone has a built in reporting script, but this should be pretty useful to show the usage.

Else, you can just do some comparison between the output of df on the volume and the repository UsedSpace property from Powershell to get the same idea. IIRC, Veeam will report the written space for block cloning volumes for the Used Space property and Freespace is a real read from the volume: https://www.veeam.com/kb2996 (see point 3)