Advice on retention and off-site backups

[sarnold]

Hello,

I'm preparing to enhance our current Veeam backup environment and am looking for advice in a few areas around backup jobs, storage repositories, and file systems.

We back up 2 primary sets of data:
•Our file servers at each remote site (roughly 14 TB of VMs)
•Our datacenter servers that serve the entire organization. (Roughly 8.5 TB of VMs)

These are configured as 2 separate jobs; a datacenter backup job (runs at 10PM daily), and a site backup job (backs up all the remote site servers at 11PM daily). We have a minimal amount of jobs (2) to take advantage of deduplication in the backup jobs.

Right now, these two jobs go to a single 56TB SAN formatted as NTFS, via iSCSI. I opened a case back in April because we were running into space issues due to keeping 30 points of backups for both jobs on the single 56TB SAN. I ended up changing this to 21 restore points on the recommendation of the Veeam agent who helped with that case, and that's kept us at around 10-15GB free on the SAN since then, with full synthetics being created weekly.

My goal is to create more of a GFS based retention so we have more history of backups instead of our current 21ish restore points (plus a few more for synthetic fulls). Something like 30 days of daily backups (with maybe even 2 backups per day during business hours to catch deleted files during the day), 3 months of weekly backups, and 2 years of monthly backups, for example. I'd also like to implement off-site backups.

Here's what I have available to me:
•56TB SAN (currently being used for our 21 days of backups on these 2 jobs)
•56TB SAN (available for use)
•4x servers with 48TB of disks each (available for use)

Note: The 56TB sans are 56TB of usable space after RAID/volume creation. The 48TB servers are 48TB raw space (12x4TB disks), unformatted, no RAID currently.

Knowing what we have, that we're only currently backing up on-site to a single SAN for both jobs, and knowing where we want to go in terms of adding a GFS policy and off-site backups, these are the questions going through my mind:
•Is ReFS a good idea for all backup respositories? I've spent several hours reading about this today and I'm torn between using it and sticking with NTFS like we are now. Yes, our synthetic fulls take between 16 and 30 hours to complete, but is it safer to stay on NTFS? Or should we go ReFS for all backup repos with 64K cluster sizes as I've seen recommended? Our backup server is Windows Server 2019 Datacenter running Veeam B&R 10A (10.0.1.4854). I've read posts of people who love it with Veeam, but have also read nearly an equal amount of posts of people who went back to NTFS with Veeam and are much happier.

•Thoughts on how I should best use the hardware we have available? Should both SANs stay in the datacenter, one for the datacenter 30 day retention backup job, the other for the remote site 30 day retention backup job? Then 2 of the 48TB servers stay in the datacenter for the 3 months of weekly, 2 years of monthly backups, and the other 2 48TB servers go off-site and have copies of those 3 months of weekly, 2 years of monthly backup jobs?

•Assuming I do keep 2 of the servers with 48TB of disks in each location (datacenter and off-site), how would you recommend I provision the storage? Standard RAID across all disks on each server, and treat them as either 2 separate repos (or with scale out repositories), or configure the disks as HBA pass through and use something like Storage Spaces Direct to combine the two servers into one large pool?

Just thinking out loud here. I'm open to any/all ideas for how to best approach this.

Thanks!

[HannesK]

Hello,

Is ReFS a good idea for all backup respositories?

Yes. At least if you like to spend your budget efficient. 64KB block size is the recommendation, yes. The wizard shows a message if 4KB is configured.

Server & Storage distribution: I did not read anything about bandwidth, but the recommendation is to have different locations for primary backup and backup copy job destination. So if you have the bandwidth, then different locations are the recommendation (https://www.veeam.com/blog/how-to-follo ... ation.html)

Servers: Scale out repository makes it easier if you have two servers per site, yes (one SOBR per site). I would go for it. And I would go for RAID. I don't like Storage Spaces Direct for it's complexity and hard to maintain performance. We had long internal discussions and probably you also find some public discussions on it. I would always go for RAID.

Best regards,
Hannes

PS: I would also go for per-VM backup chains and forget about the little amount of deduplication. The advantages of per-VM backups are just too big from my point of view. I know that many of the points are not relevant in your environment. Just the full list for reference.

- Easier tape restore
- No 16TB files on wrong formatted NTFS volumes
- More performance through parallel processing
- Easier job management (put more VMs in one job)
- Resource usage with SOBR
- Optional Windows Server 2016 Dedupe if files < 1TB
- Easy deletion of VMs from backups
- Per VM accounting

[sarnold]

Thanks very much HannesK!

I think staying with Windows Server 2019 and going ReFS would be a good bet for us in our case then, and I like your idea of going RAID and Scale Out Backup Repositories for our 4 servers (a SOBR for 2 of the servers on-site in the datacenter, and a SOBR for 2 of the servers off-site). Per VM is also interesting and something I'll look at. It looks like it's literally a check box to enable it. Mind you, how does this look when I already have existing backups in a large backup file for the job? I'm assuming this creates a new backup file?

Regarding bandwidth, we have a 2Gbps (up and down, fibre) connection at the main datacenter, and roughly a 500Mbps (up and down, fibre) connection at the off-site location.

So now I just need to decide on how we want our backups distributed. These two ideas come to mind, but I'm open to suggestions if anyone has any! Curious what people think:
•Store both daily backup jobs on one SAN like we are now (21 restore points) / Use the second SAN for storing a Backup Copy Job of weekly backups for both jobs / Use 2x servers in a SOBR for storing a Backup Copy Job of weekly (longer retention)/monthly/yearly backups / Use 2x servers in a SOBR for storing an off-site Backup Copy Job of longer stored weekly (longer retention)/monthly/yearly backups
•Store one of the backup jobs on one SAN, one of the backup jobs on the other SAN, and store 21 restore points of each (or more, if we do 2 or 3 backups per day to catch deletions during the day) / Use 2x servers in a SOBR for storing a Backup Copy Job of weekly/monthly/yearly backups / Use 2x servers in a SOBR for storing an off-site Backup Copy Job of weekly/monthly/yearly backups

Thoughts? Suggestions of something entirely different?

[HannesK]

yes, per-VM backup files create a new backup chain that has one VM per backup chain (instead of all VMs of one job in one backup chain)

I would always like to have an offsite copy of the daily backups (the backup copy job can do that).

And if you have the space, I would even keep GFS in both locations.