Coping per-vm/scale-out files into a single localtion

[ShawnKPERS]

I currently have a backup process in Veeam where I have 12 identical jobs that backup up around 90 servers to disk starting with an active full on the first Sunday of the month then incrementals once a day each day till the next first Sunday of the next month. The only difference in each of these 12 jobs is the names and the backup repositories, so each job is responsible for a specific month of the year and I call the jobs "Veeam Backup January", "Veeam Backup February", Etc... These jobs are only enabled during the 30ish day window between the first Sunday of that relative month and the first Sunday of the next month. Currently Veeam does not support this type of scheduling system so I use a script that disables all jobs and then enables the relative job for that month before the job runs.

The logic behind this is to provide the ability to restore any file or VM from within this cluster of VMs over the past 365 days and breaking it up into 12 separate jobs. Meaning each job is only responsible for a single restore point in case there is issues with the job it won’t corrupt the previous months. Plus, even though we still have other jobs that dump these servers to tape having them on disk saves so much time/trouble of calling tapes back from Underground Vault and loading them up (worth it for this one feature alone if you ask me). Last but not least using cheap storage is slow and when I first started worked fine but over time with the additions of more data and changes in the jobs it now takes longer than I would like to run each day.

This system has worked well enough over the last two years but it has its issues. First if I need to change a setting in one job I then need to repeat that change in the other 11 Jobs manually (I know this can be done in a script but I have been too lazy in the past to write one that suits my needs). Second changing of the jobs each month can be problematic when doing major Veeam upgrades since each time I seem to do a major upgrade it usually breaks the few Veeam scripts I have including the one that changes the monthly jobs and I have to scramble to get it working again or do things manually each month. Third is worrying about storage repository failures and upgrades/migrations, since this takes a massive amount of storage I tend to use cheap flimsy solutions like Qnaps or not so flimsy Windows server to go "cheep" and "deep" and if there is any issue with the storage I have to move a massive amount of data somewhere else temporarily or I just loose it. Not the end of the world as these are just backups to my main backups but they are convenient and I would not want to lose them or need to keep an equivalent amount of empty disk around ready in case I need to move them.

So now I am in the process of moving all my storage around and decided to rethink this backup process since Veeam has added a few new tricks up its sleeves over the last couple versions and I have learned a few lessons about the pitfalls of this implementation. Here is what I was thinking of trying but need help in hammering out the details.

First I like the idea of going down to a single monthly job similar in settings to the original 12 but this job would back up to two different older SAN arrays using scale-out cluster repositories and per-vm backup. I would have access to enough disk to only store a months’ worth of backups with a little room to grow but the disk is fast so it will shrink my active backup window. Then I would like to find some way to take a copy of all these per-vm backup files scattered over four or so backup repositories in the cluster onto a single folder called whatever that current month is on the cheaper disk. This can run throughout the day without impacting production. Then when it is the next month on the first Sunday the job does an active full then overwrites the previous month’s files then copies these new files to a new folder called whatever the current month is at that moment throughout the rest of that month.

I assume I will need to take advantage of the backup copy job feature to get this to work but even then I am not sure this type of schedule will work since I am just now starting to play with backup-copy jobs to see how they work. Also once a full month is done and copied to the cheaper disk I would like to copy the full and incrementals files of that entire month to tape so if there were ever storage loss or I had to move to new storage I can just restore each individual month to the new storage and not need to worry about failures or needing huge chunks of spare disk laying around.

Worse case I guess could keep the system I have now or some hybrid between the two but I figured I would ask and see if anyone has any ideas. The real issue I am having is using the per-vm/scale-out just for performance and then trying to collect the files back under a single location for archiving. Any ideas would be appreciated!

[foggy]

Switching to a single backup job looks reasonable, I do not see the reasons for your current setup with different jobs and folders for each month. Regarding the backup copy to a secondary location, are you going to use 12 rotated drives for that? I also do not see the reasons to keep each month's data in a separate folder otherwise, you can just configure it to keep 12 monthlies made on the first Sunday of the month, enable active fulls and set retention to 365.