Best Practice for Backup to Tape after Discontinuation of the Synthetic Full Backup + Rollback with Veeam B&R V12

[Mofamike]

Hello all,

first of all, sorry for my bad English, I hope you understand anyway

For the moment our backup strategy is the following:

Daily Forward Incremental + synthetic full backup + transform previous backup chains into rollbacks - Retention 14 days, the fullbackup will be done weekly = saturday
The result: We only have one full backup on disk.

Furthermore we start a tape job (file to tape) on Mondays, which backups the newly created VBK files from the weekend to tape. This process lasts until Thursday evening, after which the new synthetic full backup is created again on the weekend.
We've done well with this strategy so far

--------------------------------------------------------------------------------------------

But now we heard that with the upcoming version VBR V12 the function synthetic full backup + transform previous backup chains into rollbacks will no longer be supported.

So in our opinion we have the following options:

1. Keep going with synthetic Full every week but without the rollback feature >> The result is that we have to store more data (3 full backups and about 20 days of retention) >> more storage capacity needed not as before only 14 days and one full backup on disk,

2. Use the forever incremental backup (without creating regularly fulls) and use backup to tape job and use the feature "virtual full" but this process take so much longer (reconcatenation (temp file)) on repository and later backup to tape then (compare to the file to tape job)

Is there anyone here who uses the same backup strategy and will have problems when the rollbackup function is removed with V12? We are unsure how to proceed. We had already opened a case (05430328) for this, but didn't know that the backup to tape + virtuall full takes so much time in comparison.

Do you have any best practise ideas to continue with Backup to disk to tape with just one full backup on disk?

Thank you very much.

[HannesK]

Hello,

1. Synthetic full backups are "spaceless" on recommended architectures (XFS / REFS). But yes, you are right, it would take more incrementals.

2.

but this process take so much longer

how much longer does it take for you and what kind of backup storage are you using? (as many details as possible. vendor, raid controller, number of disks, deduplication used, file system etc.). In V11, the difference should be irrelevant (if you are not using V11, please upgrade to V11 latest version)

3. Another option is reverse incremental. But that method is not recommended in general for performance reasons. Option 2 is "the way to go".

4. a bit off-topic: Object storage could also be used. I guess the main challenge is the "minimum scale" in this situation.

Best regards,
Hannes

[Mofamike]

Hello Hannes,

thank you very much for your feedback.

1. Yes we use REFS on all our backup repositories for now, sorry forgot to mentioned that. Yeah I heared and read about the space saving by refs but did not think that further full syn-backups (besides the initial main full-backup) with forever forward incremental backups require little or no additional storage. That's definitely good to know and thanks for sharing the link. Sure as we both know, the further incremental backups will consume more space then.

2. We did a testscenario:

2.1 Forward incremental job with synfull + rollback + FileToTape job (only *.vbk) so per VM just 1 vbk file (around 1 TB Data). Duration: 1,5 hours with average speed 215 MB/s

2.2 Forever forward incremental job with no perodic full schedule + BackupToTape job (with virtual Full creation) so per VM just 1 vbk file (around 1 TB Data). Duration 2,5 hours with average speed 103 MB/s

We use Veeam B&R V11 (latest version) with REFS.

The repository server is a supermicro SSG-5048R-E1CR36L with windows server 2016 with CPU Intel Xenon CPU E5-1650 v3 3,5 GHz 6 Cores 12 Logical CPUs; 265 GB Memory, Raidcontroller Avago MegaRaid SAS 9480-8i8e - 12 Drive Groups, 12 Virtual drives - 8 Disks per Drive Group / Virtual Drive so 96 Disks- Raid6 - Hardisk typ: SEAGATE ST2000NM0045 2 TB. These 12 Virtual drives are summarized in Windows Server Manager to one Virtual Disk read / Write Health Simple
Datatransfer is via network to our tape backupserver. (10GBit 2 Port Teaming) Tapelibary is connected via FC on Tape Backupserver.

We did the comparison between File to Tape and Backup to Tape (virtual Full) same source data also on another Storage Server (MSA 2040) with 84 Disks Raid 6
HP EG0600FBDSR 600GB splitted in 2 Pools. MSA is connected to a VBR Backup server via SAN. Datatransfer is via network to our tape backupserver. (10GBit 2 Port Teaming) Tapelibary is connected via FC on Tape Backupserver. The duration of both job are nearly the same here. Datatransfer is here about 300 MB/s

We use a HP MSL3040 LTO8 Tape Library with 2 Drives.

So it seems like our Supermicro storage is having a much harder time with the virtual full than our MSA storage.
Which key aspects are important for the creation of the tape virtuall full in the course of storage performance? Filesystem REFS for sure but what is also important here?

And one more question about the virtual full: I read about that the virtual full option in BackupTotTape jobs are only available with Source "Backupjob" but not with Source "Backup Repository". Is that right?

3. Got it and yes you are right.

4. Yes we thought about this already. But we already have a tape infrastructure (not that old and a lot of customers like this way of tape security) so we wanna keep going with this scenario

Thank you very much and have a nice day.

Best wishes

Michael

[HannesK]

Hello,
thanks for the details

2) 215MByte/s from 96 disks sounds slow to me in general. But I'm surprised about the big difference for the server, yes. Hard to say what the reason is. Maybe changing cache settings on the RAID controller might help. But that should not make 2x difference in any case.

Which key aspects are important for the creation of the tape virtual full in the course of storage performance? Filesystem REFS for sure but what is also important here?

file system fragmentation is probably the main point. But my understanding is, that the two tape jobs have the same REFS file system as source. So they should have the same conditions.

I read about that the virtual full option in BackupToTape

where did you read that? I cannot remember that limitation.

Best regards,
Hannes

[Mofamike]

Hello Hannes,

thank you for your answer.

2) 215MByte/s from 96 disks sounds slow to me in general. But I'm surprised about the big difference for the server, yes. Hard to say what the reason is. Maybe changing cache settings on the RAID controller might help. But that should not make 2x difference in any case.

215MB/s is the speed we got with deactivated drive cache setting. When we activate the cache, the performance become even worse. And yes with these number of disks we except a way better result but for right now we are out of ideas. As I said this backup repo is connected via network 2x10GBIT Teaming Duplex to the vbr tape server. We can observe that this Teaming Adapter (2x intel adapter x540-T2) process only round about 2,5 GBit data traffic to the vbr tape server (total with both adapters). Theoretically, it could also be that the storage could do more, the network on the source side (repository) or target side (vbr tape server) has a bottleneck.

Strangly the MSA Storage can do 300 MB/s and the way the server is conntect to the network (Firewall / Switch) is the same. I can now also rule out a port limitation.
Our VBR Server with the MSA have a slightly different network cards (2x intel adapter x540-AT2 also in team 10GBIT duplex same config)

So if you have any idea where to check, feel free

file system fragmentation is probably the main point. But my understanding is, that the two tape jobs have the same REFS file system as source. So they should have the same conditions.

Right

where did you read that? I cannot remember that limitation.

https://helpcenter.veeam.com/docs/backu ... ml?ver=110

The virtual full feature seems like to note only at point "Source Jobs Backup Methods" and not in "Backup Repositories as Source"
We did two test jobs (1x source backup job / 1x source repository). With source "backupjob" the virtual full is created without any problems, with source "Repository" the VBK is only copied from the repository. In combination with Forever Incremental (VBK is changed daily) that would no longer make sense. But I don't want to rule out the possibility that we made also a configuration error (settings are same, just different source).
if it really should be the case that the backup to tape virtual full feature can only be used with Source "Jobs Backup Methods", then the feature is unfortunately of no interest to us, since we only use repositories as a source. The advantage for us here is that created backup jobs are automatically included in the tape backup. This variant is significantly less error-prone for us because otherwise it can easily happen that a short-term backup job is created but not added to the tape backup.
And if so, we need to go also back to incremental + weekly synfull and the result is that we need to provide more storage (sure refs helps us here but just for full not for incr.)
I would be happy if you could find out more about that, that we can plan our required hardware resources accordingly

Thank you so much.

Best wishes

Michael

[HannesK]

Hello,
yes, drive cache should be deactivated always. Only the RAID controller (with battery) should be used as cache. I would start with diskspd

Yes, "Backup Repositories as Source" does not mention anything (really nothing, and I will ask documentation team to make it clearer), but it's supposed to work. If not, please post the support case number that we can investigate.

If you can reproduce the 205MB vs. 103MB from the same source, then we can also investigate from support side, whether we can optimize things in the software. Because that's unexpected. The MSA result is "how it should be". Please post the support case number, that R&D has a chance to follow it.

Best regards,
Hannes

[Mofamike]

Hello Hannes,

thank you for your answer and your provided information.

Yes, "Backup Repositories as Source" does not mention anything (really nothing, and I will ask documentation team to make it clearer), but it's supposed to work. If not, please post the support case number that we can investigate.

Thank you that you will clearyfing that, yes we were a bit confused because our tests mentioned that backup to tape jobs are not working with virtual full when the source is a repository. Looking forward to hear from you here. The information is important for us for our hardware planning (more or less storage capicity).Casenumber is #05666124 — Initial Case was #05430328. But seems like we got our answer already when I take a look in the case details on portal (no virtual full with source Backup Repository).But It would be great if you could double check that.

If you can reproduce the 205MB vs. 103MB from the same source, then we can also investigate from support side, whether we can optimize things in the software. Because that's unexpected. The MSA result is "how it should be". Please post the support case number, that R&D has a chance to follow it.

Yes right MSA is good (File to tape and Backup to tape with virtual full nearly the same speed)

Our supermicro Storage

The repository server is a supermicro SSG-5048R-E1CR36L with windows server 2016 with CPU Intel Xenon CPU E5-1650 v3 3,5 GHz 6 Cores 12 Logical CPUs; 265 GB Memory, Raidcontroller Avago MegaRaid SAS 9480-8i8e - 12 Drive Groups, 12 Virtual drives - 8 Disks per Drive Group / Virtual Drive so 96 Disks- Raid6 - Hardisk typ: SEAGATE ST2000NM0045 2 TB. These 12 Virtual drives are summarized in Windows Server Manager to one Virtual Disk read / Write Health Simple
Datatransfer is via network to our tape backupserver. (10GBit 2 Port Teaming) Tapelibary is connected via FC on Tape Backupserver.

yes, drive cache should be deactivated always. Only the RAID controller (with battery) should be used as cache. I would start with diskspd

I think the issue has then settled for us. The slow performance only occurs when creating the virtual full. However, since this is not possible at the repository level, we do not have to do any further testing here

Thank you very much and best wishes.

Michael

[HannesK]

Hello,

yes we were a bit confused because our tests mentioned that backup to tape jobs are not working with virtual full when the source is a repository.

that makes no sense to me. I just tested it and it works fine.

Code: Select all

09:00:31	Synthetic full backup from HK-Nano.vm-56528D2022-10-14T085050_8625.vib (14.10.2022 00:00:00) will be placed into the media set 	
09:00:32	Source backup files detected. VBK map: 1 	

forward forever incremental chain, which I started yesterday:


source is repository in the tape job


Then I ran a job yesterday and today I configured synthetic fulls in the tape job



Thanks for the support case number. I will ask support about the details.

Best regards,
Hannes

[Mofamike]

Good morning Hannes,

thank you for testing.

that makes no sense to me. I just tested it and it works fine.

Code: Select all

09:00:31	Synthetic full backup from HK-Nano.vm-56528D2022-10-14T085050_8625.vib (14.10.2022 00:00:00) will be placed into the media set 	
09:00:32	Source backup files detected. VBK map: 1

Yes, we got these log entries too (in both testjobs 1. with source Job and 2. with source repository. But only in Job with source Job we see on repository side the temporary "*.vsb file"
https://helpcenter.veeam.com/docs/backu ... ml?ver=110 for virtual full creation so we asumed that the Job with source repository and
only backuped the existing *.vbk file that is already on disk, which makes no sense with Forever incremental, as this is overwritten daily.

Thanks for the support case number. I will ask support about the details.

Do you already have some new information about it?

Thank you very much.

Best wishes

Michael

[HannesK]

Hello,
yes, the engineers knows that he provided wrong information. That should not happen, but nobody is perfect.

If you see problems in getting VSB to tape with "from repository", please escalate the case / create a new case.

Best regards,
Hannes

[Mofamike]

Hello Hannes,

thank you very much for the information and for asking the responsible Case technician.
Yeah all good, just good for us to know for further hardware ressource planing.
So did I understand it right now that virtual full (*.vsb) normaly should work with backup to tape (source repository)?

If you see problems in getting VSB to tape with "from repository", please escalate the case / create a new case.

And yes we will open a new case then and will refer to this article / to the old case.

Thank you very much

[HannesK]

Hello,

So did I understand it right now that virtual full (*.vsb) normaly should work with backup to tape (source repository)?

yes, correct.

Best regards,
Hannes

[Mofamike]

Hello Hannes,

I had a team meeting today with Mr Hermann from Veeam - thanks again for your time.
We talked about our current strategy, about a necessary strategy change in V12 and about Veeam's further plans for tape support.

Our current strategy:

Daily Forward Incremental + synthetic full backup + transform previous backup chains into rollbacks - Retention 14 days, the fullbackup will be done weekly = saturday
The result: We only have one full backup on disk.

Furthermore we start a tape job (file to tape) on Mondays, which backups the newly created VBK files from the weekend to tape. This process lasts until Thursday evening, after which the new synthetic full backup is created again on the weekend.
We've done well with this strategy so far.

Also we have the SLA Customer Agreement that VBK Full Backups going for Long Termin Retention to tape (real offline backup) so object storage is not a option for us right now.

--------------------------------------------------------------------------------------------

But now we heard that with the upcoming version VBR V12 the function synthetic full backup + transform previous backup chains into rollbacks will no longer be supported.

Possible strategies in V12:

1. Keep going with synthetic Full every week but without the rollback feature >> The result is that we have to store more data (3 full backups and about 20 days of retention) >> and more storage capacity needed as before (about 20% more). We have agreed a 14-Restorepoint SLA with our customers on disk and we charge for the space usage of those 14 points. Yes there is the spaceless fulls with refs but still more incremental files on disk.
The Problem is: For some customers we have file servers with VBK files larger than 25 TB. More backup statuses > 14 are a significant additional storagecost / burden for the customer and for us. We cannot pass these additional costs directly on to our customers.



2.To work with forever incremental / reverse Incremental with just 1 VBK and 13 days incremental (14 points) on disk is not a real solution for us at least not in combination with tape.

2.1. Yes there is the virtual full for tape support but still the vbk file in disk job get overwritten every 24 hours / get in access from diskjob. So tapejob should be able to process the virtual full job for this vbk file also in 24 hours right? We have vbks file larger than 25 TB, this will be a problem. / When the virtual full creation for all VMs of a repository get started? Just after the tapejob started on Monday (for all VMs) or one after another when this VBK File is in row to process? And what happend with parallel Disk jobs then? I assume then a prioritization for disk or tape job can be configured, but of course that leads to delays In terms of time, that would be an incalculable factor for us then.

2.2 Also this forward incremental with backup to tape jobs and virtual full just work with backupjobs that are configured on the vbr server that is directly connected to tape library (Tape Administration Server) but not for VBKs on other vbr server (with no Tape library connected) For this case we need to use file to tape jobs then and transfer them via network to the library, but this is not comaptible with forward incremental disk jobs. And we need to use Option 1 (Forward + weekly fulls) strategy here. As a result, we use different backup strategies on our VBR servers, which makes management more confusing and difficult.

--------------------------------------------
Basically, variant 1 is likely the way to go for us (create fewer dependencies / problems for us in case of tape support) only the fact that we need 20% more storage space and are now faced with the problem of arguing the additional costs for our customers / us.

Maybe you see another / 3rd soluton here?

Thank you very much

[HannesK]

Hello,
As you only have 14 days, you could also just go with daily synthetic fulls. With 4 weekly & 12 monthly, that would also be 14 fulls. So I don't expect REFS to have any issues with that.

25 TB in 24h would mean around 300MByte/s. That would fit with one LTO 8 /9 drive (because there is one stream to tape per file). If that's not enough, such big VMs could be split into multiple jobs (assuming that one VMDK can compress to less than 25TB). I know, it's ugly, but it's technically possible.

For smaller machines, there is parallel streaming per machine. Example: 100 machines with 1TB each (100 TB) could stream to 6 drives in parallel with only ~215MByte/s per drive in 24h.

In general: if a tape job takes longer than 24h, then the backup storage is usually also to slow for real restores. So improving on the hardware side would be an alternative option.

File-to-tape looks like a workaround in your scenario. Is it correct, that we could solve that problem by allowing to share a tape server between multiple VBR servers?

Best regards,
Hannes

[Mofamike]

Good morning Hannes,

thank you for your answer.

As you only have 14 days, you could also just go with daily synthetic fulls. With 4 weekly & 12 monthly, that would also be 14 fulls. So I don't expect REFS to have any issues with that.

Yes, we thought about this already but I think maybe we will get trouble with File to Tape jobs then (file mask *.vbk) It might work but needs some testing.

25 TB in 24h would mean around 300MByte/s. That would fit with one LTO 8 /9 drive (because there is one stream to tape per file). If that's not enough, such big VMs could be split into multiple jobs (assuming that one VMDK can compress to less than 25TB). I know, it's ugly, but it's technically possible.

Yes, we expect that the file server will continue to grow, but then we will have to split it. However, that would not be a nice scenario.

In general: if a tape job takes longer than 24h, then the backup storage is usually also to slow for real restores. So improving on the hardware side would be an alternative option.

For our backup hardware refresh in 2023 we are planing to use faster disk short time storage to get better performance on tape.

File-to-tape looks like a workaround in your scenario. Is it correct, that we could solve that problem by allowing to share a tape server between multiple VBR servers?

Exactly, we even have to implement it this way, because we only have one tape library connected to one VBR server. This means that a "backup to tape" job only works on this server, since the corresponding short-term jobs are also configured here.

We have other VBR servers on the network that don't have a directly connecion tp the tape library. These jobs / backups must then be backuped to the tape library via a "File to Tape" job.