With a huge backup job, what should I choose?

[gberkshier-cefcu]

I have a backup job with one server in it. It's an insanely huge backup - over 10TB. As much as I want to do a weekly full and daily incrementals, I just don't know if I can get a full backup to work in the time I have available to get it done. I kicked the first backup job off over the weekend, and it took just under 24 hours to complete. The next day, the incremental only took 11 minutes. The data in the backup is pretty static. It's mainly images that accumulate, so the old stuff will never change. The daily new data shouldn't be very crazy - maybe 100GB at the most on weekdays.

I've done active full backups and synthetic fulls on other jobs, and these work ok, but for this one job, I really don't want to have to wait 24 hours every week for an active full to finish, and I'm pretty sure a synthetic full would be worse (at least, that's been my experience with other jobs). I currently have the active full and synthetic full options unchecked, and I have it set to 14 restore points. If I don't have full backups getting created, how is this going to work? Does the original full backup just stay there indefinitely, and then the incrementals just move in and out of the storage as time goes by, or until I create another active full manually?

[ChuckS42]

Yah, I don't think you can get away without having a periodic Full backup, be it weekly or monthly. Might have to bite the bullet, or see what kinds of optimizations you can do to shrink the Full backup time somewhat (e.g, exclude volumes that are not necessary for recovery).

What kind of transport are you using? Hot-Add, NFS, etc.?

[foggy]

Periodic fulls are not required provided you test your backups. Running forever incremental is totally ok, the full backup will 'move' forward every day while the oldest increment is being merged into it, maintaining the retention.

[soncscy]

Hey Glenn,

What kind of repository are you using for this? Forever Incremental will work just fine, but if the Synthetic Performance is bad, then XFS/ReFS with fast clone is probably your savior because it does metadata operations instead of IO on the physical drive.

Not to be dismissive, but 10 TB is __not that big__. Fast Clone will handle this quite gracefully, and http://calctool.org/CALC/prof/computing/transfer_time tells me you're likely moving data at 125 MB/s, which also tells me you're using NBD and your VMkkernel port is sitting on a 1 Gbit NIC.

In fact, this is pretty good from my point of view that your backup is:

1. Stable
2. Consistently maxxing out the VMKernel

If you've had bad experiences with Synthetics in the past, definitely you want block cloning. Even if you're using some NAS device, you can likely present the LUN in some way that you can benefit from XFS or ReFS; if that's the case, then you have one more Active Full in your future, and smooth sailing with Synthetic fulls or merges in the future at the cost of the space for an incremental

Give it a shot I say if you have the space for another full on your storage.

[gberkshier-cefcu]

When I had this VM in a backup with others, it had been running for almost 4 1/2 days and this one last VM was the only one left in the job that wasn't done. It was processing at less than 10MB/s. I don't know what was going on, but a lot of other stuff was just messed up in Veeam with settings on concurrent jobs. Not sure what the full extent of it all was, but since moving that VM to its own job and tweaking settings all over the place, it's been backing up much quicker on its own.

The repository and all of our Veeam servers/proxies/gateways are all on 10Gb connections. The repository is a Synology NAS setup as an SMB repository. It's not the way I wanted it to be. Early on, I had it all setup via iSCSI to a server but a bad patch from Synology completely killed the iSCSI throughput, and when this happened, I was early enough in the setup that I just scrapped it all and went the SMB route so I could actually use it. I would absolutely love to get this all started over with iSCSI and an ReFS formatted volume to take advantage of all of that great stuff, but I would have to scrap this all and start over, so at the moment, this isn't an option. But it could be soon. As soon as all of my local backups are stable and completely replicated to our DR repository, I'll have an opportunity to start it over.

[soncscy]

>The repository and all of our Veeam servers/proxies/gateways are all on 10Gb connections.

But what is the transport mode and the VMkernel speed?

If you're using Vmware's NBD backup mode it goes out over the vmkernel, and whatever NIC is tied to that.

You can have the entire infrastructure sitting on 25 Gbit+ hardware but if the source can only serve up a single gigabit, you're always going to choke on that point.

Transport mode is very important; what is the bottleneck your job shows?

[gberkshier-cefcu]

Transport mode is set to auto. Which log do I need to check to determine what mode is being used? If I should manually set, I'm not sure what to set to. The VM storage volumes are on Nimble, and the proxies have two NICs, as does the main Veeam server. One NIC is on the isolated Veeam management network, and the other NIC has direct access to the Nimble storage via iSCSI. Which transport should I be using? Does auto choose the correct one, or should I manually set to Direct or Network?

Where do I determine the VMkernel speed?

More times than not, the bottleneck is the target.

[gberkshier-cefcu]

Also should add that all of my proxies are virtual, so I guess direct storage would not be an option? Hot Add or NBD only?

[soncscy]

Yes, with virtual proxies only hotadd/nbd.

Check on your host for the vmkernel and see what nic it is set for and the link speed on that nic.

With hotadd, you can set up the proxies with a route to the repository over your faster networks. Applications don't get to choose networks, they only get to pick available routes. The OS has some calculations it will do for multiple nics, but you can massage this a bit with Preferred Network settings in Veeam. This only works between the data mover agents though, so the source proxy and target gateway into the NAS.

If these are the same machine, then the Preferred network won't work because the agents will talk using Shared Memory instead. If they are separate machines, put a gateway for your NAS as close to the NAS as you can, then you should get the benefit of Preferred Networks.

[gberkshier-cefcu]

Are you talking about the VMNICs? Those are the only NICs I see in this with speeds on them. Those are all 20 GBit/s.