Hourly Backups & Performance

[jshiflet]

Hello!

I'm wondering if anyone has their Veeam environment setup to perform hourly backups of their VMs. We're a MSP and are looking at some possible alternatives to overcome issues we've been having with some of our other backup solutions in virtual environments. The biggest advantage that some of them have are backup snapshots that can happen as low as every 15 minutes. Generally, we only schedule those backup snapshots to happen once an hour to limit backup size bloat, but we are still heavily invested in marketing our Manged Backup product to our clients as taking backups of their data at least once an hour.

So far my experience with Veeam speeds hasn't been fantastic, but I'm not sure if that's just because of the infrastructure we're using it in. We're also a private cloud provider, and use Veeam exclusively in our cloud environment as it's fully virtualized on Hyper-V. The products we use for backing up our MSP clients are more suited to physical systems than virtual, and we wanted a good backup experience for our cloud environment, so we went with Veeam because at the time they were a virtual-only company and had the best solution for it.

Back to the speeds, though...I'd be lucky to take more than maybe 3 backup snapshots a day in our cloud environment because of the sheer slowness even with a relatively small number of VMs (less than 30). Like I said, I have a feeling it's because of our architecture, mostly due to our primary storage volume lacking the ability to utilize transportable snapshots with iSCSI and VSS (the hardware manufacturer never updated their Hardware VSS Provider to support Windows Server 2012/2012 R2), so we have no choice but using on-host backup proxies.

The environment I'm about to try testing in is different though: the Hosts only use local storage. No CSV/iSCSI, no SMB 3.0...just regular spindle storage directly attached via SAS inside the machines. And at that, it's two physical machines each with two VMs. My thought is that there's potential for that to operate much more efficiently in processing the backups allowing smaller backup windows, but I also wanted to get input from users to see if anyone has been able to get Veeam to perform hourly snapshots without any real trouble.

Any thoughts would be appreciated!!
~Jason

[foggy]

Jason, whether or not you're able to have hourly backups depends on the VM activity (amount of changes occurred in it within an hour) and whether your infrastructure allows to process those changes within that backup window. What are the current backup speeds you observe and what bottleneck stats are reported by the backup jobs?

[jshiflet]

The reported bottleneck is always Source.

Here's the stats from one of our latest backups:

Processing rate: 82 MB/s
Source: 96%
Proxy: 29%
Network: 21%
Target: 13%

We have a total of 11 Backup Jobs, backing up a total of 28 VMs. Here's the job processing rates from our backup window last night for each job:

• 75 MB/s
• 70 MB/s
• 50 MB/s
• 81 MB/s
• 72 MB/s
• 114 MB/s
• 82 MB/s
• 81 MB/s
• 101 MB/s
• 73 MB/s
• 67 MB/s

Every single one of them shows the Bottleneck as Source with the Percent Busy ranging from 96%-99% on Source for each of them.

And here's a curious item: that job that showed at 50 MB/s, it's a single, very un-frequently changing VM. The Read amount was 1.9GB and Transferred 85.4 MB.

I've just done a test as well and using Windows Explorer File Copy, I average about 300 MB/s of write speed when copying a 127GB VHDX file.

[foggy]

Am I correct that those speeds do not allow you to perform hourly schedule for all of your VMs?

[jshiflet]

Am I correct that those speeds do not allow you to perform hourly schedule for all of your VMs?

Correct.

I've configured them to run in sequence, so there's only one job running at a time and parallel processing is enabled so jobs with multiple VMs will snapshot those simultaneously, but each VM is still processed individually inside the backup job...not sure why that is.

My backup window starts at 11:30pm and I'm lucky to see it finish before 4am.

[PTide]

Hi,

there's only one job running at a time and parallel processing is enabled so jobs with multiple VMs will snapshot those simultaneously, but each VM is still processed individually inside the backup job

Out of curiosity - what are your proxy settings? It seems that your proxy is limited in resources.

Thank you.

[jshiflet]

Well, because the hardware doesn't support transportable snapshots, the backup proxy is on-host rather than off-host. I'm not sure where I would specify anything for on-host resource limits. I see it in the off-host Hyper-V proxy, but that never gets used. And this is a backup of a Hyper-V Cluster with everything living on CSV storage.

And I also went and checked and my Backup Repository is set to limit concurrent tasks to 4.

[nmdange]

While this would involve getting more hardware, if you put Scale-Out File Servers in front of your storage and use SMB as the storage protocol with Hyper-V, you can do Off-Host Backup Proxy with any storage you want without having to rely on hardware VSS support. See http://helpcenter.veeam.com/backup/80/h ... ackup.html

Another item to consider with backups and snapshots is the size of your disks and the number of VMs. If you have a small number of very large disks then that limits the number of VMs that can be backed up at the same time since each time you snapshot the disk, only a max of 4 VMs are backed up. Splitting the CSVs into smaller disks with fewer VMs per CSV might help as well.

[jshiflet]

While this would involve getting more hardware, if you put Scale-Out File Servers in front of your storage and use SMB as the storage protocol with Hyper-V, you can do Off-Host Backup Proxy with any storage you want without having to rely on hardware VSS support. See http://helpcenter.veeam.com/backup/80/h ... ackup.html

We're actually evaluating options for hardware upgrades over the next year and SMB 3.0 file servers are a possible option for us. The only thing I don't like is that CBT doesn't work with it yet. Maybe once Server 2016 has been out for a little while.

Another item to consider with backups and snapshots is the size of your disks and the number of VMs. If you have a small number of very large disks then that limits the number of VMs that can be backed up at the same time since each time you snapshot the disk, only a max of 4 VMs are backed up. Splitting the CSVs into smaller disks with fewer VMs per CSV might help as well.

And we don't really have huge volumes here either. We've got less than 3TB of VMs living on the CSV right now and the volume itself is 5TB usable.

[nmdange]

CBT works with SMB if the file servers are regular servers running Windows Server 2012 R2. It doesn't work when you are using a SAN that natively supports SMB (like NetApp) because you can't install the Veeam agent on the SAN directly.

How many VMs are on your 3TB CSV? Also is that the only volume you have? I would at least consider having 2 CSVs.