Is my Veeam fast enough?

[ejenner]

How do you decide whether or not your system is performing well.

Does Veeam have a reasonable expectations of data transfer rates?

A new 1.6TB backup from a cluster took 1hr 34min.

I got the following load report:

Load: Source 81% > Proxy 91% > Network 84% > Target 2%

So the primary bottleneck was my proxy. But should we be happy with the system overall, it seems to transfer about 1TB an hour for this job.

Nothing else was running at the same time as I scheduled it outside the backup window as it was an initial backup.

Is it close to what we should expect or are there substantial gains to be made by making changes?

[DGrinev]

Hi,

All expectations are based on the hardware throughput, if your backup infrastructure allows you to transfer 1TB/1hr and you see this kind of performance, then there is nothing to worry about.
You can share the backup infrastructure specification, so we could give you some hints. Thanks!

[ejenner]

This cluster has two nodes inside a c3000 blade center. They are BL460c Gen8 with 24gb. System drive is mirrored 10k SAS drives. Storage is HPE 3PAR StoreServ 7200 2-N. Network is teamed 10gb HP flexfabric. Windows 2012.

The Veeam B&R server and repository server are ProLiant DL380 Gen9 with 64gb. System drive is mirrored SSDs. The repository storage is Smart Array P441 direct attached with 6tb SATA drives. Formatted as REFS with Windows 2016.

The B&R server is acting as the proxy and was reported as the bottleneck.

Are we getting about the right performance based on that config or should it be faster?

[foggy]

Pretty decent speed and it seems you're maxing out your setup. You could probably get a bit more of it, since the current bottleneck is proxy CPU load, but then either source or network will become bottleneck, as numbers are pretty close.

[tsightler]

You didn't say how many cores your VBR server actually has. I personally would consider this a little on the slow side based on the hardware you have, but if it was a single large VM, with one very large disk, then it's really not bad. You're basically getting 325MB/sec but I'm quite surprised at the proxy being the bottleneck unless the proxy only has like 4 cores or something, which is very low these days.

I'm also shocked at the network bottleneck being so high if you really have 10GbE, I would not expect it to be anywhere near this. I'm not completely clear from your description, do you have a separate servers for the VBR/Proxy and the repository? If so, what is the network connection between those?

[ejenner]

It's a VAW backup so I'm not sure if that makes a difference to where you should site your proxy as we're only just getting going with this new setup. At the moment the only proxy we have is the VBR server itself. It was our intention to deliberately set up proxy servers when we start to backup our Hyper-V environment.

It's a single 8-core processor in the VBR server.

[tsightler]

VAW doesn't really use proxies, effectively, with VAW/VAL, the system on which the agent is running is the "proxy". In this case it's the specs of the source server running the agent that would set the maximum throughput. How many CPUs/cores does the server that is running VAW have? It seems like the backup was largely using most of the CPU available on that server. What is the network connectivity from the server running VAW to the repository?

[bdufour]

i agree with tom, with 10ge your network bottleneck should not be that high!

[ejenner]

This was the source system spec from earlier: This cluster has two nodes inside a c3000 blade center. They are BL460c Gen8 with 24gb. System drive is mirrored 10k SAS drives. Storage is HPE 3PAR StoreServ 7200 2-N. Network is teamed 10gb HP flexfabric. Windows 2012.

The backup target and source servers are in different server rooms on different floors so there could be a slow link somewhere between the two.

Just to be clear, I'm not saying I think it is running slow, just that if there were any obvious changes we could make which would dramatically improve throughput for this particular backup then it would be interesting to look into that.

[tsightler]

This was the source system spec from earlier: This cluster has two nodes inside a c3000 blade center. They are BL460c Gen8 with 24gb. System drive is mirrored 10k SAS drives. Storage is HPE 3PAR StoreServ 7200 2-N. Network is teamed 10gb HP flexfabric. Windows 2012.

Right, but this doesn't say how many cores are in this system, and CPU cores on the source system are the #1 factor in maximum backup throughput with VAW, assuming there are no bottlenecks anywhere else. I believe a BL460c Gen8 could have anywhere from a single 4-core processor to dual 12-core processors.

That being said, I'm going to guess there is a slow, 1GbE link somewhere in there due to the high network bottleneck. Saturating a 1GbE link would give about 120MB/s, but assuming 2-3x compression, which is typical, you should get somewhere between 250-350MB/s. You're on the high end of that.

I think the biggest point of confusion was the statement "The B&R server is acting as the proxy...", which would imply you are doing hypervisor style backups, but then you said you were using VAW, which doesn't use proxies. The "proxy" bottleneck for agent based backups is the CPU on the host running VAW in that case, because effectively every agent is it's own proxy.

[bdufour]

The Veeam B&R server and repository server are ProLiant DL380 Gen9 with 64gb.

ProLiant DL380 Gen9 comes standard with 1ge. which would makes sense why your veeam server proxy is the primary bottleneck and your network bottleneck is so high.

[ejenner]

Right, but this doesn't say how many cores are in this system, and CPU cores on the source system are the #1 factor in maximum backup throughput with VAW, assuming there are no bottlenecks anywhere else. I believe a BL460c Gen8 could have anywhere from a single 4-core processor to dual 12-core processors.

I've had a quick look. Each node is a single processor with 4-cores.

I think the biggest point of confusion was the statement "The B&R server is acting as the proxy...", which would imply you are doing hypervisor style backups, but then you said you were using VAW, which doesn't use proxies. The "proxy" bottleneck for agent based backups is the CPU on the host running VAW in that case, because effectively every agent is it's own proxy.

I got the idea of a proxy being involved from the job report at the B&R console.

I didn't think there was a proxy either, we've not set one up. But then Veeam says: "Load: Source 81% > Proxy 91% > Network 84% > Target 2%" in the report and you start to wonder if there really is a proxy after all?

We do have a proxy listed in the Backup Infrastructure section, but it is the B&R server and we didn't put it there or spend any time configuring it.

ProLiant DL380 Gen9 comes standard with 1ge. which would makes sense why your veeam server proxy is the primary bottleneck and your network bottleneck is so high.

We upgraded the card as we were unpacking the server from the boxes. Never used the built-in adapter.

[ejenner]

I'm running a backup of this system during the day so have the opportunity of directly observing...

We're about 2.5 hours into the job. Neither the source nor the B&R server are doing anything particularly dramatic.

All the heavy lifting is happening at the target and the bottleneck is disk. There are 3 backups running concurrently.

All are stuck on 99% and have been for a couple of hours. It seems like the repository is jiggling files around at the back end and that's what takes more time than anything else.