Veeam 6.5 performance....

[maverick964_uk]

We have Veeam 6.5 with daily fulls (yes I know....but customer is demanding it!!).
vcenter 5.1
veeam server is a VM with 16 cores and 16GB ram.
general utilisation is 35% cpu and 7GB ram.

backups write out to HP D2D CIFS shares....which now has the firmware fix to resolve the backup file issue and replication.....

Problem now is performance.
One datastore is taking 14hrs to backup. Has 36VM's
most VM's seem to report the primary bottleneck as Source.

MB's on each VM seems to vary drastically, from 39MB/s to over 200MB/s.

Based on this information we could split the backup into multiple jobs as Veeam only seems to backup sequencially.

Are there configuration options that could help? ie, change block tracking??

thanks!

[maverick964_uk]

another odd one, is the backup reports 30% complete but its done 32 or 36 VM's. This tells me the overall percentage complete is much higher that 30%....!?

[foggy]

most VM's seem to report the primary bottleneck as Source.

What transport mode is used to retrieve source VMs data from the storage?

MB's on each VM seems to vary drastically, from 39MB/s to over 200MB/s.

In fact, this is completely expected.

Are there configuration options that could help? ie, change block tracking??

Since we are talking about every day active fulls here, CBT is not of much help actually.

[maverick964_uk]

thanks for the reply.......I believe hotadd is the transport mode but the data then goes out to the CIFS share.

yes CBT is only for incrementals really. Just wondered if it was useful as CBT saves space based on previous snapshots.

[foggy]

I believe hotadd is the transport mode but the data then goes out to the CIFS share.

Then splitting the job into several ones seems to be the proper way to optimize the backup window.

[averylarry]

If "Source" truly is the bottleneck, you will probably get minimal benefit from splitting the job. Your reads will be less sequential when running multiple jobs at the same time. The primary benefit would be when 1 job isn't reading data because of VM prep/snapshot creation/snapshot removal. So I suppose if your job spends 1 hour doing VM prep/etc., you could theoretically gain that hour back by getting (slightly) better source saturation using multiple jobs.

[averylarry]

another odd one, is the backup reports 30% complete but its done 32 or 36 VM's. This tells me the overall percentage complete is much higher that 30%....!?

From my experience (which I could be misinterpreting), this seems to be a report of the percentage of total disk space that has been backed up. And it does NOT adjust for excluded disks.

VM1 500Gb
VM2 250Gb
VM3 250Gb

After VM1 is finished, it'll say 50% complete.

VM1 700Gb
VM2 200Gb
VM3 100Gb
VM3 excluded disk 2Tb

After VM2 is finished, it'll say 30%, when it's actually 90%.

Again -- that's my interpretation of my experience.

[foggy]

If you are talking about the "Processed" counter, then it is calculated from what CBT has returned for all VMs in a job. This includes empty space and swap blocks that will be jumped through very fast. Not sure about excludes though, need to check.

[maverick964_uk]

yes I think Ted is right here, Veeam is looking at the total disk space.
We dont run any exclusions or CBT at present therefore the percentage seems to base itself on the overall disk space of the datastore I'm trying to backup.

thanks guys!!

BTW, will Veeam 7 offer some parallelism of backing up vm's concurrently in one backup job??

[foggy]

Btw, quick tests showed that excludes are neither considered in the Processed counter value, nor in the amount of data taken as 100% for the entire job to process.

BTW, will Veeam 7 offer some parallelism of backing up vm's concurrently in one backup job??

There will be some improvements in this area, stay tuned.

[maverick964_uk]

many thanks!

When we see poor MB/s how do we go about troubleshooting? It seems difficult as I do think vmware is the bottleneck (the source).
We dont seem to get the write rate and it wont make a difference if we use our virtual proxy or a dedicated one.

Maybe SAN based physical proxy will.....

[foggy]

When we see poor MB/s how do we go about troubleshooting? It seems difficult as I do think vmware is the bottleneck (the source).

Not VMware, but the storage itself, actually.

Maybe SAN based physical proxy will.....

You could try. Generally, physical proxy with direct SAN access can provide better results than hotadd.

[averylarry]

In my experience, for the full backup you're talking about, Veeam doesn't get in the way. So whatever is the slowest amongst source storage, network, destination storage, or (unlikely) destination processing power (for all the dedupe/compression); will be the bottleneck.

Unless you're failing over to network mode. That's just all kinds of unfortunate.