bizarre (slow? inaccurate?) performance

[scottmlew]

I am prototyping B&R v10, using CE for now. I am backing up a single Win10 endpoint with a 2TB SSD. This is connected to my backup server and repository over 1Gb ethernet. Backup repo is mirrored StorageSpaces. Repo and server are connected by 10Gb ethernet.

I am seeing a reported processing rate of 18MB/s, with proxy reported as the bottleneck. I assume B&R is using the backup server as the proxy, and it is using minimal memory and CPU, so I'm surprised that would be the bottleneck.

The most odd part (to me at least, since I don't understand) is why for data it is showing 21.7TB processed (97%), 989.1GB read, and 12TB transferred (0.1x). What's going on there? Does that mean it is reprocessing the same data over and over again? Recall the machine I'm backing up has only a 2TB drive (and I'm not doing anything weird with sparse files or anything like that).

Any suggestions as to what's going on?

[Egor Yakovlev]

Hi Scott.
Statistics do look strange. What type of backup mode you are running: is it Entire Computer or Volume\File level backup?
Also, please hit Job Statistic window in VBR console(double click), select Win10 machine in the list on the left side and see detailed per-disk stats in the log here. Veeam surely cannot read 22TB from 2TB SSD.
/Thanks!

[scottmlew]

Hi Egor, thanks for the reply. I am doing a Full Computer backup, and I do NOT have the option to include external USB drives checked (and there are 2 external drives, 5TB each...so even if they were somehow being included, the stats don't make sense).

I believe I am looking for the per-disk stats in the right place, but I see only the data from the incremental that ran early this morning (which has somewhat surprising but sane statistics -- 289GB read at 45MB/s). I don't see how to see the data from the original backup. Is there a way to see it?

I am looking at a screenshot I took when the job completed, and it looks like the stats adjusted themselves when the job ended. The final stats show data processed 1.6TB, read 1.4TB, and transferred 951GB (1.5x). The summary shows a duration of 16:53 with processing rate of 25MB/s, but bottleneck shows N/A.

So I am guessing there is a bug in the real-time data display that does not affect the final job stats. But this is still terrible backup performance, and I thought there should always be a bottleneck displayed?

[Egor Yakovlev]

Previous backups sessions are shown under History -> Backup -> Agent tab of Veeam console, however even incremental run should show detected source disk capacity. Is there a chance to force\wait for scheduled Full backup to confirm if stats are giving crazy numbers all the time during backup process or was it a one time event?
As for performance, I take it took 17 hours, right? With minutes and 951GB transferred data seems to give about 954MB\s actual backup traffic throughput, pretty good but impossible via 1g link to the repository. 17 hours gives 16MB\s which seems quite slow, yes.
/Thanks!

[PetrM]

Hello,

Considering that the reported "bottleneck" is Proxy, I would check CPU usage level on computer you're backing up. Most likely source Data Mover consumes CPU resources in order to perform data compression. May be it's worth testing a job with decreased compression, for example Dedupe-Friendly level (look at the advanced storage settings) ?

Thanks!

[scottmlew]

Previous backups sessions are shown under History -> Backup -> Agent tab of Veeam console, however even incremental run should show detected source disk capacity. Is there a chance to force\wait for scheduled Full backup to confirm if stats are giving crazy numbers all the time during backup process or was it a one time event?

I'm sorry to be dense...where is this history tab? I will continue to monitor and post updates to see if this is a one-time event.

As for performance, I take it took 17 hours, right? With minutes and 951GB transferred data seems to give about 954MB\s actual backup traffic throughput, pretty good but impossible via 1g link to the repository. 17 hours gives 16MB\s which seems quite slow, yes.

Yeah, it was 17 hours.

[scottmlew]

Considering that the reported "bottleneck" is Proxy, I would check CPU usage level on computer you're backing up. Most likely source Data Mover consumes CPU resources in order to perform data compression. May be it's worth testing a job with decreased compression, for example Dedupe-Friendly level (look at the advanced storage settings) ?

I will look into this possibility more, but my recollection is that the source computer was showing plenty of available CPU.

[Egor Yakovlev]

History view could be found in the bottom left corner of Veeam console. Also worth checking Protection Group settings that has deployed said agent, as by default, Veeam Agent for Windows - Workstation edition will be automatically throttling IO during backup to reduce load on working machines.
/Cheers!

[scottmlew]

Ah, I see History now...that panel was too small for it to appear. The stats shown in history match the job completion stats, so it does look like a bug with the real-time stats. There is no network throttling set for the protection group, but it does have I/O throttling applied. I will play with that setting and see what results I get. Since the machine was largely idle in that period and has an SSD, I'm not optimistic that was the issue, but I'll see.

[scottmlew]

I am continuing to try to gather data on this issue. I have replaced the destination drives with WD Reds, and that gave a big speed increase at first when I did a new full backup, but by the end of the job it was showing only ~20MB/s for an overall average of 64MB/s. Incrementals since have shown much better performance, ~180MB/s. (the incremental speed is good IMHO since it is going over a 1Gb/s link)

One thing I've noticed is that the backup proceeds in bursts. I'm not sure if that is expected. It is easier to see on another system that has less data backed up so the graph is clearer (also SSD on a 1Gb/s connection). You can see what I mean here:

There is definitely something weird going on with backup sizes. Recall my earlier data point where it said I transferred 7x the capacity of my disk. Since then, I've also noted that when I do a restore from the same system (2TB disk) it says the VM size (which should really not say VM since this is an agent on a physical host) is 9.3TB. If that's supposed to be the total size of all backups for that machine, it's still wrong, because when I sum up the amount read for the initial full and the 2 incrementals since then, it is only 1.6TB (and files in the repo are only 1.06TB)

Also, it doesn't appear that the bottleneck is reported for an agent backup -- is this the expected behavior?

[Egor Yakovlev]

Mention of "VM" in the recovery wizard for agent is a known UI bug, we will fix it. Wrong size and bottleneck detection issues should be investigated closely - I would say let's give it a shot with our support team. Please share case ID here.
/Thanks!

[scottmlew]

Ok, case 04153149 is opened. I attached the Endpoint\Backup\Managed logs; let me know if there are others I should add.