Source bottleneck cause => disk or network?

[unowned]

Hello, I work for an ISV trying to achieve veeam backup-repository certification, as part of this certification, the following performance criteria needs to be met. 4 x 100 GB VM full backup under 30 minutes.

I have the following configuration:

ALl infrastructure connected to 10GBE switch

Windows 2012 R2 Veeam VM (proxy):
- 8 CPU, 24 GB memory, 80 GB disk
- iSCSI connected 500 GB target (for Veeam target/backup repository) - connected via 10 GBe
- VM deployed on vsphere 5.5 u3 sitting on top of Dell R320, raid-0 SAS drive with H330 perc mini raid controller

vsphere 6.5 source:
- 4 x 100GB VM, I am currently trying with only 1 x 100 GB VM at the moment as I cannot even get 1 VM to backup under 30 minutes
- on RAID 0 SSD, R330
- monitor graph shows mild cpu usage

I notice on the veeam performance chart that the backup procedure runs fairly quickly the first 10 minutes ~ 100 mb/s, but a significant and consistent drop off to 40 mb/s afterwards for the rest of the 30 minutes it takes to perform the backup on a single vm, any possible explanations? The corresponding vsphere 6.5 charts also show that the same, a drop off in network and disk usage after the first 10 minutes - it doesn't seem like the source is being stressed whatsoever.

The first 10 minutes saw a peak network transfer rate of 700 mbps and peak read at 110 mb/s (source), then drops to 200 mbps and 25 mb/s.

What's going on - clearly the source is not being stressed?

Furthermore, is there a way for me to test whether I have a network bottleneck or a disk read bottleneck from the source? Usually I would use a utility such as iperf to test network speed between the proxy and vsphere/esxi, but i dont believe esxi has a tool for that, what about disk read speed from esxi? How can I test that.

[DGrinev]

Hi Ben,

Please can you clarify a few things:
1. Which transport mode was used for the backup job? You can find it in the job stats window, by selecting the VM and look for the tag next to the proxy server name selected for processing.
2. What was the bottleneck during the backup process? You will see it in the top of the job stats window. Thanks!

[unowned]

1) I've tried both network transport mode via 10GBe connection (source => 10 GBE switch => proxy), where the proxy was on a separate server than the source, as well as virtual appliance mode. Both modes yielded slow results ~ 50+ minutes for full backup of 1 vm. In fact, virtual appliance mode was actually slower due to the proxy sharing resources with the source.
2) In all instances, "source" was the bottleneck. However, as I had mentioned, it doesn't seem like vsphere was being stressed very much in cpu/memory/disk/network.

I will try to run the job again and post some graphical results, I do not have vcenter so I cannot display anything more than an hour in the past.

[unowned]

backup with source and proxy on separate servers, transport mode via network 10Gbe (notice the performance dip after the first few minutes):


https://drive.google.com/file/d/0B_3ZX1 ... sp=sharing

The bottleneck says source here.

backup with source and proxy on same server, transport mode via virtual appliance mode


https://drive.google.com/file/d/0B_3ZX1 ... sp=sharing

The bottleneck says network here, but the network was not at all being stressed, plus the initial few minutes showed very high network throughput and then it fell off a cliff.

[DGrinev]

The bottleneck source means the disk storage reader component spent most of the time reading data from the storage.
You can review a detailed description of the bottlenecks stats in the sticky thread under the bottleneck analysis section.

Do you have another host and did you try to backup another VM from different storage? We should understand if the storage has an issue or this is the host problem or something else.
I'd check the firmware and make sure you're using latest storage drivers.
Also, it's recommended to use dedicated backup proxy servers.

Please read the discussion about the similar issues on the flash arrays, as it contains a few suggestions on how to improve it.
Do not hesitate to contact the support team in order to let them take a closer look at your environment, also they will run a default test to check reading speed of the source. Thanks!

[unowned]

Thank you DGrinev,

When you mean "dedicated backup proxy servers", do you mean baremetal?

Also, I have already opened a support ticket, how can I expedite the process to receive help, I have been stuck on the source /proxy side of things for a long time, as I am very sure our backup repository target should perform well for the certification.

[DGrinev]

Hi,

Under dedicated server I mean a VM or physical server with the only backup proxy role deployed.
You can ask the support by email to escalate or update your request. Thanks!