Backup of ESX-Guest slow with Windos Agent (takes 19h for 600GB)

[juergenb]

Hello,
i would like to ask for help about some performance data.

We have some Virtual Guests on an ESX 6.7 Host.

• The Guests use a Windows Veem Agent for Windows 5.0.3.

• The Guest System is a Windows 2019 Server, with Hardwareversion 11 on an ESX 6.7 Hosts with Xeon Gold 6254.

• The Guest System has VMXNet3 Adapters, these show 10GB interfaces.

I don´t have more information from the Host in the moment.

Most of the Backup Data show a read value of 8-9 MB/s

Code: Select all

(C:) (98,9 GB) 42,3 GB read at 10 MB/s
(G:) (90,0 GB) 12,9 GB read at 19 MB/s
(E:) (400,0 GB) 206,2 GB read at 9 MB/s
(F:) (150,0 GB) 26,3 GB read at 13 MB/s
(H:) (400,0 GB) 353,8 GB read at 8 MB/s

The Details column show a Duration of approximate 19h BUT tge Backup duration shows 11:56
Is this normal for a high performance ESX cloud system?

If i do these backups on our Dell Hyper-V System i get way better results.

Code: Select all

(C:) (126,4 GB) 74,2 GB read at 98 MB/s
(D:) (126,9 GB) 67,4 GB read at 115 MB/s

thanks

[PetrM]

Hello,

The first thing to check is the "bottleneck" statistics, here is an example. If we assume that the data read performed by the agent inside VM is the slowest processing stage, you can investigate performance issues with ESXTOP.

Also, why do you use Veeam Agent to protect the virtual machines and not VMware backup job?

One more idea is to open a support case and ask our engineers to check logs and run basic performance tests to find the actual "bottleneck".

Thanks!

[juergenb]

Hi,
have no access to the ESX 6.7 at all.

I am forced to use the esx host as the hosting service provider provides it.
The ESX system is still using 6.7 and I have no control over this.
I also have to pay for such a backup service

I am interested to know if such backup performance is common or just comes from poor ESX deployment.

[PetrM]

Such backup performance is not common, but we cannot say what is the cause without debug logs and access to the infrastructure. I think you should ask our support team to work on it, please paste a support case ID over here for our reference.

Thanks!

[kylecpcs]

Make sure you are using VMXNET3 adapters (appears you are) and also make the recommended adjustments to them for improved throughput.

The following changes will improve guest throughput (of everything including file shares) by 10x or more, and are recommended by both VMware and Microsoft:

From elevated cmd prompt:
• netsh int tcp set global chimney=Disabled (not in 2019 or later)
• netsh int tcp set global ecncapability=Disabled
• netsh int tcp set global timestamps=Disabled
• netsh int tcp set global rsc=disabled
• netsh int tcp set global RSS=Enable
• netsh int tcp set global autotuninglevel=normal
• netsh int ip set global taskoffload=disabled

From advanced NIC properties, disable:
• IPv4 Checksum Offload
• IPv4 TSO Offload
• Large Send Offload V2 (IPV4)
• Large Send Offload V2 (IPV6)
• Offload IP Options
• Offload tagged traffic
• Offload TCP Options
• Recv Segment Coalescing(IPV4)
• Recv Segment Coalescing(IPV6)
• TCP Checksum Offload (IPv4)
• TCP Checksum Offload (IPv6)
• UDP Checksum Offload(IPv4)
• UDP Checksum Offload(IPv6)

[juergenb]

Thanks for you input.

I will verifiy these recommendaton.

These are the current settings (default deployment settings)

IPv4 Checksum Offload (RX & TX Enabled)
IPv4 TSO Offload (Enabled)
Large Send Offload V2 (IPV4) (Enabled)
Large Send Offload V2 (IPV6) (Enabled)
Offload IP Options (Enabled)
Offload tagged traffic (Enabled)
Offload TCP Options (Enabled)
Recv Segment Coalescing(IPV4) (Enabled)
Recv Segment Coalescing(IPV6) (Enabled)
TCP Checksum Offload (IPv4) (Enabled)
TCP Checksum Offload (IPv6) (RX & TX Enabled)
UDP Checksum Offload (IPv4) (RX & TX Enabled)
UDP Checksum Offload (IPv6) (RX & TX Enabled)

Globale TCP-Parameter
----------------------------------------------
Zustand der empfangsseitigen Skalierung : enabled
Autom. Abstimmungsgrad Empfangsfenster : normal
Add-On "Überlastungssteuerungsanbieter" : default
ECN-Funktion : enabled
RFC 1323-Zeitstempel : disabled
RTO (anfänglich) : 3000
Zustand der Empfangssegmentzusammenfügung : enabled
Nicht-SACK-RTT-Resilienz : disabled
Maximale SYN-Neuübertragungen : 2
Fast Open : disabled
Fast Open-Fallback : enabled
HyStart : enabled
Schrittsteuerungsprofil : off